Design and Application of a Data-computation Integrated Database for Meteorological Grid Data
Meteorological gridded data is typically stored in file formats within distributed file repositories, such as network-attached Storage (NAS). During operations, business systems often need to download files locally, parse them, and subsequently perform analyses and calculations. This traditional app...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Editorial Office of Journal of Applied Meteorological Science
2025-01-01
|
| Series: | 应用气象学报 |
| Subjects: | |
| Online Access: | http://qikan.camscma.cn/en/article/doi/10.11898/1001-7313.20250112 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832586897472356352 |
|---|---|
| author | Wang Shu Xu Yongjun He Wenchun Wu Huanping Gao Feng Liu Yuanyuan Liu Bei Lü Guanru Ni Xuelei |
| author_facet | Wang Shu Xu Yongjun He Wenchun Wu Huanping Gao Feng Liu Yuanyuan Liu Bei Lü Guanru Ni Xuelei |
| author_sort | Wang Shu |
| collection | DOAJ |
| description | Meteorological gridded data is typically stored in file formats within distributed file repositories, such as network-attached Storage (NAS). During operations, business systems often need to download files locally, parse them, and subsequently perform analyses and calculations. This traditional approach presents several challenges, including difficulties in data retrieval, prolonged response times, and inability to meet demands for real-time computation and interactive applications. To address these issues, National Meteorological Information Center has developed PostGrid, an integrated database for meteorological gridded data and computing, based on Tianqing Spatial Analysis Library and is specifically designed for distributed environments. The PostGrid database consists of two primary layers: Data layer and operator layer. Data layer is responsible for storing various types of gridded meteorological data. When data is imported into the database, it is stored in a standardized and uniform manner. Each dataset comprises two components: A header file and entity data, both of which are stored in binary format. The header file contains basic descriptive information about the meteorological gridded data, while the entity data store specific layers or fields obtained by partitioning the original gridded dataset. By organizing data according to various dimensions, such as weather elements, forecast start times, spatial layers, levels, and samples-the data layer facilitates the efficient retrieval and analysis of meteorological gridded data. This structured approach significantly enhances the database's capacity to read and process data, rendering it far more efficient than traditional methods. Operator layer in PostGrid is implemented using SQL functions within the database. These operators facilitate a range of operations on gridded data, including matrix calculations, spatial analysis, statistical aggregation, dimensionality reduction, and data filtering. Furthermore, operators are designed to facilitate distributed parallel computing, enabling faster and more efficient processing of large datasets. By leveraging capabilities of parallel computing, PostGrid can perform complex calculations that would typically require minutes, reducing the processing time to just milliseconds. This capability significantly enhances the performance and flexibility of meteorological data services. Performance tests and real-world applications have demonstrated that PostGrid significantly enhances the efficiency of meteorological data services, reducing the time required for traditional aggregation calculations from minutes to just milliseconds. The database's capability to integrate both data and computation within a unified platform marks a significant advancement in the management of large-scale meteorological data. It enables faster data retrieval, real-time computation, and supports more advanced interactive applications, making it an invaluable tool for meteorological services and with the potential for widespread application across various sectors within meteorology. |
| format | Article |
| id | doaj-art-eaf5f06339c94173897dd9867885f931 |
| institution | Kabale University |
| issn | 1001-7313 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Editorial Office of Journal of Applied Meteorological Science |
| record_format | Article |
| series | 应用气象学报 |
| spelling | doaj-art-eaf5f06339c94173897dd9867885f9312025-01-25T00:42:48ZengEditorial Office of Journal of Applied Meteorological Science应用气象学报1001-73132025-01-0136112112810.11898/1001-7313.20250112yyqxxb-36-1-121Design and Application of a Data-computation Integrated Database for Meteorological Grid DataWang Shu0Xu Yongjun1He Wenchun2Wu Huanping3Gao Feng4Liu Yuanyuan5Liu Bei6Lü Guanru7Ni Xuelei8National Meteorological Information Center, Beijing 100081National Meteorological Information Center, Beijing 100081National Meteorological Information Center, Beijing 100081National Climate Center, Beijing 100081National Meteorological Information Center, Beijing 100081National Meteorological Information Center, Beijing 100081National Climate Center, Beijing 100081Hunan Meteorological Information Center, Changsha 410118National Meteorological Information Center, Beijing 100081Meteorological gridded data is typically stored in file formats within distributed file repositories, such as network-attached Storage (NAS). During operations, business systems often need to download files locally, parse them, and subsequently perform analyses and calculations. This traditional approach presents several challenges, including difficulties in data retrieval, prolonged response times, and inability to meet demands for real-time computation and interactive applications. To address these issues, National Meteorological Information Center has developed PostGrid, an integrated database for meteorological gridded data and computing, based on Tianqing Spatial Analysis Library and is specifically designed for distributed environments. The PostGrid database consists of two primary layers: Data layer and operator layer. Data layer is responsible for storing various types of gridded meteorological data. When data is imported into the database, it is stored in a standardized and uniform manner. Each dataset comprises two components: A header file and entity data, both of which are stored in binary format. The header file contains basic descriptive information about the meteorological gridded data, while the entity data store specific layers or fields obtained by partitioning the original gridded dataset. By organizing data according to various dimensions, such as weather elements, forecast start times, spatial layers, levels, and samples-the data layer facilitates the efficient retrieval and analysis of meteorological gridded data. This structured approach significantly enhances the database's capacity to read and process data, rendering it far more efficient than traditional methods. Operator layer in PostGrid is implemented using SQL functions within the database. These operators facilitate a range of operations on gridded data, including matrix calculations, spatial analysis, statistical aggregation, dimensionality reduction, and data filtering. Furthermore, operators are designed to facilitate distributed parallel computing, enabling faster and more efficient processing of large datasets. By leveraging capabilities of parallel computing, PostGrid can perform complex calculations that would typically require minutes, reducing the processing time to just milliseconds. This capability significantly enhances the performance and flexibility of meteorological data services. Performance tests and real-world applications have demonstrated that PostGrid significantly enhances the efficiency of meteorological data services, reducing the time required for traditional aggregation calculations from minutes to just milliseconds. The database's capability to integrate both data and computation within a unified platform marks a significant advancement in the management of large-scale meteorological data. It enables faster data retrieval, real-time computation, and supports more advanced interactive applications, making it an invaluable tool for meteorological services and with the potential for widespread application across various sectors within meteorology.http://qikan.camscma.cn/en/article/doi/10.11898/1001-7313.20250112data-computation integrationmeteorological grid dataparallel computingdistributed systems |
| spellingShingle | Wang Shu Xu Yongjun He Wenchun Wu Huanping Gao Feng Liu Yuanyuan Liu Bei Lü Guanru Ni Xuelei Design and Application of a Data-computation Integrated Database for Meteorological Grid Data 应用气象学报 data-computation integration meteorological grid data parallel computing distributed systems |
| title | Design and Application of a Data-computation Integrated Database for Meteorological Grid Data |
| title_full | Design and Application of a Data-computation Integrated Database for Meteorological Grid Data |
| title_fullStr | Design and Application of a Data-computation Integrated Database for Meteorological Grid Data |
| title_full_unstemmed | Design and Application of a Data-computation Integrated Database for Meteorological Grid Data |
| title_short | Design and Application of a Data-computation Integrated Database for Meteorological Grid Data |
| title_sort | design and application of a data computation integrated database for meteorological grid data |
| topic | data-computation integration meteorological grid data parallel computing distributed systems |
| url | http://qikan.camscma.cn/en/article/doi/10.11898/1001-7313.20250112 |
| work_keys_str_mv | AT wangshu designandapplicationofadatacomputationintegrateddatabaseformeteorologicalgriddata AT xuyongjun designandapplicationofadatacomputationintegrateddatabaseformeteorologicalgriddata AT hewenchun designandapplicationofadatacomputationintegrateddatabaseformeteorologicalgriddata AT wuhuanping designandapplicationofadatacomputationintegrateddatabaseformeteorologicalgriddata AT gaofeng designandapplicationofadatacomputationintegrateddatabaseformeteorologicalgriddata AT liuyuanyuan designandapplicationofadatacomputationintegrateddatabaseformeteorologicalgriddata AT liubei designandapplicationofadatacomputationintegrateddatabaseformeteorologicalgriddata AT luguanru designandapplicationofadatacomputationintegrateddatabaseformeteorologicalgriddata AT nixuelei designandapplicationofadatacomputationintegrateddatabaseformeteorologicalgriddata |