Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform
With the advancement of meteorological services, various product processing systems and supporting data management systems have been developed for different business systems. However, this has led to the problem of system non-intensification. The lack of intensification in meteorological services ca...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Editorial Office of Journal of Applied Meteorological Science
2024-01-01
|
| Series: | 应用气象学报 |
| Subjects: | |
| Online Access: | http://qikan.camscma.cn/en/article/doi/10.11898/1001-7313.20240107 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850285298955583488 |
|---|---|
| author | Huo Qing He Wenchun He Lin Gao Feng Chen Shiwang Xu Yongjun |
| author_facet | Huo Qing He Wenchun He Lin Gao Feng Chen Shiwang Xu Yongjun |
| author_sort | Huo Qing |
| collection | DOAJ |
| description | With the advancement of meteorological services, various product processing systems and supporting data management systems have been developed for different business systems. However, this has led to the problem of system non-intensification. The lack of intensification in meteorological services can result in inconsistent data standards and make the operation and maintenance more challenging, which can lead to significant waste in investment due to duplicate data storage and inconsistent data caused by untimely synchronization. Moreover, the lack of information and technology hinders the integration of upstream and downstream businesses.The intensive development of meteorological business systems and the reform of "cloud+end" technology system are important measures to achieve high-quality development of meteorological business. China Meteorological Administration proposed to build a "cloud+end" technology system in 2020. CMA Big Data and Cloud Platform serves as the cloud, while the meteorological business system is the end, clarifying the positioning of CMA Big Data and Cloud Platform as a key foundational technology platform. The data processing line (DPL) is an intensive environment for meteorological algorithms. It addresses business needs such as efficient and stable processing of data products, data sharing and collaboration among business systems, and efficient and intensive business applications. To achieve this goal, algorithm libraries and task control functions are established by utilizing technologies such as integrated digital and computing, efficient task scheduling, visual process arrangement, and containers. The algorithm library facilitates the standardization, unified management and sharing of algorithms. Task control supports multiple scheduling strategies with high reliability and fault tolerance, enabling efficient and stable scheduling operations for algorithms. All functions mentioned above are in the form of interfaces for the application frontend. At the same time, based on the meteorological business comprehensive monitoring system (referred to as Tianjing) enables automatic collection of algorithm operation status and detection of abnormal alarms. Since its operation in 2021, the processing assembly line has facilitated the real-time operation of 202 business systems nationwide, resulting in a performance improvement of 1-10 times and a significant increase in efficiency. It plays an important supporting role in improving the operational efficiency of business systems, enhancing their collaboration, accelerating the process of "cloud+end" business technology system reform, and promoting the intensive development of meteorological business. |
| format | Article |
| id | doaj-art-36d2be28971e4d8099e8b2e061fd8765 |
| institution | OA Journals |
| issn | 1001-7313 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | Editorial Office of Journal of Applied Meteorological Science |
| record_format | Article |
| series | 应用气象学报 |
| spelling | doaj-art-36d2be28971e4d8099e8b2e061fd87652025-08-20T01:47:19ZengEditorial Office of Journal of Applied Meteorological Science应用气象学报1001-73132024-01-01351808910.11898/1001-7313.20240107yyqxxb-35-1-80Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud PlatformHuo Qing0He Wenchun1He Lin2Gao Feng3Chen Shiwang4Xu Yongjun5National Meteorological Information Center, Beijing 100081National Meteorological Information Center, Beijing 100081Shaanxi Meteorological Information Center, Xi'an 710014National Meteorological Information Center, Beijing 100081National Meteorological Information Center, Beijing 100081National Meteorological Information Center, Beijing 100081With the advancement of meteorological services, various product processing systems and supporting data management systems have been developed for different business systems. However, this has led to the problem of system non-intensification. The lack of intensification in meteorological services can result in inconsistent data standards and make the operation and maintenance more challenging, which can lead to significant waste in investment due to duplicate data storage and inconsistent data caused by untimely synchronization. Moreover, the lack of information and technology hinders the integration of upstream and downstream businesses.The intensive development of meteorological business systems and the reform of "cloud+end" technology system are important measures to achieve high-quality development of meteorological business. China Meteorological Administration proposed to build a "cloud+end" technology system in 2020. CMA Big Data and Cloud Platform serves as the cloud, while the meteorological business system is the end, clarifying the positioning of CMA Big Data and Cloud Platform as a key foundational technology platform. The data processing line (DPL) is an intensive environment for meteorological algorithms. It addresses business needs such as efficient and stable processing of data products, data sharing and collaboration among business systems, and efficient and intensive business applications. To achieve this goal, algorithm libraries and task control functions are established by utilizing technologies such as integrated digital and computing, efficient task scheduling, visual process arrangement, and containers. The algorithm library facilitates the standardization, unified management and sharing of algorithms. Task control supports multiple scheduling strategies with high reliability and fault tolerance, enabling efficient and stable scheduling operations for algorithms. All functions mentioned above are in the form of interfaces for the application frontend. At the same time, based on the meteorological business comprehensive monitoring system (referred to as Tianjing) enables automatic collection of algorithm operation status and detection of abnormal alarms. Since its operation in 2021, the processing assembly line has facilitated the real-time operation of 202 business systems nationwide, resulting in a performance improvement of 1-10 times and a significant increase in efficiency. It plays an important supporting role in improving the operational efficiency of business systems, enhancing their collaboration, accelerating the process of "cloud+end" business technology system reform, and promoting the intensive development of meteorological business.http://qikan.camscma.cn/en/article/doi/10.11898/1001-7313.20240107data processing linemeteorology algorithm librarytask schedulingdata and calculation integrationintensive development"cloud+end" technical system reform |
| spellingShingle | Huo Qing He Wenchun He Lin Gao Feng Chen Shiwang Xu Yongjun Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform 应用气象学报 data processing line meteorology algorithm library task scheduling data and calculation integration intensive development "cloud+end" technical system reform |
| title | Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform |
| title_full | Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform |
| title_fullStr | Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform |
| title_full_unstemmed | Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform |
| title_short | Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform |
| title_sort | design and application of algorithm intensive environment for cma big data and cloud platform |
| topic | data processing line meteorology algorithm library task scheduling data and calculation integration intensive development "cloud+end" technical system reform |
| url | http://qikan.camscma.cn/en/article/doi/10.11898/1001-7313.20240107 |
| work_keys_str_mv | AT huoqing designandapplicationofalgorithmintensiveenvironmentforcmabigdataandcloudplatform AT hewenchun designandapplicationofalgorithmintensiveenvironmentforcmabigdataandcloudplatform AT helin designandapplicationofalgorithmintensiveenvironmentforcmabigdataandcloudplatform AT gaofeng designandapplicationofalgorithmintensiveenvironmentforcmabigdataandcloudplatform AT chenshiwang designandapplicationofalgorithmintensiveenvironmentforcmabigdataandcloudplatform AT xuyongjun designandapplicationofalgorithmintensiveenvironmentforcmabigdataandcloudplatform |