Research and application of big data platform technology based on multi-centre collaborative computing
China Telecom has launched a high-efficient and collaborative wide-area big data architecture system, the cloud edge computing big data platform, for large-scale governmental and enterprise organizations spanning multiple geographies and clusters. The platform logically abstracts data partitions thr...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Beijing Xintong Media Co., Ltd
2024-05-01
|
| Series: | Dianxin kexue |
| Subjects: | |
| Online Access: | http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2024152/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841528900733108224 |
|---|---|
| author | RUAN Yilong XU Xueling FA Hu DONG Silun JIANG Lei YANG Lei YAN Yuanyuan |
| author_facet | RUAN Yilong XU Xueling FA Hu DONG Silun JIANG Lei YANG Lei YAN Yuanyuan |
| author_sort | RUAN Yilong |
| collection | DOAJ |
| description | China Telecom has launched a high-efficient and collaborative wide-area big data architecture system, the cloud edge computing big data platform, for large-scale governmental and enterprise organizations spanning multiple geographies and clusters. The platform logically abstracts data partitions through the cluster dimension, integrates multiple independent datasets into a "virtual dataset", and achieves many-to-one dataset mapping management. At the same time, the computing load dataset of the platform has generalized characteristics, which can flexibly cope with the data processing requirements in different scenarios. In addition, the platform also supports a variety of computing engines and scheduling systems using relational expressions as intermediate representations to achieve batch tasks for large-scale, complex data processing in highly fault-tolerant scenarios. At present, the cloud edge computing big data platform has been applied in a variety of application scenes. The platform has improved efficiency by 17% in 5G Core capacity scheduling subsystem (5GC) multi-centre big data job development and operation, and has achieved the collaborative scheduling of a total of 42 PB of storage, 84 TB of memory, and 24 984 VCore computing resources, with a daily average of 80 308 times of task scheduling between the front cluster and the core cluster. |
| format | Article |
| id | doaj-art-e2213e2343e943d7af60661cb542eb1c |
| institution | Kabale University |
| issn | 1000-0801 |
| language | zho |
| publishDate | 2024-05-01 |
| publisher | Beijing Xintong Media Co., Ltd |
| record_format | Article |
| series | Dianxin kexue |
| spelling | doaj-art-e2213e2343e943d7af60661cb542eb1c2025-01-15T03:33:31ZzhoBeijing Xintong Media Co., LtdDianxin kexue1000-08012024-05-014014115160129687Research and application of big data platform technology based on multi-centre collaborative computingRUAN YilongXU XuelingFA HuDONG SilunJIANG LeiYANG LeiYAN YuanyuanChina Telecom has launched a high-efficient and collaborative wide-area big data architecture system, the cloud edge computing big data platform, for large-scale governmental and enterprise organizations spanning multiple geographies and clusters. The platform logically abstracts data partitions through the cluster dimension, integrates multiple independent datasets into a "virtual dataset", and achieves many-to-one dataset mapping management. At the same time, the computing load dataset of the platform has generalized characteristics, which can flexibly cope with the data processing requirements in different scenarios. In addition, the platform also supports a variety of computing engines and scheduling systems using relational expressions as intermediate representations to achieve batch tasks for large-scale, complex data processing in highly fault-tolerant scenarios. At present, the cloud edge computing big data platform has been applied in a variety of application scenes. The platform has improved efficiency by 17% in 5G Core capacity scheduling subsystem (5GC) multi-centre big data job development and operation, and has achieved the collaborative scheduling of a total of 42 PB of storage, 84 TB of memory, and 24 984 VCore computing resources, with a daily average of 80 308 times of task scheduling between the front cluster and the core cluster.http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2024152/cloud edge collaborationuniform SQLtask optimizationbig data platform |
| spellingShingle | RUAN Yilong XU Xueling FA Hu DONG Silun JIANG Lei YANG Lei YAN Yuanyuan Research and application of big data platform technology based on multi-centre collaborative computing Dianxin kexue cloud edge collaboration uniform SQL task optimization big data platform |
| title | Research and application of big data platform technology based on multi-centre collaborative computing |
| title_full | Research and application of big data platform technology based on multi-centre collaborative computing |
| title_fullStr | Research and application of big data platform technology based on multi-centre collaborative computing |
| title_full_unstemmed | Research and application of big data platform technology based on multi-centre collaborative computing |
| title_short | Research and application of big data platform technology based on multi-centre collaborative computing |
| title_sort | research and application of big data platform technology based on multi centre collaborative computing |
| topic | cloud edge collaboration uniform SQL task optimization big data platform |
| url | http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2024152/ |
| work_keys_str_mv | AT ruanyilong researchandapplicationofbigdataplatformtechnologybasedonmulticentrecollaborativecomputing AT xuxueling researchandapplicationofbigdataplatformtechnologybasedonmulticentrecollaborativecomputing AT fahu researchandapplicationofbigdataplatformtechnologybasedonmulticentrecollaborativecomputing AT dongsilun researchandapplicationofbigdataplatformtechnologybasedonmulticentrecollaborativecomputing AT jianglei researchandapplicationofbigdataplatformtechnologybasedonmulticentrecollaborativecomputing AT yanglei researchandapplicationofbigdataplatformtechnologybasedonmulticentrecollaborativecomputing AT yanyuanyuan researchandapplicationofbigdataplatformtechnologybasedonmulticentrecollaborativecomputing |