Research on optimization of storage system in intelligent computing center
The intelligent computing center uses distributed file storage for data preprocessing and model training, distributed object storage for the acquisition of raw data and model release, and distributed block storage to provide storage for the resource management platform. Meanwhile, high-performance d...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Beijing Xintong Media Co., Ltd
2025-07-01
|
| Series: | Dianxin kexue |
| Subjects: | |
| Online Access: | http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2025160/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849398144655163392 |
|---|---|
| author | CAO Yuanming LEI Ming LIU Qin NIU Yingxia WU Zhenyu PAN Jie |
| author_facet | CAO Yuanming LEI Ming LIU Qin NIU Yingxia WU Zhenyu PAN Jie |
| author_sort | CAO Yuanming |
| collection | DOAJ |
| description | The intelligent computing center uses distributed file storage for data preprocessing and model training, distributed object storage for the acquisition of raw data and model release, and distributed block storage to provide storage for the resource management platform. Meanwhile, high-performance distributed file storage is used to shorten the read and write time of checkpoint during the training process and improve the training efficiency of the cluster. The entire life cycle of large model training requires data copying and migration between storage systems with different storage protocols and different read-write performances, resulting in duplicate data storage. Additionally, data copying requires computing resources and network bandwidth. To address the above issues and provide a unified namespace for the intelligent computing clusters, a file and object converged storage and a hierarchical file storage scheme were proposed to solve the problem of data transfer between different storage protocols and enable automatic data flow between high-performance file storage (all SSD) and ordinary-performance file storage (SSD and HDD), providing a reference for the optimization of storage systems of ultra-large-scale intelligent computing clusters. |
| format | Article |
| id | doaj-art-8a89a2e225ad40b8b047f7e0fc3a0022 |
| institution | Kabale University |
| issn | 1000-0801 |
| language | zho |
| publishDate | 2025-07-01 |
| publisher | Beijing Xintong Media Co., Ltd |
| record_format | Article |
| series | Dianxin kexue |
| spelling | doaj-art-8a89a2e225ad40b8b047f7e0fc3a00222025-08-20T03:38:43ZzhoBeijing Xintong Media Co., LtdDianxin kexue1000-08012025-07-0141164175120127889Research on optimization of storage system in intelligent computing centerCAO YuanmingLEI MingLIU QinNIU YingxiaWU ZhenyuPAN JieThe intelligent computing center uses distributed file storage for data preprocessing and model training, distributed object storage for the acquisition of raw data and model release, and distributed block storage to provide storage for the resource management platform. Meanwhile, high-performance distributed file storage is used to shorten the read and write time of checkpoint during the training process and improve the training efficiency of the cluster. The entire life cycle of large model training requires data copying and migration between storage systems with different storage protocols and different read-write performances, resulting in duplicate data storage. Additionally, data copying requires computing resources and network bandwidth. To address the above issues and provide a unified namespace for the intelligent computing clusters, a file and object converged storage and a hierarchical file storage scheme were proposed to solve the problem of data transfer between different storage protocols and enable automatic data flow between high-performance file storage (all SSD) and ordinary-performance file storage (SSD and HDD), providing a reference for the optimization of storage systems of ultra-large-scale intelligent computing clusters.http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2025160/converged storagehierarchical storagedistributed file storagedistributed object storage |
| spellingShingle | CAO Yuanming LEI Ming LIU Qin NIU Yingxia WU Zhenyu PAN Jie Research on optimization of storage system in intelligent computing center Dianxin kexue converged storage hierarchical storage distributed file storage distributed object storage |
| title | Research on optimization of storage system in intelligent computing center |
| title_full | Research on optimization of storage system in intelligent computing center |
| title_fullStr | Research on optimization of storage system in intelligent computing center |
| title_full_unstemmed | Research on optimization of storage system in intelligent computing center |
| title_short | Research on optimization of storage system in intelligent computing center |
| title_sort | research on optimization of storage system in intelligent computing center |
| topic | converged storage hierarchical storage distributed file storage distributed object storage |
| url | http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2025160/ |
| work_keys_str_mv | AT caoyuanming researchonoptimizationofstoragesysteminintelligentcomputingcenter AT leiming researchonoptimizationofstoragesysteminintelligentcomputingcenter AT liuqin researchonoptimizationofstoragesysteminintelligentcomputingcenter AT niuyingxia researchonoptimizationofstoragesysteminintelligentcomputingcenter AT wuzhenyu researchonoptimizationofstoragesysteminintelligentcomputingcenter AT panjie researchonoptimizationofstoragesysteminintelligentcomputingcenter |