Research on optimization of storage system in intelligent computing center
The intelligent computing center uses distributed file storage for data preprocessing and model training, distributed object storage for the acquisition of raw data and model release, and distributed block storage to provide storage for the resource management platform. Meanwhile, high-performance d...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Beijing Xintong Media Co., Ltd
2025-07-01
|
| Series: | Dianxin kexue |
| Subjects: | |
| Online Access: | http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2025160/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | The intelligent computing center uses distributed file storage for data preprocessing and model training, distributed object storage for the acquisition of raw data and model release, and distributed block storage to provide storage for the resource management platform. Meanwhile, high-performance distributed file storage is used to shorten the read and write time of checkpoint during the training process and improve the training efficiency of the cluster. The entire life cycle of large model training requires data copying and migration between storage systems with different storage protocols and different read-write performances, resulting in duplicate data storage. Additionally, data copying requires computing resources and network bandwidth. To address the above issues and provide a unified namespace for the intelligent computing clusters, a file and object converged storage and a hierarchical file storage scheme were proposed to solve the problem of data transfer between different storage protocols and enable automatic data flow between high-performance file storage (all SSD) and ordinary-performance file storage (SSD and HDD), providing a reference for the optimization of storage systems of ultra-large-scale intelligent computing clusters. |
|---|---|
| ISSN: | 1000-0801 |