Multimodal data generative fusion method for complex system health condition estimation
Abstract For the health management of complex systems, the high value of such systems often necessitates multimodal monitoring data, including video surveillance, internal sensors, empirical formulas, and even digital twins. Therefore, it is essential to design an effective intelligent fusion method...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-06-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-025-04985-3 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850136567349248000 |
|---|---|
| author | Weijie Kang Xianyang Zhang Jiarui Zhang Xudan Chen Honglan Huang Bing He Weiwei Qin Haizhen Zhu |
| author_facet | Weijie Kang Xianyang Zhang Jiarui Zhang Xudan Chen Honglan Huang Bing He Weiwei Qin Haizhen Zhu |
| author_sort | Weijie Kang |
| collection | DOAJ |
| description | Abstract For the health management of complex systems, the high value of such systems often necessitates multimodal monitoring data, including video surveillance, internal sensors, empirical formulas, and even digital twins. Therefore, it is essential to design an effective intelligent fusion method for multimodal data. Firstly, a global monotonicity calculation method and a time series data augmentation technique are developed to address the inconsistencies arising from varying temporal lengths across different modalities. Secondly, in response to the need for efficient time series fusion, we propose a fast sequential learning network architecture along with a time series generative data structure. Finally, we introduce a many-to-many transfer training approach that culminates in the formation of a Multi-source Generative Adversarial Network (Ms-GAN). Numerical experiments and monitoring datasets are employed to validate the effectiveness of this multimodal generative fusion method. Notably, Ms-GAN enhances traditional GANs—typically limited to learning single data distributions—by enabling multimodal data fusion capabilities. This advancement holds significant promise for applications in various fields such as multimedia processing and medical diagnosis. |
| format | Article |
| id | doaj-art-b71b9aeceecf4711a4c0e43e9d526638 |
| institution | OA Journals |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-b71b9aeceecf4711a4c0e43e9d5266382025-08-20T02:31:04ZengNature PortfolioScientific Reports2045-23222025-06-0115111710.1038/s41598-025-04985-3Multimodal data generative fusion method for complex system health condition estimationWeijie Kang0Xianyang Zhang1Jiarui Zhang2Xudan Chen3Honglan Huang4Bing He5Weiwei Qin6Haizhen Zhu7Rocket Force University of EngineeringRocket Force University of EngineeringNorthwest Institute of Nuclear TechnologyRocket Force University of EngineeringRocket Force University of EngineeringRocket Force University of EngineeringRocket Force University of EngineeringAir Force Engineering UniversityAbstract For the health management of complex systems, the high value of such systems often necessitates multimodal monitoring data, including video surveillance, internal sensors, empirical formulas, and even digital twins. Therefore, it is essential to design an effective intelligent fusion method for multimodal data. Firstly, a global monotonicity calculation method and a time series data augmentation technique are developed to address the inconsistencies arising from varying temporal lengths across different modalities. Secondly, in response to the need for efficient time series fusion, we propose a fast sequential learning network architecture along with a time series generative data structure. Finally, we introduce a many-to-many transfer training approach that culminates in the formation of a Multi-source Generative Adversarial Network (Ms-GAN). Numerical experiments and monitoring datasets are employed to validate the effectiveness of this multimodal generative fusion method. Notably, Ms-GAN enhances traditional GANs—typically limited to learning single data distributions—by enabling multimodal data fusion capabilities. This advancement holds significant promise for applications in various fields such as multimedia processing and medical diagnosis.https://doi.org/10.1038/s41598-025-04985-3Multimodal informationHealth managementHealth condition EstimationGenerative adversarial networksKernel canonical correlation analysis |
| spellingShingle | Weijie Kang Xianyang Zhang Jiarui Zhang Xudan Chen Honglan Huang Bing He Weiwei Qin Haizhen Zhu Multimodal data generative fusion method for complex system health condition estimation Scientific Reports Multimodal information Health management Health condition Estimation Generative adversarial networks Kernel canonical correlation analysis |
| title | Multimodal data generative fusion method for complex system health condition estimation |
| title_full | Multimodal data generative fusion method for complex system health condition estimation |
| title_fullStr | Multimodal data generative fusion method for complex system health condition estimation |
| title_full_unstemmed | Multimodal data generative fusion method for complex system health condition estimation |
| title_short | Multimodal data generative fusion method for complex system health condition estimation |
| title_sort | multimodal data generative fusion method for complex system health condition estimation |
| topic | Multimodal information Health management Health condition Estimation Generative adversarial networks Kernel canonical correlation analysis |
| url | https://doi.org/10.1038/s41598-025-04985-3 |
| work_keys_str_mv | AT weijiekang multimodaldatagenerativefusionmethodforcomplexsystemhealthconditionestimation AT xianyangzhang multimodaldatagenerativefusionmethodforcomplexsystemhealthconditionestimation AT jiaruizhang multimodaldatagenerativefusionmethodforcomplexsystemhealthconditionestimation AT xudanchen multimodaldatagenerativefusionmethodforcomplexsystemhealthconditionestimation AT honglanhuang multimodaldatagenerativefusionmethodforcomplexsystemhealthconditionestimation AT binghe multimodaldatagenerativefusionmethodforcomplexsystemhealthconditionestimation AT weiweiqin multimodaldatagenerativefusionmethodforcomplexsystemhealthconditionestimation AT haizhenzhu multimodaldatagenerativefusionmethodforcomplexsystemhealthconditionestimation |