Multimodal Metaverse Healthcare: A Collaborative Representation and Adaptive Fusion Approach for Generative Artificial-Intelligence-Driven Diagnosis
The metaverse enables immersive virtual healthcare environments, presenting opportunities for enhanced care delivery. A key challenge lies in effectively combining multimodal healthcare data and generative artificial intelligence abilities within metaverse-based healthcare applications, which is a p...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Association for the Advancement of Science (AAAS)
2025-01-01
|
| Series: | Research |
| Online Access: | https://spj.science.org/doi/10.34133/research.0616 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850255657853255680 |
|---|---|
| author | Jianhui Lv Adam Slowik Shalli Rani Byung-Gyu Kim Chien-Ming Chen Saru Kumari Keqin Li Xiaohong Lyu Huamao Jiang |
| author_facet | Jianhui Lv Adam Slowik Shalli Rani Byung-Gyu Kim Chien-Ming Chen Saru Kumari Keqin Li Xiaohong Lyu Huamao Jiang |
| author_sort | Jianhui Lv |
| collection | DOAJ |
| description | The metaverse enables immersive virtual healthcare environments, presenting opportunities for enhanced care delivery. A key challenge lies in effectively combining multimodal healthcare data and generative artificial intelligence abilities within metaverse-based healthcare applications, which is a problem that needs to be addressed. This paper proposes a novel multimodal learning framework for metaverse healthcare, MMLMH, based on collaborative intra- and intersample representation and adaptive fusion. Our framework introduces a collaborative representation learning approach that captures shared and modality-specific features across text, audio, and visual health data. By combining modality-specific and shared encoders with carefully formulated intrasample and intersample collaboration mechanisms, MMLMH achieves superior feature representation for complex health assessments. The framework’s adaptive fusion approach, utilizing attention mechanisms and gated neural networks, demonstrates robust performance across varying noise levels and data quality conditions. Experiments on metaverse healthcare datasets demonstrate MMLMH’s superior performance over baseline methods across multiple evaluation metrics. Longitudinal studies and visualization further illustrate MMLMH’s adaptability to evolving virtual environments and balanced performance across diagnostic accuracy, patient–system interaction efficacy, and data integration complexity. The proposed framework has a unique advantage in that a similar level of performance is maintained across various patient populations and virtual avatars, which could lead to greater personalization of healthcare experiences in the metaverse. MMLMH’s successful functioning in such complicated circumstances suggests that it can combine and process information streams from several sources. They can be successfully utilized in next-generation healthcare delivery through virtual reality. |
| format | Article |
| id | doaj-art-ef3a2fa849d44df386c3d2957ab48f97 |
| institution | OA Journals |
| issn | 2639-5274 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | American Association for the Advancement of Science (AAAS) |
| record_format | Article |
| series | Research |
| spelling | doaj-art-ef3a2fa849d44df386c3d2957ab48f972025-08-20T01:56:49ZengAmerican Association for the Advancement of Science (AAAS)Research2639-52742025-01-01810.34133/research.0616Multimodal Metaverse Healthcare: A Collaborative Representation and Adaptive Fusion Approach for Generative Artificial-Intelligence-Driven DiagnosisJianhui Lv0Adam Slowik1Shalli Rani2Byung-Gyu Kim3Chien-Ming Chen4Saru Kumari5Keqin Li6Xiaohong Lyu7Huamao Jiang8The First Affiliated Hospital of Jinzhou Medical University, Jinzhou 121012, China.Koszalin University of Technology, Koszalin 98701, Poland.Chitkara University, Rajpura, Punjab 140401, India.Sookmyung Women’s University, Seoul, Republic of Korea.Nanjing University of Information Science & Technology, Nanjing, China.Chaudhary Charan Singh University, Meerut, India.State University of New York, New Paltz, NY 12561, USA.The First Affiliated Hospital of Jinzhou Medical University, Jinzhou 121012, China.The First Affiliated Hospital of Jinzhou Medical University, Jinzhou 121012, China.The metaverse enables immersive virtual healthcare environments, presenting opportunities for enhanced care delivery. A key challenge lies in effectively combining multimodal healthcare data and generative artificial intelligence abilities within metaverse-based healthcare applications, which is a problem that needs to be addressed. This paper proposes a novel multimodal learning framework for metaverse healthcare, MMLMH, based on collaborative intra- and intersample representation and adaptive fusion. Our framework introduces a collaborative representation learning approach that captures shared and modality-specific features across text, audio, and visual health data. By combining modality-specific and shared encoders with carefully formulated intrasample and intersample collaboration mechanisms, MMLMH achieves superior feature representation for complex health assessments. The framework’s adaptive fusion approach, utilizing attention mechanisms and gated neural networks, demonstrates robust performance across varying noise levels and data quality conditions. Experiments on metaverse healthcare datasets demonstrate MMLMH’s superior performance over baseline methods across multiple evaluation metrics. Longitudinal studies and visualization further illustrate MMLMH’s adaptability to evolving virtual environments and balanced performance across diagnostic accuracy, patient–system interaction efficacy, and data integration complexity. The proposed framework has a unique advantage in that a similar level of performance is maintained across various patient populations and virtual avatars, which could lead to greater personalization of healthcare experiences in the metaverse. MMLMH’s successful functioning in such complicated circumstances suggests that it can combine and process information streams from several sources. They can be successfully utilized in next-generation healthcare delivery through virtual reality.https://spj.science.org/doi/10.34133/research.0616 |
| spellingShingle | Jianhui Lv Adam Slowik Shalli Rani Byung-Gyu Kim Chien-Ming Chen Saru Kumari Keqin Li Xiaohong Lyu Huamao Jiang Multimodal Metaverse Healthcare: A Collaborative Representation and Adaptive Fusion Approach for Generative Artificial-Intelligence-Driven Diagnosis Research |
| title | Multimodal Metaverse Healthcare: A Collaborative Representation and Adaptive Fusion Approach for Generative Artificial-Intelligence-Driven Diagnosis |
| title_full | Multimodal Metaverse Healthcare: A Collaborative Representation and Adaptive Fusion Approach for Generative Artificial-Intelligence-Driven Diagnosis |
| title_fullStr | Multimodal Metaverse Healthcare: A Collaborative Representation and Adaptive Fusion Approach for Generative Artificial-Intelligence-Driven Diagnosis |
| title_full_unstemmed | Multimodal Metaverse Healthcare: A Collaborative Representation and Adaptive Fusion Approach for Generative Artificial-Intelligence-Driven Diagnosis |
| title_short | Multimodal Metaverse Healthcare: A Collaborative Representation and Adaptive Fusion Approach for Generative Artificial-Intelligence-Driven Diagnosis |
| title_sort | multimodal metaverse healthcare a collaborative representation and adaptive fusion approach for generative artificial intelligence driven diagnosis |
| url | https://spj.science.org/doi/10.34133/research.0616 |
| work_keys_str_mv | AT jianhuilv multimodalmetaversehealthcareacollaborativerepresentationandadaptivefusionapproachforgenerativeartificialintelligencedrivendiagnosis AT adamslowik multimodalmetaversehealthcareacollaborativerepresentationandadaptivefusionapproachforgenerativeartificialintelligencedrivendiagnosis AT shallirani multimodalmetaversehealthcareacollaborativerepresentationandadaptivefusionapproachforgenerativeartificialintelligencedrivendiagnosis AT byunggyukim multimodalmetaversehealthcareacollaborativerepresentationandadaptivefusionapproachforgenerativeartificialintelligencedrivendiagnosis AT chienmingchen multimodalmetaversehealthcareacollaborativerepresentationandadaptivefusionapproachforgenerativeartificialintelligencedrivendiagnosis AT sarukumari multimodalmetaversehealthcareacollaborativerepresentationandadaptivefusionapproachforgenerativeartificialintelligencedrivendiagnosis AT keqinli multimodalmetaversehealthcareacollaborativerepresentationandadaptivefusionapproachforgenerativeartificialintelligencedrivendiagnosis AT xiaohonglyu multimodalmetaversehealthcareacollaborativerepresentationandadaptivefusionapproachforgenerativeartificialintelligencedrivendiagnosis AT huamaojiang multimodalmetaversehealthcareacollaborativerepresentationandadaptivefusionapproachforgenerativeartificialintelligencedrivendiagnosis |