Knee Osteoarthritis Diagnosis With Unimodal and Multi-Modal Neural Networks: Data From the Osteoarthritis Initiative
Knee osteoarthritis (OA) is a prevalent musculoskeletal condition affecting millions worldwide, posing significant health and economic burdens. Characterized by the degeneration of joint cartilage, the progression of knee OA varies significantly among individuals, making its prediction a complex iss...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10704620/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850283076040523776 |
|---|---|
| author | Xin Yu Teh Pauline Shan Qing Yeoh Tao Wang Xiang Wu Khairunnisa Hasikin Khin Wee Lai |
| author_facet | Xin Yu Teh Pauline Shan Qing Yeoh Tao Wang Xiang Wu Khairunnisa Hasikin Khin Wee Lai |
| author_sort | Xin Yu Teh |
| collection | DOAJ |
| description | Knee osteoarthritis (OA) is a prevalent musculoskeletal condition affecting millions worldwide, posing significant health and economic burdens. Characterized by the degeneration of joint cartilage, the progression of knee OA varies significantly among individuals, making its prediction a complex issue. Previous studies on automated knee OA diagnosis have primarily relied on unimodal data, often overlooking the valuable information present in multi-modal data. Multi-modal learning, which integrates information from various modalities, is increasingly recognized for its potential to enhance diagnostic performance in medical applications. However, such models incur a higher computational load due to the additional data required. This research investigates the feasibility of multi-modal neural networks in knee OA diagnosis by integrating structural demographic data with unstructured imaging data. Three deep learning unimodal models (InceptionV3, DIKO, and EfficientNetv2) were transformed into multi-modal architectures (MF_InceptionNet, MF_DIKO, and MF_Eff) to compare their diagnostic capabilities. The proposed multi-modal models share a common architecture, with unimodal models acting as image feature extraction backbones and separate embedding layers for demographic data. The image features and demographic embeddings are combined into a unified vector before classification. Extensive experiments were conducted to evaluate the performance of these models across different class categories and dataset sizes. MF_DIKO and InceptionV3 emerged as the best multi-modal and unimodal neural networks, respectively, with overall accuracies of 0.67 and 0.75 for 3-class severity classification. Contrary to existing literature, our findings reveal that unimodal neural networks using only imaging features outperform multi-modal networks, suggesting unimodal models might suffice in certain applications. |
| format | Article |
| id | doaj-art-0cd12dcfc98949b7839c8e983f3a5cca |
| institution | OA Journals |
| issn | 2169-3536 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-0cd12dcfc98949b7839c8e983f3a5cca2025-08-20T01:47:50ZengIEEEIEEE Access2169-35362024-01-011214669814671710.1109/ACCESS.2024.347265410704620Knee Osteoarthritis Diagnosis With Unimodal and Multi-Modal Neural Networks: Data From the Osteoarthritis InitiativeXin Yu Teh0https://orcid.org/0009-0009-3362-3582Pauline Shan Qing Yeoh1https://orcid.org/0000-0003-3643-4479Tao Wang2https://orcid.org/0009-0009-2786-9869Xiang Wu3https://orcid.org/0000-0001-5190-9781Khairunnisa Hasikin4https://orcid.org/0000-0002-0471-3820Khin Wee Lai5https://orcid.org/0000-0002-8602-0533Department of Biomedical Engineering, Universiti Malaya, Kuala Lumpur, MalaysiaDepartment of Biomedical Engineering, Universiti Malaya, Kuala Lumpur, MalaysiaBlood Center, Lianyungang City, Jiangsu, ChinaDepartment of Biomedical Engineering, Universiti Malaya, Kuala Lumpur, MalaysiaDepartment of Biomedical Engineering, Universiti Malaya, Kuala Lumpur, MalaysiaDepartment of Biomedical Engineering, Universiti Malaya, Kuala Lumpur, MalaysiaKnee osteoarthritis (OA) is a prevalent musculoskeletal condition affecting millions worldwide, posing significant health and economic burdens. Characterized by the degeneration of joint cartilage, the progression of knee OA varies significantly among individuals, making its prediction a complex issue. Previous studies on automated knee OA diagnosis have primarily relied on unimodal data, often overlooking the valuable information present in multi-modal data. Multi-modal learning, which integrates information from various modalities, is increasingly recognized for its potential to enhance diagnostic performance in medical applications. However, such models incur a higher computational load due to the additional data required. This research investigates the feasibility of multi-modal neural networks in knee OA diagnosis by integrating structural demographic data with unstructured imaging data. Three deep learning unimodal models (InceptionV3, DIKO, and EfficientNetv2) were transformed into multi-modal architectures (MF_InceptionNet, MF_DIKO, and MF_Eff) to compare their diagnostic capabilities. The proposed multi-modal models share a common architecture, with unimodal models acting as image feature extraction backbones and separate embedding layers for demographic data. The image features and demographic embeddings are combined into a unified vector before classification. Extensive experiments were conducted to evaluate the performance of these models across different class categories and dataset sizes. MF_DIKO and InceptionV3 emerged as the best multi-modal and unimodal neural networks, respectively, with overall accuracies of 0.67 and 0.75 for 3-class severity classification. Contrary to existing literature, our findings reveal that unimodal neural networks using only imaging features outperform multi-modal networks, suggesting unimodal models might suffice in certain applications.https://ieeexplore.ieee.org/document/10704620/Knee osteoarthritisdeep learningX-raymulti-modal fusionclassification |
| spellingShingle | Xin Yu Teh Pauline Shan Qing Yeoh Tao Wang Xiang Wu Khairunnisa Hasikin Khin Wee Lai Knee Osteoarthritis Diagnosis With Unimodal and Multi-Modal Neural Networks: Data From the Osteoarthritis Initiative IEEE Access Knee osteoarthritis deep learning X-ray multi-modal fusion classification |
| title | Knee Osteoarthritis Diagnosis With Unimodal and Multi-Modal Neural Networks: Data From the Osteoarthritis Initiative |
| title_full | Knee Osteoarthritis Diagnosis With Unimodal and Multi-Modal Neural Networks: Data From the Osteoarthritis Initiative |
| title_fullStr | Knee Osteoarthritis Diagnosis With Unimodal and Multi-Modal Neural Networks: Data From the Osteoarthritis Initiative |
| title_full_unstemmed | Knee Osteoarthritis Diagnosis With Unimodal and Multi-Modal Neural Networks: Data From the Osteoarthritis Initiative |
| title_short | Knee Osteoarthritis Diagnosis With Unimodal and Multi-Modal Neural Networks: Data From the Osteoarthritis Initiative |
| title_sort | knee osteoarthritis diagnosis with unimodal and multi modal neural networks data from the osteoarthritis initiative |
| topic | Knee osteoarthritis deep learning X-ray multi-modal fusion classification |
| url | https://ieeexplore.ieee.org/document/10704620/ |
| work_keys_str_mv | AT xinyuteh kneeosteoarthritisdiagnosiswithunimodalandmultimodalneuralnetworksdatafromtheosteoarthritisinitiative AT paulineshanqingyeoh kneeosteoarthritisdiagnosiswithunimodalandmultimodalneuralnetworksdatafromtheosteoarthritisinitiative AT taowang kneeosteoarthritisdiagnosiswithunimodalandmultimodalneuralnetworksdatafromtheosteoarthritisinitiative AT xiangwu kneeosteoarthritisdiagnosiswithunimodalandmultimodalneuralnetworksdatafromtheosteoarthritisinitiative AT khairunnisahasikin kneeosteoarthritisdiagnosiswithunimodalandmultimodalneuralnetworksdatafromtheosteoarthritisinitiative AT khinweelai kneeosteoarthritisdiagnosiswithunimodalandmultimodalneuralnetworksdatafromtheosteoarthritisinitiative |