A Comprehensive Drift-Adaptive Framework for Sustaining Model Performance in COVID-19 Detection From Dynamic Cough Audio Data: Model Development and Validation
BackgroundThe COVID-19 pandemic has highlighted the need for robust and adaptable diagnostic tools capable of detecting the disease from diverse and continuously evolving data sources. Machine learning models, particularly convolutional neural networks, are promising in this...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-06-01
|
| Series: | Journal of Medical Internet Research |
| Online Access: | https://www.jmir.org/2025/1/e66919 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849735524786372608 |
|---|---|
| author | Theofanis Ganitidis Maria Athanasiou Konstantinos Mitsis Konstantia Zarkogianni Konstantina S Nikita |
| author_facet | Theofanis Ganitidis Maria Athanasiou Konstantinos Mitsis Konstantia Zarkogianni Konstantina S Nikita |
| author_sort | Theofanis Ganitidis |
| collection | DOAJ |
| description |
BackgroundThe COVID-19 pandemic has highlighted the need for robust and adaptable diagnostic tools capable of detecting the disease from diverse and continuously evolving data sources. Machine learning models, particularly convolutional neural networks, are promising in this regard. However, the dynamic nature of real-world data can lead to model drift, where the model’s performance degrades over time, as the underlying data distribution changes due to evolving disease characteristics, demographic shifts, and variations in recording conditions. Addressing this challenge is crucial to maintaining the accuracy and reliability of these models in ongoing diagnostic applications.
ObjectiveThis study aims to develop a comprehensive framework that not only monitors model drift over time but also uses adaptation mechanisms to mitigate performance fluctuations in COVID-19 detection models trained on dynamic cough audio data.
MethodsTwo crowdsourced COVID-19 audio datasets, namely COVID-19 Sounds and Coswara, were used for development and evaluation purposes. Each dataset was divided into 2 distinct periods, namely the development period and postdevelopment period. A baseline convolutional neural network model was initially trained and evaluated using data (ie, coughs from COVID-19 Sounds and shallow coughs from Coswara dataset) from the development period. To detect changes in data distributions and the model’s performance between these periods, the maximum mean discrepancy distance was used. Upon detecting significant drift, a retraining procedure was triggered to update the baseline model. The study explored 2 model adaptation approaches, unsupervised domain adaptation and active learning, both of which were comparatively assessed.
ResultsThe baseline model achieved an area under the receiver operating characteristic curve of 69.13% and a balanced accuracy of 63.38% on the development test set of the COVID-19 Sounds dataset, while for the Coswara dataset, the corresponding values were 66.8% and 61.64%. A decline in performance was observed when the model was evaluated on data from the postdevelopment period, indicating the presence of model drift. The application of the unsupervised domain adaptation approach led to performance improvement in terms of balanced accuracy by up to 22% and 24% for the COVID-19 Sounds and Coswara datasets, respectively. The active learning approach yielded even greater improvement, corresponding to a balanced accuracy increase of up to 30% and 60% for the 2 datasets, respectively.
ConclusionsThe proposed framework successfully addresses the challenge of model drift in COVID-19 detection by enabling continuous adaptation to evolving data distributions. This approach ensures sustained model performance over time, contributing to the development of robust and adaptable diagnostic tools for COVID-19 and potentially other infectious diseases. |
| format | Article |
| id | doaj-art-6d3ad1ae41314a7d9cdfe657bde05284 |
| institution | DOAJ |
| issn | 1438-8871 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | Journal of Medical Internet Research |
| spelling | doaj-art-6d3ad1ae41314a7d9cdfe657bde052842025-08-20T03:07:32ZengJMIR PublicationsJournal of Medical Internet Research1438-88712025-06-0127e6691910.2196/66919A Comprehensive Drift-Adaptive Framework for Sustaining Model Performance in COVID-19 Detection From Dynamic Cough Audio Data: Model Development and ValidationTheofanis Ganitidishttps://orcid.org/0009-0006-7794-9793Maria Athanasiouhttps://orcid.org/0000-0003-1575-9100Konstantinos Mitsishttps://orcid.org/0000-0002-4629-2163Konstantia Zarkogiannihttps://orcid.org/0000-0003-3886-1618Konstantina S Nikitahttps://orcid.org/0000-0001-8255-4354 BackgroundThe COVID-19 pandemic has highlighted the need for robust and adaptable diagnostic tools capable of detecting the disease from diverse and continuously evolving data sources. Machine learning models, particularly convolutional neural networks, are promising in this regard. However, the dynamic nature of real-world data can lead to model drift, where the model’s performance degrades over time, as the underlying data distribution changes due to evolving disease characteristics, demographic shifts, and variations in recording conditions. Addressing this challenge is crucial to maintaining the accuracy and reliability of these models in ongoing diagnostic applications. ObjectiveThis study aims to develop a comprehensive framework that not only monitors model drift over time but also uses adaptation mechanisms to mitigate performance fluctuations in COVID-19 detection models trained on dynamic cough audio data. MethodsTwo crowdsourced COVID-19 audio datasets, namely COVID-19 Sounds and Coswara, were used for development and evaluation purposes. Each dataset was divided into 2 distinct periods, namely the development period and postdevelopment period. A baseline convolutional neural network model was initially trained and evaluated using data (ie, coughs from COVID-19 Sounds and shallow coughs from Coswara dataset) from the development period. To detect changes in data distributions and the model’s performance between these periods, the maximum mean discrepancy distance was used. Upon detecting significant drift, a retraining procedure was triggered to update the baseline model. The study explored 2 model adaptation approaches, unsupervised domain adaptation and active learning, both of which were comparatively assessed. ResultsThe baseline model achieved an area under the receiver operating characteristic curve of 69.13% and a balanced accuracy of 63.38% on the development test set of the COVID-19 Sounds dataset, while for the Coswara dataset, the corresponding values were 66.8% and 61.64%. A decline in performance was observed when the model was evaluated on data from the postdevelopment period, indicating the presence of model drift. The application of the unsupervised domain adaptation approach led to performance improvement in terms of balanced accuracy by up to 22% and 24% for the COVID-19 Sounds and Coswara datasets, respectively. The active learning approach yielded even greater improvement, corresponding to a balanced accuracy increase of up to 30% and 60% for the 2 datasets, respectively. ConclusionsThe proposed framework successfully addresses the challenge of model drift in COVID-19 detection by enabling continuous adaptation to evolving data distributions. This approach ensures sustained model performance over time, contributing to the development of robust and adaptable diagnostic tools for COVID-19 and potentially other infectious diseases.https://www.jmir.org/2025/1/e66919 |
| spellingShingle | Theofanis Ganitidis Maria Athanasiou Konstantinos Mitsis Konstantia Zarkogianni Konstantina S Nikita A Comprehensive Drift-Adaptive Framework for Sustaining Model Performance in COVID-19 Detection From Dynamic Cough Audio Data: Model Development and Validation Journal of Medical Internet Research |
| title | A Comprehensive Drift-Adaptive Framework for Sustaining Model Performance in COVID-19 Detection From Dynamic Cough Audio Data: Model Development and Validation |
| title_full | A Comprehensive Drift-Adaptive Framework for Sustaining Model Performance in COVID-19 Detection From Dynamic Cough Audio Data: Model Development and Validation |
| title_fullStr | A Comprehensive Drift-Adaptive Framework for Sustaining Model Performance in COVID-19 Detection From Dynamic Cough Audio Data: Model Development and Validation |
| title_full_unstemmed | A Comprehensive Drift-Adaptive Framework for Sustaining Model Performance in COVID-19 Detection From Dynamic Cough Audio Data: Model Development and Validation |
| title_short | A Comprehensive Drift-Adaptive Framework for Sustaining Model Performance in COVID-19 Detection From Dynamic Cough Audio Data: Model Development and Validation |
| title_sort | comprehensive drift adaptive framework for sustaining model performance in covid 19 detection from dynamic cough audio data model development and validation |
| url | https://www.jmir.org/2025/1/e66919 |
| work_keys_str_mv | AT theofanisganitidis acomprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation AT mariaathanasiou acomprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation AT konstantinosmitsis acomprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation AT konstantiazarkogianni acomprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation AT konstantinasnikita acomprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation AT theofanisganitidis comprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation AT mariaathanasiou comprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation AT konstantinosmitsis comprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation AT konstantiazarkogianni comprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation AT konstantinasnikita comprehensivedriftadaptiveframeworkforsustainingmodelperformanceincovid19detectionfromdynamiccoughaudiodatamodeldevelopmentandvalidation |