Evolution of an Artificial Intelligence-Powered Application for Mammography
<b>Background:</b> The implementation of radiological artificial intelligence (AI) solutions remains challenging due to limitations in existing testing methodologies. This study assesses the efficacy of a comprehensive methodology for performance testing and monitoring of commercial-grad...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-03-01
|
| Series: | Diagnostics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2075-4418/15/7/822 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849730622244782080 |
|---|---|
| author | Yuriy Vasilev Denis Rumyantsev Anton Vladzymyrskyy Olga Omelyanskaya Lev Pestrenin Igor Shulkin Evgeniy Nikitin Artem Kapninskiy Kirill Arzamasov |
| author_facet | Yuriy Vasilev Denis Rumyantsev Anton Vladzymyrskyy Olga Omelyanskaya Lev Pestrenin Igor Shulkin Evgeniy Nikitin Artem Kapninskiy Kirill Arzamasov |
| author_sort | Yuriy Vasilev |
| collection | DOAJ |
| description | <b>Background:</b> The implementation of radiological artificial intelligence (AI) solutions remains challenging due to limitations in existing testing methodologies. This study assesses the efficacy of a comprehensive methodology for performance testing and monitoring of commercial-grade mammographic AI models. <b>Methods:</b> We utilized a combination of retrospective and prospective multicenter approaches to evaluate a neural network based on the Faster R-CNN architecture with a ResNet-50 backbone, trained on a dataset of 3641 mammograms. The methodology encompassed functional and calibration testing, coupled with routine technical and clinical monitoring. Feedback from testers and radiologists was relayed to the developers, who made updates to the AI model. The test dataset comprised 112 medical organizations, representing 10 manufacturers of mammography equipment and encompassing 593,365 studies. The evaluation metrics included the area under the curve (AUC), accuracy, sensitivity, specificity, technical defects, and clinical assessment scores. <b>Results:</b> The results demonstrated significant enhancement in the AI model’s performance through collaborative efforts among developers, testers, and radiologists. Notable improvements included functionality, diagnostic accuracy, and technical stability. Specifically, the AUC rose by 24.7% (from 0.73 to 0.91), the accuracy improved by 15.6% (from 0.77 to 0.89), sensitivity grew by 37.1% (from 0.62 to 0.85), and specificity increased by 10.7% (from 0.84 to 0.93). The average proportion of technical defects declined from 9.0% to 1.0%, while the clinical assessment score improved from 63.4 to 72.0. Following 2 years and 9 months of testing, the AI solution was integrated into the compulsory health insurance system. <b>Conclusions:</b> The multi-stage, lifecycle-based testing methodology demonstrated substantial potential in software enhancement and integration into clinical practice. Key elements of this methodology include robust functional and diagnostic requirements, continuous testing and updates, systematic feedback collection from testers and radiologists, and prospective monitoring. |
| format | Article |
| id | doaj-art-a8ad1c66ccc446e99a9ad307d0cbd7bb |
| institution | DOAJ |
| issn | 2075-4418 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Diagnostics |
| spelling | doaj-art-a8ad1c66ccc446e99a9ad307d0cbd7bb2025-08-20T03:08:48ZengMDPI AGDiagnostics2075-44182025-03-0115782210.3390/diagnostics15070822Evolution of an Artificial Intelligence-Powered Application for MammographyYuriy Vasilev0Denis Rumyantsev1Anton Vladzymyrskyy2Olga Omelyanskaya3Lev Pestrenin4Igor Shulkin5Evgeniy Nikitin6Artem Kapninskiy7Kirill Arzamasov8Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, RussiaResearch and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, RussiaResearch and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, RussiaResearch and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, RussiaResearch and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, RussiaResearch and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, RussiaCelsus (Medical Screening Systems), Viktorenko St., Bldg. 11, Room 21N, 125167 Moscow, RussiaCelsus (Medical Screening Systems), Viktorenko St., Bldg. 11, Room 21N, 125167 Moscow, RussiaResearch and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia<b>Background:</b> The implementation of radiological artificial intelligence (AI) solutions remains challenging due to limitations in existing testing methodologies. This study assesses the efficacy of a comprehensive methodology for performance testing and monitoring of commercial-grade mammographic AI models. <b>Methods:</b> We utilized a combination of retrospective and prospective multicenter approaches to evaluate a neural network based on the Faster R-CNN architecture with a ResNet-50 backbone, trained on a dataset of 3641 mammograms. The methodology encompassed functional and calibration testing, coupled with routine technical and clinical monitoring. Feedback from testers and radiologists was relayed to the developers, who made updates to the AI model. The test dataset comprised 112 medical organizations, representing 10 manufacturers of mammography equipment and encompassing 593,365 studies. The evaluation metrics included the area under the curve (AUC), accuracy, sensitivity, specificity, technical defects, and clinical assessment scores. <b>Results:</b> The results demonstrated significant enhancement in the AI model’s performance through collaborative efforts among developers, testers, and radiologists. Notable improvements included functionality, diagnostic accuracy, and technical stability. Specifically, the AUC rose by 24.7% (from 0.73 to 0.91), the accuracy improved by 15.6% (from 0.77 to 0.89), sensitivity grew by 37.1% (from 0.62 to 0.85), and specificity increased by 10.7% (from 0.84 to 0.93). The average proportion of technical defects declined from 9.0% to 1.0%, while the clinical assessment score improved from 63.4 to 72.0. Following 2 years and 9 months of testing, the AI solution was integrated into the compulsory health insurance system. <b>Conclusions:</b> The multi-stage, lifecycle-based testing methodology demonstrated substantial potential in software enhancement and integration into clinical practice. Key elements of this methodology include robust functional and diagnostic requirements, continuous testing and updates, systematic feedback collection from testers and radiologists, and prospective monitoring.https://www.mdpi.com/2075-4418/15/7/822artificial intelligenceradiologymammographysoftwaresoftware validation |
| spellingShingle | Yuriy Vasilev Denis Rumyantsev Anton Vladzymyrskyy Olga Omelyanskaya Lev Pestrenin Igor Shulkin Evgeniy Nikitin Artem Kapninskiy Kirill Arzamasov Evolution of an Artificial Intelligence-Powered Application for Mammography Diagnostics artificial intelligence radiology mammography software software validation |
| title | Evolution of an Artificial Intelligence-Powered Application for Mammography |
| title_full | Evolution of an Artificial Intelligence-Powered Application for Mammography |
| title_fullStr | Evolution of an Artificial Intelligence-Powered Application for Mammography |
| title_full_unstemmed | Evolution of an Artificial Intelligence-Powered Application for Mammography |
| title_short | Evolution of an Artificial Intelligence-Powered Application for Mammography |
| title_sort | evolution of an artificial intelligence powered application for mammography |
| topic | artificial intelligence radiology mammography software software validation |
| url | https://www.mdpi.com/2075-4418/15/7/822 |
| work_keys_str_mv | AT yuriyvasilev evolutionofanartificialintelligencepoweredapplicationformammography AT denisrumyantsev evolutionofanartificialintelligencepoweredapplicationformammography AT antonvladzymyrskyy evolutionofanartificialintelligencepoweredapplicationformammography AT olgaomelyanskaya evolutionofanartificialintelligencepoweredapplicationformammography AT levpestrenin evolutionofanartificialintelligencepoweredapplicationformammography AT igorshulkin evolutionofanartificialintelligencepoweredapplicationformammography AT evgeniynikitin evolutionofanartificialintelligencepoweredapplicationformammography AT artemkapninskiy evolutionofanartificialintelligencepoweredapplicationformammography AT kirillarzamasov evolutionofanartificialintelligencepoweredapplicationformammography |