Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST)
Abstract Background Asbestosis, a rare pneumoconiosis marked by diffuse pulmonary fibrosis, arises from prolonged asbestos exposure. Its diagnosis, guided by the Helsinki criteria, relies on exposure history, clinical findings, radiology, and lung function. However, interobserver variability complic...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2025-08-01
|
| Series: | European Radiology Experimental |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s41747-025-00619-5 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849334095625060352 |
|---|---|
| author | Illaa Smesseim Kevin B. W. Groot Lipman Stefano Trebeschi Martijn M. Stuiver Renaud Tissier Jacobus A. Burgers Cornedine J. de Gooijer |
| author_facet | Illaa Smesseim Kevin B. W. Groot Lipman Stefano Trebeschi Martijn M. Stuiver Renaud Tissier Jacobus A. Burgers Cornedine J. de Gooijer |
| author_sort | Illaa Smesseim |
| collection | DOAJ |
| description | Abstract Background Asbestosis, a rare pneumoconiosis marked by diffuse pulmonary fibrosis, arises from prolonged asbestos exposure. Its diagnosis, guided by the Helsinki criteria, relies on exposure history, clinical findings, radiology, and lung function. However, interobserver variability complicates diagnoses and financial compensation. This study prospectively validated the sensitivity of an AI-driven assessment for asbestosis compensation in the Netherlands. Secondary objectives included evaluating specificity, accuracy, predictive values, area under the curve of the receiver operating characteristic (ROC-AUC), area under the precision-recall curve (PR-AUC), and interobserver variability. Materials and methods Between September 2020 and July 2022, 92 adult compensation applicants were assessed using both AI models and pulmonologists’ reviews based on Dutch Health Council criteria. The AI model assigned an asbestosis probability score: negative (< 35), uncertain (35–66), or positive (≥ 66). Uncertain cases underwent additional reviews for a final determination. Results The AI assessment demonstrated sensitivity of 0.86 (95% confidence interval: 0.77–0.95), specificity of 0.85 (0.76–0.97), accuracy of 0.87 (0.79–0.93), ROC-AUC of 0.92 (0.84–0.97), and PR-AUC of 0.95 (0.89–0.99). Despite strong metrics, the sensitivity target of 98% was unmet. Pulmonologist reviews showed moderate to substantial interobserver variability. Conclusion The AI-driven approach demonstrated robust accuracy but insufficient sensitivity for validation. Addressing interobserver variability and incorporating objective fibrosis measurements could enhance future reliability in clinical and compensation settings. Relevance statement The AI-driven assessment for financial compensation of asbestosis showed adequate accuracy but did not meet the required sensitivity for validation. Key Points We prospectively assessed the sensitivity of an AI-driven assessment procedure for financial compensation of asbestosis. The AI-driven asbestosis probability score underperformed across all metrics compared to internal testing. The AI-driven assessment procedure achieved a sensitivity of 0.86 (95% confidence interval: 0.77–0.95). It did not meet the predefined sensitivity target. Graphical Abstract |
| format | Article |
| id | doaj-art-5b48ca20a9e84c198220cf2aed157e6d |
| institution | Kabale University |
| issn | 2509-9280 |
| language | English |
| publishDate | 2025-08-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | European Radiology Experimental |
| spelling | doaj-art-5b48ca20a9e84c198220cf2aed157e6d2025-08-20T03:45:40ZengSpringerOpenEuropean Radiology Experimental2509-92802025-08-01911810.1186/s41747-025-00619-5Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST)Illaa Smesseim0Kevin B. W. Groot Lipman1Stefano Trebeschi2Martijn M. Stuiver3Renaud Tissier4Jacobus A. Burgers5Cornedine J. de Gooijer6Department of Thoracic Oncology, Netherlands Cancer InstituteDepartment of Thoracic Oncology, Netherlands Cancer InstituteGROW School for Oncology and Developmental Biology, Maastricht University Medical CenterDivision of Psychosocial Research and Epidemiology, Netherlands Cancer InstituteDivision of Psychosocial Research and Epidemiology, Netherlands Cancer InstituteDepartment of Thoracic Oncology, Netherlands Cancer InstituteDepartment of Thoracic Oncology, Netherlands Cancer InstituteAbstract Background Asbestosis, a rare pneumoconiosis marked by diffuse pulmonary fibrosis, arises from prolonged asbestos exposure. Its diagnosis, guided by the Helsinki criteria, relies on exposure history, clinical findings, radiology, and lung function. However, interobserver variability complicates diagnoses and financial compensation. This study prospectively validated the sensitivity of an AI-driven assessment for asbestosis compensation in the Netherlands. Secondary objectives included evaluating specificity, accuracy, predictive values, area under the curve of the receiver operating characteristic (ROC-AUC), area under the precision-recall curve (PR-AUC), and interobserver variability. Materials and methods Between September 2020 and July 2022, 92 adult compensation applicants were assessed using both AI models and pulmonologists’ reviews based on Dutch Health Council criteria. The AI model assigned an asbestosis probability score: negative (< 35), uncertain (35–66), or positive (≥ 66). Uncertain cases underwent additional reviews for a final determination. Results The AI assessment demonstrated sensitivity of 0.86 (95% confidence interval: 0.77–0.95), specificity of 0.85 (0.76–0.97), accuracy of 0.87 (0.79–0.93), ROC-AUC of 0.92 (0.84–0.97), and PR-AUC of 0.95 (0.89–0.99). Despite strong metrics, the sensitivity target of 98% was unmet. Pulmonologist reviews showed moderate to substantial interobserver variability. Conclusion The AI-driven approach demonstrated robust accuracy but insufficient sensitivity for validation. Addressing interobserver variability and incorporating objective fibrosis measurements could enhance future reliability in clinical and compensation settings. Relevance statement The AI-driven assessment for financial compensation of asbestosis showed adequate accuracy but did not meet the required sensitivity for validation. Key Points We prospectively assessed the sensitivity of an AI-driven assessment procedure for financial compensation of asbestosis. The AI-driven asbestosis probability score underperformed across all metrics compared to internal testing. The AI-driven assessment procedure achieved a sensitivity of 0.86 (95% confidence interval: 0.77–0.95). It did not meet the predefined sensitivity target. Graphical Abstracthttps://doi.org/10.1186/s41747-025-00619-5Artificial intelligenceAsbestosisLung disease (interstitial)PneumoconiosisPulmonologists |
| spellingShingle | Illaa Smesseim Kevin B. W. Groot Lipman Stefano Trebeschi Martijn M. Stuiver Renaud Tissier Jacobus A. Burgers Cornedine J. de Gooijer Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST) European Radiology Experimental Artificial intelligence Asbestosis Lung disease (interstitial) Pneumoconiosis Pulmonologists |
| title | Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST) |
| title_full | Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST) |
| title_fullStr | Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST) |
| title_full_unstemmed | Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST) |
| title_short | Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST) |
| title_sort | prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis prosbest |
| topic | Artificial intelligence Asbestosis Lung disease (interstitial) Pneumoconiosis Pulmonologists |
| url | https://doi.org/10.1186/s41747-025-00619-5 |
| work_keys_str_mv | AT illaasmesseim prospectivevalidationofanartificialintelligenceassessmentinacohortofapplicantsseekingfinancialcompensationforasbestosisprosbest AT kevinbwgrootlipman prospectivevalidationofanartificialintelligenceassessmentinacohortofapplicantsseekingfinancialcompensationforasbestosisprosbest AT stefanotrebeschi prospectivevalidationofanartificialintelligenceassessmentinacohortofapplicantsseekingfinancialcompensationforasbestosisprosbest AT martijnmstuiver prospectivevalidationofanartificialintelligenceassessmentinacohortofapplicantsseekingfinancialcompensationforasbestosisprosbest AT renaudtissier prospectivevalidationofanartificialintelligenceassessmentinacohortofapplicantsseekingfinancialcompensationforasbestosisprosbest AT jacobusaburgers prospectivevalidationofanartificialintelligenceassessmentinacohortofapplicantsseekingfinancialcompensationforasbestosisprosbest AT cornedinejdegooijer prospectivevalidationofanartificialintelligenceassessmentinacohortofapplicantsseekingfinancialcompensationforasbestosisprosbest |