Development, validation and recalibration of a prediction model for prediabetes: an EHR and NHANES-based study
Abstract Background A prediction model that estimates the risk of elevated glycated hemoglobin (HbA1c) was developed from electronic health record (EHR) data to identify adult patients at risk for prediabetes who may otherwise go undetected. We aimed to assess the internal performance of a new penal...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-12-01
|
| Series: | BMC Medical Informatics and Decision Making |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12911-024-02803-w |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850244648888434688 |
|---|---|
| author | Nicholas J. Casacchia Kristin M. Lenoir Joseph Rigdon Brian J. Wells |
| author_facet | Nicholas J. Casacchia Kristin M. Lenoir Joseph Rigdon Brian J. Wells |
| author_sort | Nicholas J. Casacchia |
| collection | DOAJ |
| description | Abstract Background A prediction model that estimates the risk of elevated glycated hemoglobin (HbA1c) was developed from electronic health record (EHR) data to identify adult patients at risk for prediabetes who may otherwise go undetected. We aimed to assess the internal performance of a new penalized regression model using the same EHR data and compare it to the previously developed stepdown approximation for predicting HbA1c ≥ 5.7%, the cut-off for prediabetes. Additionally, we sought to externally validate and recalibrate the approximation model using 2017–2020 pre-pandemic National Health and Nutrition Examination Survey (NHANES) data. Methods We developed logistic regression models using EHR data through two approaches: the Least Absolute Shrinkage and Selection Operator (LASSO) and stepdown approximation. Internal validation was performed using the bootstrap method, with internal performance evaluated by the Brier score, C-statistic, calibration intercept and slope, and the integrated calibration index. We externally validated the approximation model by applying original model coefficients to NHANES, and we examined the approximation model’s performance after recalibration in NHANES. Results The EHR cohort included 22,635 patients, with 26% identified as having prediabetes. Both the LASSO and approximation models demonstrated similar discrimination in the EHR cohort, with optimism-corrected C-statistics of 0.760 and 0.763, respectively. The LASSO model included 23 predictor variables, while the approximation model contained 8. Among the 2,348 NHANES participants who met the inclusion criteria, 30.1% had prediabetes. External validation of the LASSO model was not possible due to the unavailability of some predictor variables. The approximation model discriminated well in the NHANES dataset, achieving a C-statistic of 0.787. Conclusion The approximation method demonstrated comparable performance to LASSO in the EHR development cohort, making it a viable option for healthcare organizations with limited resources to collect a comprehensive set of candidate predictor variables. NHANES data may be suitable for externally validating a clinical prediction model developed with EHR data to assess generalizability to a nationally representative sample, depending on the model’s intended use and the alignment of predictor variable definitions with those used in the model’s original development. |
| format | Article |
| id | doaj-art-58dc3dc0b74b4d9f8eb910a2b1cc9ce1 |
| institution | OA Journals |
| issn | 1472-6947 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Informatics and Decision Making |
| spelling | doaj-art-58dc3dc0b74b4d9f8eb910a2b1cc9ce12025-08-20T01:59:40ZengBMCBMC Medical Informatics and Decision Making1472-69472024-12-0124111110.1186/s12911-024-02803-wDevelopment, validation and recalibration of a prediction model for prediabetes: an EHR and NHANES-based studyNicholas J. Casacchia0Kristin M. Lenoir1Joseph Rigdon2Brian J. Wells3Center for Value-Based Care Research, Primary Care Institute, Cleveland ClinicDivision of Public Health Sciences, Department of Biostatistics and Data Science, Wake Forest University School of MedicineDivision of Public Health Sciences, Department of Biostatistics and Data Science, Wake Forest University School of MedicineDivision of Public Health Sciences, Department of Biostatistics and Data Science, Wake Forest University School of MedicineAbstract Background A prediction model that estimates the risk of elevated glycated hemoglobin (HbA1c) was developed from electronic health record (EHR) data to identify adult patients at risk for prediabetes who may otherwise go undetected. We aimed to assess the internal performance of a new penalized regression model using the same EHR data and compare it to the previously developed stepdown approximation for predicting HbA1c ≥ 5.7%, the cut-off for prediabetes. Additionally, we sought to externally validate and recalibrate the approximation model using 2017–2020 pre-pandemic National Health and Nutrition Examination Survey (NHANES) data. Methods We developed logistic regression models using EHR data through two approaches: the Least Absolute Shrinkage and Selection Operator (LASSO) and stepdown approximation. Internal validation was performed using the bootstrap method, with internal performance evaluated by the Brier score, C-statistic, calibration intercept and slope, and the integrated calibration index. We externally validated the approximation model by applying original model coefficients to NHANES, and we examined the approximation model’s performance after recalibration in NHANES. Results The EHR cohort included 22,635 patients, with 26% identified as having prediabetes. Both the LASSO and approximation models demonstrated similar discrimination in the EHR cohort, with optimism-corrected C-statistics of 0.760 and 0.763, respectively. The LASSO model included 23 predictor variables, while the approximation model contained 8. Among the 2,348 NHANES participants who met the inclusion criteria, 30.1% had prediabetes. External validation of the LASSO model was not possible due to the unavailability of some predictor variables. The approximation model discriminated well in the NHANES dataset, achieving a C-statistic of 0.787. Conclusion The approximation method demonstrated comparable performance to LASSO in the EHR development cohort, making it a viable option for healthcare organizations with limited resources to collect a comprehensive set of candidate predictor variables. NHANES data may be suitable for externally validating a clinical prediction model developed with EHR data to assess generalizability to a nationally representative sample, depending on the model’s intended use and the alignment of predictor variable definitions with those used in the model’s original development.https://doi.org/10.1186/s12911-024-02803-wElectronic health recordsNHANESPrediabetesPrediction modelLogistic regressionLASSO |
| spellingShingle | Nicholas J. Casacchia Kristin M. Lenoir Joseph Rigdon Brian J. Wells Development, validation and recalibration of a prediction model for prediabetes: an EHR and NHANES-based study BMC Medical Informatics and Decision Making Electronic health records NHANES Prediabetes Prediction model Logistic regression LASSO |
| title | Development, validation and recalibration of a prediction model for prediabetes: an EHR and NHANES-based study |
| title_full | Development, validation and recalibration of a prediction model for prediabetes: an EHR and NHANES-based study |
| title_fullStr | Development, validation and recalibration of a prediction model for prediabetes: an EHR and NHANES-based study |
| title_full_unstemmed | Development, validation and recalibration of a prediction model for prediabetes: an EHR and NHANES-based study |
| title_short | Development, validation and recalibration of a prediction model for prediabetes: an EHR and NHANES-based study |
| title_sort | development validation and recalibration of a prediction model for prediabetes an ehr and nhanes based study |
| topic | Electronic health records NHANES Prediabetes Prediction model Logistic regression LASSO |
| url | https://doi.org/10.1186/s12911-024-02803-w |
| work_keys_str_mv | AT nicholasjcasacchia developmentvalidationandrecalibrationofapredictionmodelforprediabetesanehrandnhanesbasedstudy AT kristinmlenoir developmentvalidationandrecalibrationofapredictionmodelforprediabetesanehrandnhanesbasedstudy AT josephrigdon developmentvalidationandrecalibrationofapredictionmodelforprediabetesanehrandnhanesbasedstudy AT brianjwells developmentvalidationandrecalibrationofapredictionmodelforprediabetesanehrandnhanesbasedstudy |