The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators
Objective:. To compare the performance of the ACS NSQIP “universal” risk calculator (N-RC) to operation-specific RCs. Background:. Resources have been directed toward building operation-specific RCs because of an implicit belief that they would provide more accurate risk estimates than the N-RC. How...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wolters Kluwer Health
2023-12-01
|
| Series: | Annals of Surgery Open |
| Online Access: | http://journals.lww.com/10.1097/AS9.0000000000000358 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832589716569980928 |
|---|---|
| author | Mark E. Cohen, PhD Yaoming Liu, PhD Bruce L. Hall, MD, PhD, MBA, FACS Clifford Y. Ko, MD, MS, MSHS, FACS |
| author_facet | Mark E. Cohen, PhD Yaoming Liu, PhD Bruce L. Hall, MD, PhD, MBA, FACS Clifford Y. Ko, MD, MS, MSHS, FACS |
| author_sort | Mark E. Cohen, PhD |
| collection | DOAJ |
| description | Objective:. To compare the performance of the ACS NSQIP “universal” risk calculator (N-RC) to operation-specific RCs.
Background:. Resources have been directed toward building operation-specific RCs because of an implicit belief that they would provide more accurate risk estimates than the N-RC. However, operation-specific calculators may not provide sufficient improvements in accuracy to justify the costs in development, maintenance, and access.
Methods:. For the N-RC, a cohort of 5,020,713 NSQIP patient records were randomly divided into 80% for machine learning algorithm training and 20% for validation. Operation-specific risk calculators (OS-RC) and OS-RCs with operation-specific predictors (OSP-RC) were independently developed for each of 6 operative groups (colectomy, whipple pancreatectomy, thyroidectomy, abdominal aortic aneurysm (open), hysterectomy/myomectomy, and total knee arthroplasty) and 14 outcomes using the same 80%/20% rule applied to the appropriate subsets of the 5M records. Predictive accuracy was evaluated using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), and Hosmer-Lemeshow (H-L) P values, for 13 binary outcomes, and mean squared error for the length of stay outcome.
Results:. The N-RC was found to have greater AUROC (P = 0.002) and greater AUPRC (P < 0.001) compared to the OS-RC. No other statistically significant differences in accuracy, across the 3 risk calculator types, were found. There was an inverse relationship between the operation group sample size and magnitude of the difference in AUROC (r = −0.278; P = 0.014) and in AUPRC (r = −0.425; P < 0.001) between N-RC and OS-RC. The smaller the sample size, the greater the superiority of the N-RC.
Conclusions:. While operation-specific RCs might be assumed to have advantages over a universal RC, their reliance on smaller datasets may reduce their ability to accurately estimate predictor effects. In the present study, this tradeoff between operation specificity and accuracy, in estimating the effects of predictor variables, favors the N-R, though the clinical impact is likely to be negligible. |
| format | Article |
| id | doaj-art-9f37d7cc8e3d4867933d698d0b708a3c |
| institution | Kabale University |
| issn | 2691-3593 |
| language | English |
| publishDate | 2023-12-01 |
| publisher | Wolters Kluwer Health |
| record_format | Article |
| series | Annals of Surgery Open |
| spelling | doaj-art-9f37d7cc8e3d4867933d698d0b708a3c2025-01-24T09:18:12ZengWolters Kluwer HealthAnnals of Surgery Open2691-35932023-12-0144e35810.1097/AS9.0000000000000358202312000-00023The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific CalculatorsMark E. Cohen, PhD0Yaoming Liu, PhD1Bruce L. Hall, MD, PhD, MBA, FACS2Clifford Y. Ko, MD, MS, MSHS, FACS3From the * Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, ILFrom the * Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, ILFrom the * Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, ILFrom the * Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, ILObjective:. To compare the performance of the ACS NSQIP “universal” risk calculator (N-RC) to operation-specific RCs. Background:. Resources have been directed toward building operation-specific RCs because of an implicit belief that they would provide more accurate risk estimates than the N-RC. However, operation-specific calculators may not provide sufficient improvements in accuracy to justify the costs in development, maintenance, and access. Methods:. For the N-RC, a cohort of 5,020,713 NSQIP patient records were randomly divided into 80% for machine learning algorithm training and 20% for validation. Operation-specific risk calculators (OS-RC) and OS-RCs with operation-specific predictors (OSP-RC) were independently developed for each of 6 operative groups (colectomy, whipple pancreatectomy, thyroidectomy, abdominal aortic aneurysm (open), hysterectomy/myomectomy, and total knee arthroplasty) and 14 outcomes using the same 80%/20% rule applied to the appropriate subsets of the 5M records. Predictive accuracy was evaluated using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), and Hosmer-Lemeshow (H-L) P values, for 13 binary outcomes, and mean squared error for the length of stay outcome. Results:. The N-RC was found to have greater AUROC (P = 0.002) and greater AUPRC (P < 0.001) compared to the OS-RC. No other statistically significant differences in accuracy, across the 3 risk calculator types, were found. There was an inverse relationship between the operation group sample size and magnitude of the difference in AUROC (r = −0.278; P = 0.014) and in AUPRC (r = −0.425; P < 0.001) between N-RC and OS-RC. The smaller the sample size, the greater the superiority of the N-RC. Conclusions:. While operation-specific RCs might be assumed to have advantages over a universal RC, their reliance on smaller datasets may reduce their ability to accurately estimate predictor effects. In the present study, this tradeoff between operation specificity and accuracy, in estimating the effects of predictor variables, favors the N-R, though the clinical impact is likely to be negligible.http://journals.lww.com/10.1097/AS9.0000000000000358 |
| spellingShingle | Mark E. Cohen, PhD Yaoming Liu, PhD Bruce L. Hall, MD, PhD, MBA, FACS Clifford Y. Ko, MD, MS, MSHS, FACS The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators Annals of Surgery Open |
| title | The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators |
| title_full | The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators |
| title_fullStr | The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators |
| title_full_unstemmed | The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators |
| title_short | The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators |
| title_sort | accuracy of the nsqip universal surgical risk calculator compared to operation specific calculators |
| url | http://journals.lww.com/10.1097/AS9.0000000000000358 |
| work_keys_str_mv | AT markecohenphd theaccuracyofthensqipuniversalsurgicalriskcalculatorcomparedtooperationspecificcalculators AT yaomingliuphd theaccuracyofthensqipuniversalsurgicalriskcalculatorcomparedtooperationspecificcalculators AT brucelhallmdphdmbafacs theaccuracyofthensqipuniversalsurgicalriskcalculatorcomparedtooperationspecificcalculators AT cliffordykomdmsmshsfacs theaccuracyofthensqipuniversalsurgicalriskcalculatorcomparedtooperationspecificcalculators AT markecohenphd accuracyofthensqipuniversalsurgicalriskcalculatorcomparedtooperationspecificcalculators AT yaomingliuphd accuracyofthensqipuniversalsurgicalriskcalculatorcomparedtooperationspecificcalculators AT brucelhallmdphdmbafacs accuracyofthensqipuniversalsurgicalriskcalculatorcomparedtooperationspecificcalculators AT cliffordykomdmsmshsfacs accuracyofthensqipuniversalsurgicalriskcalculatorcomparedtooperationspecificcalculators |