Assessing the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies
Abstract Background Utilization of large-volume clinical registries for observational research has gained popularity in orthopaedic literature. However, concerns exist regarding inadequate reporting of methodology in this type of research. Despite these concerns, the reproducibility of such studies...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-03-01
|
| Series: | Journal of Orthopaedic Surgery and Research |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s13018-025-05538-0 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850026084187242496 |
|---|---|
| author | Ayobami S. Ogunsola Michael C. Marinier Arman C. Hlas Jacob M. Elkins |
| author_facet | Ayobami S. Ogunsola Michael C. Marinier Arman C. Hlas Jacob M. Elkins |
| author_sort | Ayobami S. Ogunsola |
| collection | DOAJ |
| description | Abstract Background Utilization of large-volume clinical registries for observational research has gained popularity in orthopaedic literature. However, concerns exist regarding inadequate reporting of methodology in this type of research. Despite these concerns, the reproducibility of such studies has not been adequately assessed in existing literature. This study aims to assess the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies on smoking as a risk factor for poor surgical outcomes by employing identical datasets and statistical methods. Methods A systematic PubMed search between 2013 and 2023 identified ACS-NSQIP studies involving hip or knee arthroplasty and smoking as a potential risk factor for poor surgical outcomes. Each study’s methods were reproduced by a trained statistician based on the reported methodology. In cases where certain steps were not explicitly stated, the statistician made informed decisions to reproduce those steps. Adjusted odds ratios (aORs) and p-values (α = 0.05) were compared between the original and reanalyzed datasets. Results The initial search yielded 43 studies, with 11 meeting inclusion criteria resulting in the reanalysis of 268 aORs. Upon reanalysis, 12.69% of the original studies’ aORs changed in interpretation, while 13.43% experienced a change in statistical significance. The average magnitude change of each aOR across all studies was 17.22%, and the sample size (N) in reanalysis varied by up to 47.84%. Among the 11 commonly cited studies, approximately one in eight objective conclusions changed in interpretation or statistical significance. Conclusion Inconsistent reproducibility exists across many arthroplasty studies that utilize the ACS-NSQIP database. These findings highlight the importance of rigorous reporting of study methodology, data collection, and statistical analyses when utilizing large-volume databases in orthopaedic research. This burden of responsibility should be shared among authors, peer reviewers, and orthopaedic journals to confirm the accuracy and validity of published database research. Level of evidence This study systematically reviewed and analyzed, in attempt to reproduce, published arthroplasty studies utilizing ACS-NSQIP database to assess smoking as a potential risk factor for poor surgical outcomes. All analyzed studies included Level III Evidence, therefore this current study compares reproduced Level III Evidence to the original Level III Evidence. |
| format | Article |
| id | doaj-art-3f0a8250fa234c3db960ace62eaf5d5e |
| institution | DOAJ |
| issn | 1749-799X |
| language | English |
| publishDate | 2025-03-01 |
| publisher | BMC |
| record_format | Article |
| series | Journal of Orthopaedic Surgery and Research |
| spelling | doaj-art-3f0a8250fa234c3db960ace62eaf5d5e2025-08-20T03:00:39ZengBMCJournal of Orthopaedic Surgery and Research1749-799X2025-03-012011910.1186/s13018-025-05538-0Assessing the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studiesAyobami S. Ogunsola0Michael C. Marinier1Arman C. Hlas2Jacob M. Elkins3Department of Orthopedics and Rehabilitation, University of IowaDepartment of Orthopedics and Rehabilitation, University of IowaDepartment of Orthopedics and Rehabilitation, University of IowaDepartment of Orthopedics and Rehabilitation, University of IowaAbstract Background Utilization of large-volume clinical registries for observational research has gained popularity in orthopaedic literature. However, concerns exist regarding inadequate reporting of methodology in this type of research. Despite these concerns, the reproducibility of such studies has not been adequately assessed in existing literature. This study aims to assess the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies on smoking as a risk factor for poor surgical outcomes by employing identical datasets and statistical methods. Methods A systematic PubMed search between 2013 and 2023 identified ACS-NSQIP studies involving hip or knee arthroplasty and smoking as a potential risk factor for poor surgical outcomes. Each study’s methods were reproduced by a trained statistician based on the reported methodology. In cases where certain steps were not explicitly stated, the statistician made informed decisions to reproduce those steps. Adjusted odds ratios (aORs) and p-values (α = 0.05) were compared between the original and reanalyzed datasets. Results The initial search yielded 43 studies, with 11 meeting inclusion criteria resulting in the reanalysis of 268 aORs. Upon reanalysis, 12.69% of the original studies’ aORs changed in interpretation, while 13.43% experienced a change in statistical significance. The average magnitude change of each aOR across all studies was 17.22%, and the sample size (N) in reanalysis varied by up to 47.84%. Among the 11 commonly cited studies, approximately one in eight objective conclusions changed in interpretation or statistical significance. Conclusion Inconsistent reproducibility exists across many arthroplasty studies that utilize the ACS-NSQIP database. These findings highlight the importance of rigorous reporting of study methodology, data collection, and statistical analyses when utilizing large-volume databases in orthopaedic research. This burden of responsibility should be shared among authors, peer reviewers, and orthopaedic journals to confirm the accuracy and validity of published database research. Level of evidence This study systematically reviewed and analyzed, in attempt to reproduce, published arthroplasty studies utilizing ACS-NSQIP database to assess smoking as a potential risk factor for poor surgical outcomes. All analyzed studies included Level III Evidence, therefore this current study compares reproduced Level III Evidence to the original Level III Evidence.https://doi.org/10.1186/s13018-025-05538-0ReproducibilityArthroplastyACS-NSQIPSmokingRisk factorsSurgical outcomes |
| spellingShingle | Ayobami S. Ogunsola Michael C. Marinier Arman C. Hlas Jacob M. Elkins Assessing the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies Journal of Orthopaedic Surgery and Research Reproducibility Arthroplasty ACS-NSQIP Smoking Risk factors Surgical outcomes |
| title | Assessing the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies |
| title_full | Assessing the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies |
| title_fullStr | Assessing the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies |
| title_full_unstemmed | Assessing the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies |
| title_short | Assessing the reproducibility of American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) arthroplasty studies |
| title_sort | assessing the reproducibility of american college of surgeons national surgical quality improvement program acs nsqip arthroplasty studies |
| topic | Reproducibility Arthroplasty ACS-NSQIP Smoking Risk factors Surgical outcomes |
| url | https://doi.org/10.1186/s13018-025-05538-0 |
| work_keys_str_mv | AT ayobamisogunsola assessingthereproducibilityofamericancollegeofsurgeonsnationalsurgicalqualityimprovementprogramacsnsqiparthroplastystudies AT michaelcmarinier assessingthereproducibilityofamericancollegeofsurgeonsnationalsurgicalqualityimprovementprogramacsnsqiparthroplastystudies AT armanchlas assessingthereproducibilityofamericancollegeofsurgeonsnationalsurgicalqualityimprovementprogramacsnsqiparthroplastystudies AT jacobmelkins assessingthereproducibilityofamericancollegeofsurgeonsnationalsurgicalqualityimprovementprogramacsnsqiparthroplastystudies |