Assessment of inverse publication bias in safety outcomes: an empirical analysis
Abstract Background The aims of this study were to assess the presence of inverse publication bias (IPB) in adverse events, evaluate the performance of visual examination, and explore the impact of considering effect direction in statistical tests for such assessments. Methods We conducted a cross-s...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-10-01
|
| Series: | BMC Medicine |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12916-024-03707-2 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850203721534799872 |
|---|---|
| author | Xing Xing Jianan Zhu Linyu Shi Chang Xu Lifeng Lin |
| author_facet | Xing Xing Jianan Zhu Linyu Shi Chang Xu Lifeng Lin |
| author_sort | Xing Xing |
| collection | DOAJ |
| description | Abstract Background The aims of this study were to assess the presence of inverse publication bias (IPB) in adverse events, evaluate the performance of visual examination, and explore the impact of considering effect direction in statistical tests for such assessments. Methods We conducted a cross-sectional study using the SMART Safety, the largest dataset for evidence synthesis of adverse events. The visual assessment was performed using contour-enhanced funnel plots, trim-and-fill funnel plots, and sample-size-based funnel plots. Two authors conducted visual assessments of these plots independently, and their agreements were quantified by the kappa statistics. Additionally, IPB was quantitatively assessed using both the one- and two-sided Egger’s and Peters’ tests. Results In the SMART Safety dataset, we identified 277 main meta-analyses of safety outcomes with at least 10 individual estimates after dropping missing data. We found that about 13.7–16.2% of meta-analyses exhibited IPB according to the one-sided test results. The kappa statistics for the visual assessments roughly ranged from 0.3 to 0.5, indicating fair to moderate agreement. Using the one-sided Egger’s test, 57 out of 72 (79.2%) meta-analyses that initially showed significant IPB in the two-sided test changed to non-significant, while the remaining 15 (20.8%) meta-analyses changed from non-significant to significant. Conclusions Our findings provide supporting evidence of IPB in the SMART Safety dataset of adverse events. They also suggest the importance of researchers carefully accounting for the direction of statistical tests for IPB, as well as the challenges of assessing IPB using statistical methods, especially considering that the number of studies is typically small. Qualitative assessments may be a necessary supplement to gain a more comprehensive understanding of IPB. |
| format | Article |
| id | doaj-art-4a197845bb7246c488652709fdd18320 |
| institution | OA Journals |
| issn | 1741-7015 |
| language | English |
| publishDate | 2024-10-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medicine |
| spelling | doaj-art-4a197845bb7246c488652709fdd183202025-08-20T02:11:26ZengBMCBMC Medicine1741-70152024-10-012211910.1186/s12916-024-03707-2Assessment of inverse publication bias in safety outcomes: an empirical analysisXing Xing0Jianan Zhu1Linyu Shi2Chang Xu3Lifeng Lin4Department of Biostatistics, Johns Hopkins Bloomberg School of Public HealthDepartment of Biostatistics, School of Global Public Health, New York UniversityAbbVie IncProof of Concept Center, Eastern Hepatobiliary Surgery Hospital, Third Affiliated Hospital, Second Military Medical University, Naval Medical UniversityDepartment of Epidemiology and Biostatistics, University of ArizonaAbstract Background The aims of this study were to assess the presence of inverse publication bias (IPB) in adverse events, evaluate the performance of visual examination, and explore the impact of considering effect direction in statistical tests for such assessments. Methods We conducted a cross-sectional study using the SMART Safety, the largest dataset for evidence synthesis of adverse events. The visual assessment was performed using contour-enhanced funnel plots, trim-and-fill funnel plots, and sample-size-based funnel plots. Two authors conducted visual assessments of these plots independently, and their agreements were quantified by the kappa statistics. Additionally, IPB was quantitatively assessed using both the one- and two-sided Egger’s and Peters’ tests. Results In the SMART Safety dataset, we identified 277 main meta-analyses of safety outcomes with at least 10 individual estimates after dropping missing data. We found that about 13.7–16.2% of meta-analyses exhibited IPB according to the one-sided test results. The kappa statistics for the visual assessments roughly ranged from 0.3 to 0.5, indicating fair to moderate agreement. Using the one-sided Egger’s test, 57 out of 72 (79.2%) meta-analyses that initially showed significant IPB in the two-sided test changed to non-significant, while the remaining 15 (20.8%) meta-analyses changed from non-significant to significant. Conclusions Our findings provide supporting evidence of IPB in the SMART Safety dataset of adverse events. They also suggest the importance of researchers carefully accounting for the direction of statistical tests for IPB, as well as the challenges of assessing IPB using statistical methods, especially considering that the number of studies is typically small. Qualitative assessments may be a necessary supplement to gain a more comprehensive understanding of IPB.https://doi.org/10.1186/s12916-024-03707-2Adverse eventFunnel plotInverse publication biasPublication biasSystematic review |
| spellingShingle | Xing Xing Jianan Zhu Linyu Shi Chang Xu Lifeng Lin Assessment of inverse publication bias in safety outcomes: an empirical analysis BMC Medicine Adverse event Funnel plot Inverse publication bias Publication bias Systematic review |
| title | Assessment of inverse publication bias in safety outcomes: an empirical analysis |
| title_full | Assessment of inverse publication bias in safety outcomes: an empirical analysis |
| title_fullStr | Assessment of inverse publication bias in safety outcomes: an empirical analysis |
| title_full_unstemmed | Assessment of inverse publication bias in safety outcomes: an empirical analysis |
| title_short | Assessment of inverse publication bias in safety outcomes: an empirical analysis |
| title_sort | assessment of inverse publication bias in safety outcomes an empirical analysis |
| topic | Adverse event Funnel plot Inverse publication bias Publication bias Systematic review |
| url | https://doi.org/10.1186/s12916-024-03707-2 |
| work_keys_str_mv | AT xingxing assessmentofinversepublicationbiasinsafetyoutcomesanempiricalanalysis AT jiananzhu assessmentofinversepublicationbiasinsafetyoutcomesanempiricalanalysis AT linyushi assessmentofinversepublicationbiasinsafetyoutcomesanempiricalanalysis AT changxu assessmentofinversepublicationbiasinsafetyoutcomesanempiricalanalysis AT lifenglin assessmentofinversepublicationbiasinsafetyoutcomesanempiricalanalysis |