A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses.
<h4>Background</h4>Heterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of unobserved heterogeneity and the validity o...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2013-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0069930 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849331737225592832 |
|---|---|
| author | Evangelos Kontopantelis David A Springate David Reeves |
| author_facet | Evangelos Kontopantelis David A Springate David Reeves |
| author_sort | Evangelos Kontopantelis |
| collection | DOAJ |
| description | <h4>Background</h4>Heterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of unobserved heterogeneity and the validity of the assumption of homogeneity; b) assess the performance of various meta-analysis methods; c) apply the findings to published meta-analyses.<h4>Methods and findings</h4>We accessed 57,397 meta-analyses, available in the Cochrane Library in August 2012. Using simulated data we assessed the performance of various meta-analysis methods in different scenarios. The prevalence of a zero heterogeneity estimate in the simulated scenarios was compared with that in the Cochrane data, to estimate the degree of unobserved heterogeneity in the latter. We re-analysed all meta-analyses using all methods and assessed the sensitivity of the statistical conclusions. Levels of unobserved heterogeneity in the Cochrane data appeared to be high, especially for small meta-analyses. A bootstrapped version of the DerSimonian-Laird approach performed best in both detecting heterogeneity and in returning more accurate overall effect estimates. Re-analysing all meta-analyses with this new method we found that in cases where heterogeneity had originally been detected but ignored, 17-20% of the statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it into account, between 1% and 3%.<h4>Conclusions</h4>When evidence for heterogeneity is lacking, standard practice is to assume homogeneity and apply a simpler fixed-effect meta-analysis. We find that assuming homogeneity often results in a misleading analysis, since heterogeneity is very likely present but undetected. Our new method represents a small improvement but the problem largely remains, especially for very small meta-analyses. One solution is to test the sensitivity of the meta-analysis conclusions to assumed moderate and large degrees of heterogeneity. Equally, whenever heterogeneity is detected, it should not be ignored. |
| format | Article |
| id | doaj-art-7abb25b3251a4e40887d719d97892424 |
| institution | Kabale University |
| issn | 1932-6203 |
| language | English |
| publishDate | 2013-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-7abb25b3251a4e40887d719d978924242025-08-20T03:46:24ZengPublic Library of Science (PLoS)PLoS ONE1932-62032013-01-0187e6993010.1371/journal.pone.0069930A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses.Evangelos KontopantelisDavid A SpringateDavid Reeves<h4>Background</h4>Heterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of unobserved heterogeneity and the validity of the assumption of homogeneity; b) assess the performance of various meta-analysis methods; c) apply the findings to published meta-analyses.<h4>Methods and findings</h4>We accessed 57,397 meta-analyses, available in the Cochrane Library in August 2012. Using simulated data we assessed the performance of various meta-analysis methods in different scenarios. The prevalence of a zero heterogeneity estimate in the simulated scenarios was compared with that in the Cochrane data, to estimate the degree of unobserved heterogeneity in the latter. We re-analysed all meta-analyses using all methods and assessed the sensitivity of the statistical conclusions. Levels of unobserved heterogeneity in the Cochrane data appeared to be high, especially for small meta-analyses. A bootstrapped version of the DerSimonian-Laird approach performed best in both detecting heterogeneity and in returning more accurate overall effect estimates. Re-analysing all meta-analyses with this new method we found that in cases where heterogeneity had originally been detected but ignored, 17-20% of the statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it into account, between 1% and 3%.<h4>Conclusions</h4>When evidence for heterogeneity is lacking, standard practice is to assume homogeneity and apply a simpler fixed-effect meta-analysis. We find that assuming homogeneity often results in a misleading analysis, since heterogeneity is very likely present but undetected. Our new method represents a small improvement but the problem largely remains, especially for very small meta-analyses. One solution is to test the sensitivity of the meta-analysis conclusions to assumed moderate and large degrees of heterogeneity. Equally, whenever heterogeneity is detected, it should not be ignored.https://doi.org/10.1371/journal.pone.0069930 |
| spellingShingle | Evangelos Kontopantelis David A Springate David Reeves A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses. PLoS ONE |
| title | A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses. |
| title_full | A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses. |
| title_fullStr | A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses. |
| title_full_unstemmed | A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses. |
| title_short | A re-analysis of the Cochrane Library data: the dangers of unobserved heterogeneity in meta-analyses. |
| title_sort | re analysis of the cochrane library data the dangers of unobserved heterogeneity in meta analyses |
| url | https://doi.org/10.1371/journal.pone.0069930 |
| work_keys_str_mv | AT evangeloskontopantelis areanalysisofthecochranelibrarydatathedangersofunobservedheterogeneityinmetaanalyses AT davidaspringate areanalysisofthecochranelibrarydatathedangersofunobservedheterogeneityinmetaanalyses AT davidreeves areanalysisofthecochranelibrarydatathedangersofunobservedheterogeneityinmetaanalyses AT evangeloskontopantelis reanalysisofthecochranelibrarydatathedangersofunobservedheterogeneityinmetaanalyses AT davidaspringate reanalysisofthecochranelibrarydatathedangersofunobservedheterogeneityinmetaanalyses AT davidreeves reanalysisofthecochranelibrarydatathedangersofunobservedheterogeneityinmetaanalyses |