An empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivity
Abstract Background Transitivity assumption is the cornerstone of network meta-analysis (NMA). Investigating the plausibility of transitivity can unveil the credibility of NMA results. The commonness of transitivity was examined based on study dissimilarities regarding several study-level aggregate...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-12-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-024-02436-7 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850102844667985920 |
|---|---|
| author | Loukia M. Spineli |
| author_facet | Loukia M. Spineli |
| author_sort | Loukia M. Spineli |
| collection | DOAJ |
| description | Abstract Background Transitivity assumption is the cornerstone of network meta-analysis (NMA). Investigating the plausibility of transitivity can unveil the credibility of NMA results. The commonness of transitivity was examined based on study dissimilarities regarding several study-level aggregate clinical and methodological characteristics reported in the systematic reviews. The present study also demonstrated the disadvantages of using multiple statistical tests to assess transitivity and compared the conclusions drawn from multiple statistical tests with those from the approach of study dissimilarities for transitivity assessment. Methods An empirical study was conducted using 209 published systematic reviews with NMA to create a database of study-level aggregate clinical and methodological characteristics found in the tracenma R package. For each systematic review, the network of the primary outcome was considered to create a dataset with extracted study-level aggregate clinical and methodological characteristics reported in the systematic review that may act as effect modifiers. Transitivity was evaluated by calculating study dissimilarities based on the extracted characteristics to provide a measure of overall dissimilarity within and between the observed treatment comparisons. Empirically driven thresholds of low dissimilarity were employed to determine the proportion of datasets with evidence of likely intransitivity. One-way ANOVA and chi-squared test were employed for each characteristic to investigate comparison dissimilarity at a significance level of 5%. Results Study dissimilarities covered a wide range of possible values across the datasets. A ‘likely concerning’ extent of study dissimilarities, both intra-comparison and inter-comparison, dominated the analysed datasets. Using a higher dissimilarity threshold, a ‘likely concerning’ extent of study dissimilarities persisted for objective outcomes but decreased substantially for subjective outcomes. A likely intransitivity prevailed in all datasets; however, using a higher dissimilarity threshold resulted in few networks with transitivity for semi-objective and subjective outcomes. Statistical tests were feasible in 127 (61%) datasets, yielding conflicting conclusions with the approach of study dissimilarities in many datasets. Conclusions Study dissimilarity, manifested from variations in the effect modifiers’ distribution across the studies, should be expected and properly quantified. Measuring the overall study dissimilarity between observed comparisons and comparing it with a proper threshold can aid in determining whether concerns of likely intransitivity are warranted. |
| format | Article |
| id | doaj-art-859c2aa399264a6d870e7c1b8bbff156 |
| institution | DOAJ |
| issn | 1471-2288 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj-art-859c2aa399264a6d870e7c1b8bbff1562025-08-20T02:39:40ZengBMCBMC Medical Research Methodology1471-22882024-12-0124111410.1186/s12874-024-02436-7An empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivityLoukia M. Spineli0Midwifery Research and Education Unit (OE 9210), Hannover Medical SchoolAbstract Background Transitivity assumption is the cornerstone of network meta-analysis (NMA). Investigating the plausibility of transitivity can unveil the credibility of NMA results. The commonness of transitivity was examined based on study dissimilarities regarding several study-level aggregate clinical and methodological characteristics reported in the systematic reviews. The present study also demonstrated the disadvantages of using multiple statistical tests to assess transitivity and compared the conclusions drawn from multiple statistical tests with those from the approach of study dissimilarities for transitivity assessment. Methods An empirical study was conducted using 209 published systematic reviews with NMA to create a database of study-level aggregate clinical and methodological characteristics found in the tracenma R package. For each systematic review, the network of the primary outcome was considered to create a dataset with extracted study-level aggregate clinical and methodological characteristics reported in the systematic review that may act as effect modifiers. Transitivity was evaluated by calculating study dissimilarities based on the extracted characteristics to provide a measure of overall dissimilarity within and between the observed treatment comparisons. Empirically driven thresholds of low dissimilarity were employed to determine the proportion of datasets with evidence of likely intransitivity. One-way ANOVA and chi-squared test were employed for each characteristic to investigate comparison dissimilarity at a significance level of 5%. Results Study dissimilarities covered a wide range of possible values across the datasets. A ‘likely concerning’ extent of study dissimilarities, both intra-comparison and inter-comparison, dominated the analysed datasets. Using a higher dissimilarity threshold, a ‘likely concerning’ extent of study dissimilarities persisted for objective outcomes but decreased substantially for subjective outcomes. A likely intransitivity prevailed in all datasets; however, using a higher dissimilarity threshold resulted in few networks with transitivity for semi-objective and subjective outcomes. Statistical tests were feasible in 127 (61%) datasets, yielding conflicting conclusions with the approach of study dissimilarities in many datasets. Conclusions Study dissimilarity, manifested from variations in the effect modifiers’ distribution across the studies, should be expected and properly quantified. Measuring the overall study dissimilarity between observed comparisons and comparing it with a proper threshold can aid in determining whether concerns of likely intransitivity are warranted.https://doi.org/10.1186/s12874-024-02436-7Systematic reviewNetwork meta-analysisTransitivityHeterogeneityDissimilarity |
| spellingShingle | Loukia M. Spineli An empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivity BMC Medical Research Methodology Systematic review Network meta-analysis Transitivity Heterogeneity Dissimilarity |
| title | An empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivity |
| title_full | An empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivity |
| title_fullStr | An empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivity |
| title_full_unstemmed | An empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivity |
| title_short | An empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivity |
| title_sort | empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivity |
| topic | Systematic review Network meta-analysis Transitivity Heterogeneity Dissimilarity |
| url | https://doi.org/10.1186/s12874-024-02436-7 |
| work_keys_str_mv | AT loukiamspineli anempiricalstudyon209networksoftreatmentsrevealedintransitivitytobecommonandmultiplestatisticaltestssuboptimaltoassesstransitivity AT loukiamspineli empiricalstudyon209networksoftreatmentsrevealedintransitivitytobecommonandmultiplestatisticaltestssuboptimaltoassesstransitivity |