Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen’s kappa
Objective To provide a framework for quantifying the robustness of treatment ranks based on Surface Under the Cumulative RAnking curve (SUCRA) in network meta-analysis (NMA) and investigating potential factors associated with lack of robustness.Methods We propose the use of Cohen’s kappa to quantify...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMJ Publishing Group
2019-09-01
|
| Series: | BMJ Open |
| Online Access: | https://bmjopen.bmj.com/content/9/9/e024625.full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850066534986153984 |
|---|---|
| author | Lehana Thabane Sharon E Straus Caitlin H Daly Binod Neupane Joseph Beyene Jemila S Hamid |
| author_facet | Lehana Thabane Sharon E Straus Caitlin H Daly Binod Neupane Joseph Beyene Jemila S Hamid |
| author_sort | Lehana Thabane |
| collection | DOAJ |
| description | Objective To provide a framework for quantifying the robustness of treatment ranks based on Surface Under the Cumulative RAnking curve (SUCRA) in network meta-analysis (NMA) and investigating potential factors associated with lack of robustness.Methods We propose the use of Cohen’s kappa to quantify the agreement between SUCRA-based treatment ranks estimated through NMA of a complete data set and a subset of it. We illustrate our approach using five published NMA data sets, where robustness was assessed by removing studies one at a time.Results Overall, SUCRA-based treatment ranks were robust to individual studies in the five data sets we considered. We observed more incidences of disagreement between ranks in the networks with larger numbers of treatments. Most treatments moved only one or two ranks up or down. The lowest quadratic weighted kappa estimate observed across all networks was in the network with the smallest number of treatments (4), where weighted kappa=40%. In the network with the largest number of treatments (12), the lowest observed quadratic weighted kappa=89%, reflecting a small shift in this network's treatment ranks overall. Preliminary observations suggest that a study’s size, the number of studies making a treatment comparison, and the agreement of a study’s estimated treatment effect(s) with those estimated by other studies making the same comparison(s) may explain the overall robustness of treatment ranks to studies.Conclusions Investigating robustness or sensitivity in an NMA may reveal outlying rank changes that are clinically or policy-relevant. Cohen’s kappa is a useful measure that permits investigation into study characteristics that may explain varying sensitivity to individual studies. However, this study presents a framework as a proof of concept and further investigation is required to identify potential factors associated with the robustness of treatment ranks using more extensive empirical evaluations. |
| format | Article |
| id | doaj-art-87636d70ce3d4187a07f443aef3faaad |
| institution | DOAJ |
| issn | 2044-6055 |
| language | English |
| publishDate | 2019-09-01 |
| publisher | BMJ Publishing Group |
| record_format | Article |
| series | BMJ Open |
| spelling | doaj-art-87636d70ce3d4187a07f443aef3faaad2025-08-20T02:48:42ZengBMJ Publishing GroupBMJ Open2044-60552019-09-019910.1136/bmjopen-2018-024625Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen’s kappaLehana Thabane0Sharon E Straus1Caitlin H Daly2Binod Neupane3Joseph Beyene4Jemila S Hamid57 Department of Health Research Method, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada10 Department of Medicine, University of Toronto, Toronto, Ontario, Canada1 Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada1 Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada2 Department of Mathematics and Statistics, McMaster University, Hamilton, Ontario, CanadaDepartment of Mathematics and Statistics, University of Ottawa, Ottawa, Ontario, CanadaObjective To provide a framework for quantifying the robustness of treatment ranks based on Surface Under the Cumulative RAnking curve (SUCRA) in network meta-analysis (NMA) and investigating potential factors associated with lack of robustness.Methods We propose the use of Cohen’s kappa to quantify the agreement between SUCRA-based treatment ranks estimated through NMA of a complete data set and a subset of it. We illustrate our approach using five published NMA data sets, where robustness was assessed by removing studies one at a time.Results Overall, SUCRA-based treatment ranks were robust to individual studies in the five data sets we considered. We observed more incidences of disagreement between ranks in the networks with larger numbers of treatments. Most treatments moved only one or two ranks up or down. The lowest quadratic weighted kappa estimate observed across all networks was in the network with the smallest number of treatments (4), where weighted kappa=40%. In the network with the largest number of treatments (12), the lowest observed quadratic weighted kappa=89%, reflecting a small shift in this network's treatment ranks overall. Preliminary observations suggest that a study’s size, the number of studies making a treatment comparison, and the agreement of a study’s estimated treatment effect(s) with those estimated by other studies making the same comparison(s) may explain the overall robustness of treatment ranks to studies.Conclusions Investigating robustness or sensitivity in an NMA may reveal outlying rank changes that are clinically or policy-relevant. Cohen’s kappa is a useful measure that permits investigation into study characteristics that may explain varying sensitivity to individual studies. However, this study presents a framework as a proof of concept and further investigation is required to identify potential factors associated with the robustness of treatment ranks using more extensive empirical evaluations.https://bmjopen.bmj.com/content/9/9/e024625.full |
| spellingShingle | Lehana Thabane Sharon E Straus Caitlin H Daly Binod Neupane Joseph Beyene Jemila S Hamid Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen’s kappa BMJ Open |
| title | Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen’s kappa |
| title_full | Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen’s kappa |
| title_fullStr | Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen’s kappa |
| title_full_unstemmed | Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen’s kappa |
| title_short | Empirical evaluation of SUCRA-based treatment ranks in network meta-analysis: quantifying robustness using Cohen’s kappa |
| title_sort | empirical evaluation of sucra based treatment ranks in network meta analysis quantifying robustness using cohen s kappa |
| url | https://bmjopen.bmj.com/content/9/9/e024625.full |
| work_keys_str_mv | AT lehanathabane empiricalevaluationofsucrabasedtreatmentranksinnetworkmetaanalysisquantifyingrobustnessusingcohenskappa AT sharonestraus empiricalevaluationofsucrabasedtreatmentranksinnetworkmetaanalysisquantifyingrobustnessusingcohenskappa AT caitlinhdaly empiricalevaluationofsucrabasedtreatmentranksinnetworkmetaanalysisquantifyingrobustnessusingcohenskappa AT binodneupane empiricalevaluationofsucrabasedtreatmentranksinnetworkmetaanalysisquantifyingrobustnessusingcohenskappa AT josephbeyene empiricalevaluationofsucrabasedtreatmentranksinnetworkmetaanalysisquantifyingrobustnessusingcohenskappa AT jemilashamid empiricalevaluationofsucrabasedtreatmentranksinnetworkmetaanalysisquantifyingrobustnessusingcohenskappa |