Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions
Summary Background Dichotomisation of statistical significance, rather than interpretation of effect sizes supported by confidence intervals, is a long-standing problem. Methods We distributed an online survey to clinical trial statisticians across the UK, Australia and Canada asking about their exp...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-10-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-024-02366-4 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850179373607419904 |
|---|---|
| author | Karla Hemming Laura Kudrna Sam Watson Monica Taljaard Sheila Greenfield Beatriz Goulao Richard Lilford |
| author_facet | Karla Hemming Laura Kudrna Sam Watson Monica Taljaard Sheila Greenfield Beatriz Goulao Richard Lilford |
| author_sort | Karla Hemming |
| collection | DOAJ |
| description | Summary Background Dichotomisation of statistical significance, rather than interpretation of effect sizes supported by confidence intervals, is a long-standing problem. Methods We distributed an online survey to clinical trial statisticians across the UK, Australia and Canada asking about their experiences, perspectives and practices with respect to interpretation of statistical findings from randomised trials. We report a descriptive analysis of the closed-ended questions and a thematic analysis of the open-ended questions. Results We obtained 101 responses across a broad range of career stages (24% professors; 51% senior lecturers; 22% junior statisticians) and areas of work (28% early phase trials; 44% drug trials; 38% health service trials). The majority (93%) believed that statistical findings should be interpreted by considering (minimal) clinical importance of treatment effects, but many (61%) said quantifying clinically important effect sizes was difficult, and fewer (54%) followed this approach in practice. Thematic analysis identified several barriers to forming a consensus on the statistical interpretation of the study findings, including: the dynamics within teams, lack of knowledge or difficulties in communicating that knowledge, as well as external pressures. External pressures included the pressure to publish definitive findings and statistical review which can sometimes be unhelpful but can at times be a saving grace. However, the concept of the minimally important difference was identified as a particularly poorly defined, even nebulous, construct which lies at the heart of much disagreement and confusion in the field. Conclusion The majority of participating statisticians believed that it is important to interpret statistical findings based on the clinically important effect size, but report this is difficult to operationalise. Reaching a consensus on the interpretation of a study is a social process involving disparate members of the research team along with editors and reviewers, as well as patients who likely have a role in the elicitation of minimally important differences. |
| format | Article |
| id | doaj-art-0f9aac66bb5344c7a7a063ffdeca1add |
| institution | OA Journals |
| issn | 1471-2288 |
| language | English |
| publishDate | 2024-10-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj-art-0f9aac66bb5344c7a7a063ffdeca1add2025-08-20T02:18:31ZengBMCBMC Medical Research Methodology1471-22882024-10-0124111310.1186/s12874-024-02366-4Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questionsKarla Hemming0Laura Kudrna1Sam Watson2Monica Taljaard3Sheila Greenfield4Beatriz Goulao5Richard Lilford6Institute of Applied Health Research, University of BirminghamInstitute of Applied Health Research, University of BirminghamInstitute of Applied Health Research, University of BirminghamClinical Epidemiology Program, Ottawa Hospital Research InstituteInstitute of Applied Health Research, University of BirminghamHealth Services Research Unit, University of AberdeenInstitute of Applied Health Research, University of BirminghamSummary Background Dichotomisation of statistical significance, rather than interpretation of effect sizes supported by confidence intervals, is a long-standing problem. Methods We distributed an online survey to clinical trial statisticians across the UK, Australia and Canada asking about their experiences, perspectives and practices with respect to interpretation of statistical findings from randomised trials. We report a descriptive analysis of the closed-ended questions and a thematic analysis of the open-ended questions. Results We obtained 101 responses across a broad range of career stages (24% professors; 51% senior lecturers; 22% junior statisticians) and areas of work (28% early phase trials; 44% drug trials; 38% health service trials). The majority (93%) believed that statistical findings should be interpreted by considering (minimal) clinical importance of treatment effects, but many (61%) said quantifying clinically important effect sizes was difficult, and fewer (54%) followed this approach in practice. Thematic analysis identified several barriers to forming a consensus on the statistical interpretation of the study findings, including: the dynamics within teams, lack of knowledge or difficulties in communicating that knowledge, as well as external pressures. External pressures included the pressure to publish definitive findings and statistical review which can sometimes be unhelpful but can at times be a saving grace. However, the concept of the minimally important difference was identified as a particularly poorly defined, even nebulous, construct which lies at the heart of much disagreement and confusion in the field. Conclusion The majority of participating statisticians believed that it is important to interpret statistical findings based on the clinically important effect size, but report this is difficult to operationalise. Reaching a consensus on the interpretation of a study is a social process involving disparate members of the research team along with editors and reviewers, as well as patients who likely have a role in the elicitation of minimally important differences.https://doi.org/10.1186/s12874-024-02366-4Confidence intervalsP-valuesMisinterpretationRCTs |
| spellingShingle | Karla Hemming Laura Kudrna Sam Watson Monica Taljaard Sheila Greenfield Beatriz Goulao Richard Lilford Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions BMC Medical Research Methodology Confidence intervals P-values Misinterpretation RCTs |
| title | Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions |
| title_full | Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions |
| title_fullStr | Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions |
| title_full_unstemmed | Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions |
| title_short | Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions |
| title_sort | interpretation of statistical findings in randomised trials a survey of statisticians using thematic analysis of open ended questions |
| topic | Confidence intervals P-values Misinterpretation RCTs |
| url | https://doi.org/10.1186/s12874-024-02366-4 |
| work_keys_str_mv | AT karlahemming interpretationofstatisticalfindingsinrandomisedtrialsasurveyofstatisticiansusingthematicanalysisofopenendedquestions AT laurakudrna interpretationofstatisticalfindingsinrandomisedtrialsasurveyofstatisticiansusingthematicanalysisofopenendedquestions AT samwatson interpretationofstatisticalfindingsinrandomisedtrialsasurveyofstatisticiansusingthematicanalysisofopenendedquestions AT monicataljaard interpretationofstatisticalfindingsinrandomisedtrialsasurveyofstatisticiansusingthematicanalysisofopenendedquestions AT sheilagreenfield interpretationofstatisticalfindingsinrandomisedtrialsasurveyofstatisticiansusingthematicanalysisofopenendedquestions AT beatrizgoulao interpretationofstatisticalfindingsinrandomisedtrialsasurveyofstatisticiansusingthematicanalysisofopenendedquestions AT richardlilford interpretationofstatisticalfindingsinrandomisedtrialsasurveyofstatisticiansusingthematicanalysisofopenendedquestions |