Assessing the replicability of RCTs in RWE emulations
Abstract Background The standard regulatory approach to assess replication success is the two-trials rule, requiring both the original and the replication study to be significant with effect estimates in the same direction. The sceptical p-value was recently presented as an alternative method for th...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-05-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-025-02589-z |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850269185284767744 |
|---|---|
| author | Jeanette Köppe Charlotte Micheloud Stella Erdmann Rachel Heyard Leonhard Held |
| author_facet | Jeanette Köppe Charlotte Micheloud Stella Erdmann Rachel Heyard Leonhard Held |
| author_sort | Jeanette Köppe |
| collection | DOAJ |
| description | Abstract Background The standard regulatory approach to assess replication success is the two-trials rule, requiring both the original and the replication study to be significant with effect estimates in the same direction. The sceptical p-value was recently presented as an alternative method for the statistical assessment of the replicability of study results. Methods We review the statistical properties of the sceptical p-value and compare those to the two-trials rule. We extend the methodology to non-inferiority trials and describe how to invert the sceptical p-value to obtain confidence intervals. We illustrate the performance of the different methods using real-world evidence emulations of randomized controlled trials (RCTs) conducted within the RCT DUPLICATE initiative. Results The sceptical p-value depends not only on the two p-values, but also on sample size and effect size of the two studies. It can be calibrated to have the same Type-I error rate as the two-trials rule, but has larger power to detect an existing effect. In the application to the results from the RCT DUPLICATE initiative, the sceptical p-value leads to qualitatively similar results than the two-trials rule, but tends to show more evidence for treatment effects compared to the two-trials rule. Conclusion The sceptical p-value represents a valid statistical measure to assess the replicability of study results and is useful in the context of real-world evidence emulations. |
| format | Article |
| id | doaj-art-2a6b3cfa016e4ab8ad61b4eac9433f72 |
| institution | OA Journals |
| issn | 1471-2288 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj-art-2a6b3cfa016e4ab8ad61b4eac9433f722025-08-20T01:53:14ZengBMCBMC Medical Research Methodology1471-22882025-05-0125111510.1186/s12874-025-02589-zAssessing the replicability of RCTs in RWE emulationsJeanette Köppe0Charlotte Micheloud1Stella Erdmann2Rachel Heyard3Leonhard Held4Institute of Biostatistics and Clinical Research, University of MuensterEpidemiology, Biostatistics and Prevention Institute, University of ZurichInstitute of Medical Biometry, University of HeidelbergEpidemiology, Biostatistics and Prevention Institute, University of ZurichEpidemiology, Biostatistics and Prevention Institute, University of ZurichAbstract Background The standard regulatory approach to assess replication success is the two-trials rule, requiring both the original and the replication study to be significant with effect estimates in the same direction. The sceptical p-value was recently presented as an alternative method for the statistical assessment of the replicability of study results. Methods We review the statistical properties of the sceptical p-value and compare those to the two-trials rule. We extend the methodology to non-inferiority trials and describe how to invert the sceptical p-value to obtain confidence intervals. We illustrate the performance of the different methods using real-world evidence emulations of randomized controlled trials (RCTs) conducted within the RCT DUPLICATE initiative. Results The sceptical p-value depends not only on the two p-values, but also on sample size and effect size of the two studies. It can be calibrated to have the same Type-I error rate as the two-trials rule, but has larger power to detect an existing effect. In the application to the results from the RCT DUPLICATE initiative, the sceptical p-value leads to qualitatively similar results than the two-trials rule, but tends to show more evidence for treatment effects compared to the two-trials rule. Conclusion The sceptical p-value represents a valid statistical measure to assess the replicability of study results and is useful in the context of real-world evidence emulations.https://doi.org/10.1186/s12874-025-02589-zRandomized clinical trialReal-world evidenceReplicationSceptical p-valueTwo-trials rule |
| spellingShingle | Jeanette Köppe Charlotte Micheloud Stella Erdmann Rachel Heyard Leonhard Held Assessing the replicability of RCTs in RWE emulations BMC Medical Research Methodology Randomized clinical trial Real-world evidence Replication Sceptical p-value Two-trials rule |
| title | Assessing the replicability of RCTs in RWE emulations |
| title_full | Assessing the replicability of RCTs in RWE emulations |
| title_fullStr | Assessing the replicability of RCTs in RWE emulations |
| title_full_unstemmed | Assessing the replicability of RCTs in RWE emulations |
| title_short | Assessing the replicability of RCTs in RWE emulations |
| title_sort | assessing the replicability of rcts in rwe emulations |
| topic | Randomized clinical trial Real-world evidence Replication Sceptical p-value Two-trials rule |
| url | https://doi.org/10.1186/s12874-025-02589-z |
| work_keys_str_mv | AT jeanettekoppe assessingthereplicabilityofrctsinrweemulations AT charlottemicheloud assessingthereplicabilityofrctsinrweemulations AT stellaerdmann assessingthereplicabilityofrctsinrweemulations AT rachelheyard assessingthereplicabilityofrctsinrweemulations AT leonhardheld assessingthereplicabilityofrctsinrweemulations |