On the reliability of published findings using the regression discontinuity design in political science
The regression discontinuity (RD) design offers identification of causal effects under weak assumptions, earning it a position as a standard method in modern political science research. But identification does not necessarily imply that causal effects can be estimated accurately with limited data. I...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SAGE Publishing
2023-04-01
|
| Series: | Research & Politics |
| Online Access: | https://doi.org/10.1177/20531680231166457 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850119665091608576 |
|---|---|
| author | Drew Stommes P. M. Aronow Fredrik Sävje |
| author_facet | Drew Stommes P. M. Aronow Fredrik Sävje |
| author_sort | Drew Stommes |
| collection | DOAJ |
| description | The regression discontinuity (RD) design offers identification of causal effects under weak assumptions, earning it a position as a standard method in modern political science research. But identification does not necessarily imply that causal effects can be estimated accurately with limited data. In this paper, we highlight that estimation under the RD design involves serious statistical challenges and investigate how these challenges manifest themselves in the empirical literature in political science. We collect all RD-based findings published in top political science journals in the period 2009–2018. The distribution of published results exhibits pathological features; estimates tend to bunch just above the conventional level of statistical significance. A reanalysis of all studies with available data suggests that researcher discretion is not a major driver of these features. However, researchers tend to use inappropriate methods for inference, rendering standard errors artificially small. A retrospective power analysis reveals that most of these studies were underpowered to detect all but large effects. The issues we uncover, combined with well-documented selection pressures in academic publishing, cause concern that many published findings using the RD design may be exaggerated. |
| format | Article |
| id | doaj-art-e37156be430c49aea79b81eb176c61b0 |
| institution | OA Journals |
| issn | 2053-1680 |
| language | English |
| publishDate | 2023-04-01 |
| publisher | SAGE Publishing |
| record_format | Article |
| series | Research & Politics |
| spelling | doaj-art-e37156be430c49aea79b81eb176c61b02025-08-20T02:35:34ZengSAGE PublishingResearch & Politics2053-16802023-04-011010.1177/20531680231166457On the reliability of published findings using the regression discontinuity design in political scienceDrew StommesP. M. AronowFredrik SävjeThe regression discontinuity (RD) design offers identification of causal effects under weak assumptions, earning it a position as a standard method in modern political science research. But identification does not necessarily imply that causal effects can be estimated accurately with limited data. In this paper, we highlight that estimation under the RD design involves serious statistical challenges and investigate how these challenges manifest themselves in the empirical literature in political science. We collect all RD-based findings published in top political science journals in the period 2009–2018. The distribution of published results exhibits pathological features; estimates tend to bunch just above the conventional level of statistical significance. A reanalysis of all studies with available data suggests that researcher discretion is not a major driver of these features. However, researchers tend to use inappropriate methods for inference, rendering standard errors artificially small. A retrospective power analysis reveals that most of these studies were underpowered to detect all but large effects. The issues we uncover, combined with well-documented selection pressures in academic publishing, cause concern that many published findings using the RD design may be exaggerated.https://doi.org/10.1177/20531680231166457 |
| spellingShingle | Drew Stommes P. M. Aronow Fredrik Sävje On the reliability of published findings using the regression discontinuity design in political science Research & Politics |
| title | On the reliability of published findings using the regression discontinuity design in political science |
| title_full | On the reliability of published findings using the regression discontinuity design in political science |
| title_fullStr | On the reliability of published findings using the regression discontinuity design in political science |
| title_full_unstemmed | On the reliability of published findings using the regression discontinuity design in political science |
| title_short | On the reliability of published findings using the regression discontinuity design in political science |
| title_sort | on the reliability of published findings using the regression discontinuity design in political science |
| url | https://doi.org/10.1177/20531680231166457 |
| work_keys_str_mv | AT drewstommes onthereliabilityofpublishedfindingsusingtheregressiondiscontinuitydesigninpoliticalscience AT pmaronow onthereliabilityofpublishedfindingsusingtheregressiondiscontinuitydesigninpoliticalscience AT fredriksavje onthereliabilityofpublishedfindingsusingtheregressiondiscontinuitydesigninpoliticalscience |