Compatibility ranges as a practical alternative to the “significant/non-significant” statistical dichotomy - Alessandro Rovetta (Public Health and Toxicology)
Science can be defined as a social system built on the concept of critical agreement on evidentiary states. The latter must be achieved through rational thinking, communication, and the so-called ‘scientific method’, which involves a series of procedures aimed at ensuring the replicability of investigative experiments. In this regard, the dichotomization of statistical results into ‘significant’ and ‘non-significant’ has led to a long series of replication failures and fostered the misleading expectation that a mere numerical criterion can replace analytical reasoning. Especially in fields like toxicology and public health, such misuse can have serious consequences. Indeed, no study can prove that a result is (not) significant since uncertainty is always part of scientific research. At most, based on the above considerations and a comprehensive analysis of costs, risks, and benefits, it can be decided whether a certain phenomenon meets the threshold of scientific evidence required to undertake concrete actions. In light of this, the present manuscript proposes and discusses alternative concepts to statistical dichotomies, such as ranges of compatibility and effect size. Furthermore, it emphasizes the necessity to investigate the compatibility of the experimental data with all the relevant target hypotheses (not just the null one) and all the background assumptions. Finally, it proposes a compact framework for a complete presentation of results, including effect size. In this regard, the adoption of multiple confidence/compatibility intervals or surprisal intervals is recommended.