A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments
Forecasting the reproducibility of research findings is one of the key challenges of metascience. Above-chance predictions have mainly been achieved by pooling the subjective ratings of experts, and how these predictions are formed remains to be understood. Here, we show that reproducibility forecas...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2025-07-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/10.1098/rsos.241446 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849246257515593728 |
|---|---|
| author | Daniele Fanelli Pedro Batista Tan Olavo B. Amaral Kleber Neves |
| author_facet | Daniele Fanelli Pedro Batista Tan Olavo B. Amaral Kleber Neves |
| author_sort | Daniele Fanelli |
| collection | DOAJ |
| description | Forecasting the reproducibility of research findings is one of the key challenges of metascience. Above-chance predictions have mainly been achieved by pooling the subjective ratings of experts, and how these predictions are formed remains to be understood. Here, we show that reproducibility forecasts made for the Brazilian Reproducibility Initiative (BRI), a large-scale replication of experiments in the life sciences, are significantly correlated with K, a principled metric of knowledge as information compression. For each study in the BRI sample, we calculated K by dividing the effect size, measured in bits of Shannon entropy, by the descriptive length (a proxy of the complexity) of the study’s methodology, calculated as the optimal Shannon encoding of a conceptual graph representing the replication protocol. We found that experts’ predictions about reproducibility were statistically associated with K values and with the complexity of protocols. This relation was robust to controlling for study methodology and other possible confounding factors. These results suggest that expert raters partially judge the reproducibility of findings by assessing the ratio between the information yielded and the information required by a study, and they support the hypothesis that scientific knowledge may be understood and studied through the lenses of information compression. |
| format | Article |
| id | doaj-art-48e920dc40d44121ae85a2245a354b3f |
| institution | Kabale University |
| issn | 2054-5703 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj-art-48e920dc40d44121ae85a2245a354b3f2025-08-20T03:58:35ZengThe Royal SocietyRoyal Society Open Science2054-57032025-07-0112710.1098/rsos.241446A metric of knowledge as information compression reflects reproducibility predictions for biomedical experimentsDaniele Fanelli0Pedro Batista Tan1Olavo B. Amaral2Kleber Neves3Theoretical and Empirical METaknowledge (TEMET) lab, School of Social Sciences, Heriot-Watt University, Edinburgh, UKVrije Universiteit Amsterdam, Amsterdam, The NetherlandsFederal University of Rio de Janeiro, Rio de Janeiro, BrazilFederal University of Rio de Janeiro, Rio de Janeiro, BrazilForecasting the reproducibility of research findings is one of the key challenges of metascience. Above-chance predictions have mainly been achieved by pooling the subjective ratings of experts, and how these predictions are formed remains to be understood. Here, we show that reproducibility forecasts made for the Brazilian Reproducibility Initiative (BRI), a large-scale replication of experiments in the life sciences, are significantly correlated with K, a principled metric of knowledge as information compression. For each study in the BRI sample, we calculated K by dividing the effect size, measured in bits of Shannon entropy, by the descriptive length (a proxy of the complexity) of the study’s methodology, calculated as the optimal Shannon encoding of a conceptual graph representing the replication protocol. We found that experts’ predictions about reproducibility were statistically associated with K values and with the complexity of protocols. This relation was robust to controlling for study methodology and other possible confounding factors. These results suggest that expert raters partially judge the reproducibility of findings by assessing the ratio between the information yielded and the information required by a study, and they support the hypothesis that scientific knowledge may be understood and studied through the lenses of information compression.https://royalsocietypublishing.org/doi/10.1098/rsos.241446metasciencemetaresearchreproducibilitycomplexityinformation compressionphilosophy of science |
| spellingShingle | Daniele Fanelli Pedro Batista Tan Olavo B. Amaral Kleber Neves A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments Royal Society Open Science metascience metaresearch reproducibility complexity information compression philosophy of science |
| title | A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments |
| title_full | A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments |
| title_fullStr | A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments |
| title_full_unstemmed | A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments |
| title_short | A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments |
| title_sort | metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments |
| topic | metascience metaresearch reproducibility complexity information compression philosophy of science |
| url | https://royalsocietypublishing.org/doi/10.1098/rsos.241446 |
| work_keys_str_mv | AT danielefanelli ametricofknowledgeasinformationcompressionreflectsreproducibilitypredictionsforbiomedicalexperiments AT pedrobatistatan ametricofknowledgeasinformationcompressionreflectsreproducibilitypredictionsforbiomedicalexperiments AT olavobamaral ametricofknowledgeasinformationcompressionreflectsreproducibilitypredictionsforbiomedicalexperiments AT kleberneves ametricofknowledgeasinformationcompressionreflectsreproducibilitypredictionsforbiomedicalexperiments AT danielefanelli metricofknowledgeasinformationcompressionreflectsreproducibilitypredictionsforbiomedicalexperiments AT pedrobatistatan metricofknowledgeasinformationcompressionreflectsreproducibilitypredictionsforbiomedicalexperiments AT olavobamaral metricofknowledgeasinformationcompressionreflectsreproducibilitypredictionsforbiomedicalexperiments AT kleberneves metricofknowledgeasinformationcompressionreflectsreproducibilitypredictionsforbiomedicalexperiments |