A large scale randomized controlled trial on herding in peer-review discussions.
Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing submissions and making the final acceptance/rejection decisions. Given that human decision-making is known to be susceptible to various cognitive biases, it is important to un...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2023-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0287443 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849248424172453888 |
|---|---|
| author | Ivan Stelmakh Charvi Rastogi Nihar B Shah Aarti Singh Hal Daumé |
| author_facet | Ivan Stelmakh Charvi Rastogi Nihar B Shah Aarti Singh Hal Daumé |
| author_sort | Ivan Stelmakh |
| collection | DOAJ |
| description | Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing submissions and making the final acceptance/rejection decisions. Given that human decision-making is known to be susceptible to various cognitive biases, it is important to understand which (if any) biases are present in the peer-review process, and design the pipeline such that the impact of these biases is minimized. In this work, we focus on the dynamics of discussions between reviewers and investigate the presence of herding behaviour therein. Specifically, we aim to understand whether reviewers and discussion chairs get disproportionately influenced by the first argument presented in the discussion when (in case of reviewers) they form an independent opinion about the paper before discussing it with others. In conjunction with the review process of a large, top tier machine learning conference, we design and execute a randomized controlled trial that involves 1,544 papers and 2,797 reviewers with the goal of testing for the conditional causal effect of the discussion initiator's opinion on the outcome of a paper. Our experiment reveals no evidence of herding in peer-review discussions. This observation is in contrast with past work that has documented an undue influence of the first piece of information on the final decision (e.g., anchoring effect) and analyzed herding behaviour in other applications (e.g., financial markets). Regarding policy implications, the absence of the herding effect suggests that the current status quo of the absence of a unified policy towards discussion initiation does not result in an increased arbitrariness of the resulting decisions. |
| format | Article |
| id | doaj-art-ee45fb1c501b4aa4a109479a91f7b9b9 |
| institution | Kabale University |
| issn | 1932-6203 |
| language | English |
| publishDate | 2023-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-ee45fb1c501b4aa4a109479a91f7b9b92025-08-20T03:57:52ZengPublic Library of Science (PLoS)PLoS ONE1932-62032023-01-01187e028744310.1371/journal.pone.0287443A large scale randomized controlled trial on herding in peer-review discussions.Ivan StelmakhCharvi RastogiNihar B ShahAarti SinghHal DauméPeer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing submissions and making the final acceptance/rejection decisions. Given that human decision-making is known to be susceptible to various cognitive biases, it is important to understand which (if any) biases are present in the peer-review process, and design the pipeline such that the impact of these biases is minimized. In this work, we focus on the dynamics of discussions between reviewers and investigate the presence of herding behaviour therein. Specifically, we aim to understand whether reviewers and discussion chairs get disproportionately influenced by the first argument presented in the discussion when (in case of reviewers) they form an independent opinion about the paper before discussing it with others. In conjunction with the review process of a large, top tier machine learning conference, we design and execute a randomized controlled trial that involves 1,544 papers and 2,797 reviewers with the goal of testing for the conditional causal effect of the discussion initiator's opinion on the outcome of a paper. Our experiment reveals no evidence of herding in peer-review discussions. This observation is in contrast with past work that has documented an undue influence of the first piece of information on the final decision (e.g., anchoring effect) and analyzed herding behaviour in other applications (e.g., financial markets). Regarding policy implications, the absence of the herding effect suggests that the current status quo of the absence of a unified policy towards discussion initiation does not result in an increased arbitrariness of the resulting decisions.https://doi.org/10.1371/journal.pone.0287443 |
| spellingShingle | Ivan Stelmakh Charvi Rastogi Nihar B Shah Aarti Singh Hal Daumé A large scale randomized controlled trial on herding in peer-review discussions. PLoS ONE |
| title | A large scale randomized controlled trial on herding in peer-review discussions. |
| title_full | A large scale randomized controlled trial on herding in peer-review discussions. |
| title_fullStr | A large scale randomized controlled trial on herding in peer-review discussions. |
| title_full_unstemmed | A large scale randomized controlled trial on herding in peer-review discussions. |
| title_short | A large scale randomized controlled trial on herding in peer-review discussions. |
| title_sort | large scale randomized controlled trial on herding in peer review discussions |
| url | https://doi.org/10.1371/journal.pone.0287443 |
| work_keys_str_mv | AT ivanstelmakh alargescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions AT charvirastogi alargescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions AT niharbshah alargescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions AT aartisingh alargescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions AT haldaume alargescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions AT ivanstelmakh largescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions AT charvirastogi largescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions AT niharbshah largescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions AT aartisingh largescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions AT haldaume largescalerandomizedcontrolledtrialonherdinginpeerreviewdiscussions |