How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions?
How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we surveyed the authors on three questions: (i) their predicted proba...
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2024-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0300710 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850069989403394048 |
|---|---|
| author | Charvi Rastogi Ivan Stelmakh Alina Beygelzimer Yann N Dauphin Percy Liang Jennifer Wortman Vaughan Zhenyu Xue Hal Daumé Iii Emma Pierson Nihar B Shah |
| author_facet | Charvi Rastogi Ivan Stelmakh Alina Beygelzimer Yann N Dauphin Percy Liang Jennifer Wortman Vaughan Zhenyu Xue Hal Daumé Iii Emma Pierson Nihar B Shah |
| author_sort | Charvi Rastogi |
| collection | DOAJ |
| description | How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we surveyed the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews. The salient results are: (1) Authors had roughly a three-fold overestimate of the acceptance probability of their papers: The median prediction was 70% for an approximately 25% acceptance rate. (2) Female authors exhibited a marginally higher (statistically significant) miscalibration than male authors; predictions of authors invited to serve as meta-reviewers or reviewers were similarly calibrated, but better than authors who were not invited to review. (3) Authors' relative ranking of scientific contribution of two submissions they made generally agreed with their predicted acceptance probabilities (93% agreement), but there was a notable 7% responses where authors predicted a worse outcome for their better paper. (4) The author-provided rankings disagreed with the peer-review decisions about a third of the time; when co-authors ranked their jointly authored papers, co-authors disagreed at a similar rate-about a third of the time. (5) At least 30% of respondents of both accepted and rejected papers said that their perception of their own paper improved after the review process. The stakeholders in peer review should take these findings into account in setting their expectations from peer review. |
| format | Article |
| id | doaj-art-71cb7dcdf65a420d86a4fd1696eb9441 |
| institution | DOAJ |
| issn | 1932-6203 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-71cb7dcdf65a420d86a4fd1696eb94412025-08-20T02:47:38ZengPublic Library of Science (PLoS)PLoS ONE1932-62032024-01-01194e030071010.1371/journal.pone.0300710How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions?Charvi RastogiIvan StelmakhAlina BeygelzimerYann N DauphinPercy LiangJennifer Wortman VaughanZhenyu XueHal Daumé IiiEmma PiersonNihar B ShahHow do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we surveyed the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews. The salient results are: (1) Authors had roughly a three-fold overestimate of the acceptance probability of their papers: The median prediction was 70% for an approximately 25% acceptance rate. (2) Female authors exhibited a marginally higher (statistically significant) miscalibration than male authors; predictions of authors invited to serve as meta-reviewers or reviewers were similarly calibrated, but better than authors who were not invited to review. (3) Authors' relative ranking of scientific contribution of two submissions they made generally agreed with their predicted acceptance probabilities (93% agreement), but there was a notable 7% responses where authors predicted a worse outcome for their better paper. (4) The author-provided rankings disagreed with the peer-review decisions about a third of the time; when co-authors ranked their jointly authored papers, co-authors disagreed at a similar rate-about a third of the time. (5) At least 30% of respondents of both accepted and rejected papers said that their perception of their own paper improved after the review process. The stakeholders in peer review should take these findings into account in setting their expectations from peer review.https://doi.org/10.1371/journal.pone.0300710 |
| spellingShingle | Charvi Rastogi Ivan Stelmakh Alina Beygelzimer Yann N Dauphin Percy Liang Jennifer Wortman Vaughan Zhenyu Xue Hal Daumé Iii Emma Pierson Nihar B Shah How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions? PLoS ONE |
| title | How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions? |
| title_full | How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions? |
| title_fullStr | How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions? |
| title_full_unstemmed | How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions? |
| title_short | How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions? |
| title_sort | how do authors perceptions of their papers compare with co authors perceptions and peer review decisions |
| url | https://doi.org/10.1371/journal.pone.0300710 |
| work_keys_str_mv | AT charvirastogi howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions AT ivanstelmakh howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions AT alinabeygelzimer howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions AT yannndauphin howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions AT percyliang howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions AT jenniferwortmanvaughan howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions AT zhenyuxue howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions AT haldaumeiii howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions AT emmapierson howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions AT niharbshah howdoauthorsperceptionsoftheirpaperscomparewithcoauthorsperceptionsandpeerreviewdecisions |