Evaluating science: A comparison of human and AI reviewers
Scientists have started to explore whether novel artificial intelligence (AI) tools based on large language models, such as GPT-4, could support the scientific peer review process. We sought to understand (i) whether AI versus human reviewers are able to distinguish between made-up AI-generated and...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2024-01-01
|
| Series: | Judgment and Decision Making |
| Subjects: | |
| Online Access: | https://www.cambridge.org/core/product/identifier/S193029752400024X/type/journal_article |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846161412044357632 |
|---|---|
| author | Anna Shcherbiak Hooman Habibnia Robert Böhm Susann Fiedler |
| author_facet | Anna Shcherbiak Hooman Habibnia Robert Böhm Susann Fiedler |
| author_sort | Anna Shcherbiak |
| collection | DOAJ |
| description | Scientists have started to explore whether novel artificial intelligence (AI) tools based on large language models, such as GPT-4, could support the scientific peer review process. We sought to understand (i) whether AI versus human reviewers are able to distinguish between made-up AI-generated and human-written conference abstracts reporting on actual research, and (ii) how the quality assessments by AI versus human reviewers of the reported research correspond to each other. We conducted a large-scale field experiment during a medium-sized scientific conference, relying on 305 human-written and 20 AI-written abstracts that were reviewed either by AI or 217 human reviewers. The results show that human reviewers and GPTZero were better in discerning (AI vs. human) authorship than GPT-4. Regarding quality assessments, there was rather low agreement between both human–human and human–AI reviewer pairs, but AI reviewers were more aligned with human reviewers in classifying the very best abstracts. This indicates that AI could become a prescreening tool for scientific abstracts. The results are discussed with regard to the future development and use of AI tools during the scientific peer review process. |
| format | Article |
| id | doaj-art-77b7e2b6ba6046d9915c704b5d20d1be |
| institution | Kabale University |
| issn | 1930-2975 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| series | Judgment and Decision Making |
| spelling | doaj-art-77b7e2b6ba6046d9915c704b5d20d1be2024-11-21T07:18:15ZengCambridge University PressJudgment and Decision Making1930-29752024-01-011910.1017/jdm.2024.24Evaluating science: A comparison of human and AI reviewersAnna Shcherbiak0Hooman Habibnia1https://orcid.org/0009-0000-3915-2624Robert Böhm2Susann Fiedler3Institute for Cognition and Behavior, WU Vienna University of Economics and Business, Vienna, AustriaInstitute for Cognition and Behavior, WU Vienna University of Economics and Business, Vienna, AustriaFaculty of Psychology, University of Vienna, Vienna, Austria Department of Psychology and Copenhagen Center for Social Data Science (SODAS), University of Copenhagen, Copenhagen, DenmarkInstitute for Cognition and Behavior, WU Vienna University of Economics and Business, Vienna, AustriaScientists have started to explore whether novel artificial intelligence (AI) tools based on large language models, such as GPT-4, could support the scientific peer review process. We sought to understand (i) whether AI versus human reviewers are able to distinguish between made-up AI-generated and human-written conference abstracts reporting on actual research, and (ii) how the quality assessments by AI versus human reviewers of the reported research correspond to each other. We conducted a large-scale field experiment during a medium-sized scientific conference, relying on 305 human-written and 20 AI-written abstracts that were reviewed either by AI or 217 human reviewers. The results show that human reviewers and GPTZero were better in discerning (AI vs. human) authorship than GPT-4. Regarding quality assessments, there was rather low agreement between both human–human and human–AI reviewer pairs, but AI reviewers were more aligned with human reviewers in classifying the very best abstracts. This indicates that AI could become a prescreening tool for scientific abstracts. The results are discussed with regard to the future development and use of AI tools during the scientific peer review process.https://www.cambridge.org/core/product/identifier/S193029752400024X/type/journal_articleartificial intelligencelarge language modelspeer reviewquality assessment |
| spellingShingle | Anna Shcherbiak Hooman Habibnia Robert Böhm Susann Fiedler Evaluating science: A comparison of human and AI reviewers Judgment and Decision Making artificial intelligence large language models peer review quality assessment |
| title | Evaluating science: A comparison of human and AI reviewers |
| title_full | Evaluating science: A comparison of human and AI reviewers |
| title_fullStr | Evaluating science: A comparison of human and AI reviewers |
| title_full_unstemmed | Evaluating science: A comparison of human and AI reviewers |
| title_short | Evaluating science: A comparison of human and AI reviewers |
| title_sort | evaluating science a comparison of human and ai reviewers |
| topic | artificial intelligence large language models peer review quality assessment |
| url | https://www.cambridge.org/core/product/identifier/S193029752400024X/type/journal_article |
| work_keys_str_mv | AT annashcherbiak evaluatingscienceacomparisonofhumanandaireviewers AT hoomanhabibnia evaluatingscienceacomparisonofhumanandaireviewers AT robertbohm evaluatingscienceacomparisonofhumanandaireviewers AT susannfiedler evaluatingscienceacomparisonofhumanandaireviewers |