Language models outperform cloze predictability in a cognitive model of reading.
Although word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely capture r...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2024-09-01
|
| Series: | PLoS Computational Biology |
| Online Access: | https://doi.org/10.1371/journal.pcbi.1012117 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850250843651047424 |
|---|---|
| author | Adrielli Tina Lopes Rego Joshua Snell Martijn Meeter |
| author_facet | Adrielli Tina Lopes Rego Joshua Snell Martijn Meeter |
| author_sort | Adrielli Tina Lopes Rego |
| collection | DOAJ |
| description | Although word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely capture remains unclear. This study investigates whether large language models (LLMs) can fill this gap. Contextual predictions are implemented via a novel parallel-graded mechanism, where all predicted words at a given position are pre-activated as a function of contextual certainty, which varies dynamically as text processing unfolds. Through reading simulations with OB1-reader, a cognitive model of word recognition and eye-movement control in reading, we compare the model's fit to eye-movement data when using predictability values derived from a cloze task against those derived from LLMs (GPT-2 and LLaMA). Root Mean Square Error between simulated and human eye movements indicates that LLM predictability provides a better fit than cloze. This is the first study to use LLMs to augment a cognitive model of reading with higher-order language processing while proposing a mechanism on the interplay between word predictability and eye movements. |
| format | Article |
| id | doaj-art-fe3098f9dfd2402088a5afd7c1ec4437 |
| institution | OA Journals |
| issn | 1553-734X 1553-7358 |
| language | English |
| publishDate | 2024-09-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Computational Biology |
| spelling | doaj-art-fe3098f9dfd2402088a5afd7c1ec44372025-08-20T01:58:04ZengPublic Library of Science (PLoS)PLoS Computational Biology1553-734X1553-73582024-09-01209e101211710.1371/journal.pcbi.1012117Language models outperform cloze predictability in a cognitive model of reading.Adrielli Tina Lopes RegoJoshua SnellMartijn MeeterAlthough word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely capture remains unclear. This study investigates whether large language models (LLMs) can fill this gap. Contextual predictions are implemented via a novel parallel-graded mechanism, where all predicted words at a given position are pre-activated as a function of contextual certainty, which varies dynamically as text processing unfolds. Through reading simulations with OB1-reader, a cognitive model of word recognition and eye-movement control in reading, we compare the model's fit to eye-movement data when using predictability values derived from a cloze task against those derived from LLMs (GPT-2 and LLaMA). Root Mean Square Error between simulated and human eye movements indicates that LLM predictability provides a better fit than cloze. This is the first study to use LLMs to augment a cognitive model of reading with higher-order language processing while proposing a mechanism on the interplay between word predictability and eye movements.https://doi.org/10.1371/journal.pcbi.1012117 |
| spellingShingle | Adrielli Tina Lopes Rego Joshua Snell Martijn Meeter Language models outperform cloze predictability in a cognitive model of reading. PLoS Computational Biology |
| title | Language models outperform cloze predictability in a cognitive model of reading. |
| title_full | Language models outperform cloze predictability in a cognitive model of reading. |
| title_fullStr | Language models outperform cloze predictability in a cognitive model of reading. |
| title_full_unstemmed | Language models outperform cloze predictability in a cognitive model of reading. |
| title_short | Language models outperform cloze predictability in a cognitive model of reading. |
| title_sort | language models outperform cloze predictability in a cognitive model of reading |
| url | https://doi.org/10.1371/journal.pcbi.1012117 |
| work_keys_str_mv | AT adriellitinalopesrego languagemodelsoutperformclozepredictabilityinacognitivemodelofreading AT joshuasnell languagemodelsoutperformclozepredictabilityinacognitivemodelofreading AT martijnmeeter languagemodelsoutperformclozepredictabilityinacognitivemodelofreading |