Assessing how accurately large language models encode and apply the common European framework of reference for languages
Large Language Models (LLMs) can have a transformative effect on a variety of domains, including education, and it is therefore pressing to understand whether these models have knowledge of – or, in other words, how they have encoded – the specific pedagogical requirements of different educational d...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-06-01
|
| Series: | Computers and Education: Artificial Intelligence |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666920X24001565 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841545524561313792 |
|---|---|
| author | Luca Benedetto Gabrielle Gaudeau Andrew Caines Paula Buttery |
| author_facet | Luca Benedetto Gabrielle Gaudeau Andrew Caines Paula Buttery |
| author_sort | Luca Benedetto |
| collection | DOAJ |
| description | Large Language Models (LLMs) can have a transformative effect on a variety of domains, including education, and it is therefore pressing to understand whether these models have knowledge of – or, in other words, how they have encoded – the specific pedagogical requirements of different educational domains, and whether they use this when performing educational tasks. In this work, we propose an approach to evaluate the knowledge – or encoding – that the LLMs have of the Common European Framework of Reference for Languages (CEFR), and use it to evaluate five modern LLMs. Our study shows that the suite of tasks we propose is quite challenging for all the LLMs, and they often provide results which are not satisfactory and would be unusable in educational applications, suggesting that – even if they encode some information about the CEFR – this knowledge is not really leveraged when performing downstream tasks. |
| format | Article |
| id | doaj-art-9be6d70d787f4da39bddfb24cecc070a |
| institution | Kabale University |
| issn | 2666-920X |
| language | English |
| publishDate | 2025-06-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Computers and Education: Artificial Intelligence |
| spelling | doaj-art-9be6d70d787f4da39bddfb24cecc070a2025-01-12T05:26:11ZengElsevierComputers and Education: Artificial Intelligence2666-920X2025-06-018100353Assessing how accurately large language models encode and apply the common European framework of reference for languagesLuca Benedetto0Gabrielle Gaudeau1Andrew Caines2Paula Buttery3Corresponding author.; ALTA Institute, Department of Computer Science & Technology, University of Cambridge, Cambridge, United KingdomALTA Institute, Department of Computer Science & Technology, University of Cambridge, Cambridge, United KingdomALTA Institute, Department of Computer Science & Technology, University of Cambridge, Cambridge, United KingdomALTA Institute, Department of Computer Science & Technology, University of Cambridge, Cambridge, United KingdomLarge Language Models (LLMs) can have a transformative effect on a variety of domains, including education, and it is therefore pressing to understand whether these models have knowledge of – or, in other words, how they have encoded – the specific pedagogical requirements of different educational domains, and whether they use this when performing educational tasks. In this work, we propose an approach to evaluate the knowledge – or encoding – that the LLMs have of the Common European Framework of Reference for Languages (CEFR), and use it to evaluate five modern LLMs. Our study shows that the suite of tasks we propose is quite challenging for all the LLMs, and they often provide results which are not satisfactory and would be unusable in educational applications, suggesting that – even if they encode some information about the CEFR – this knowledge is not really leveraged when performing downstream tasks.http://www.sciencedirect.com/science/article/pii/S2666920X24001565Large language modelsLanguage learningCommon European Framework of Reference for Languages |
| spellingShingle | Luca Benedetto Gabrielle Gaudeau Andrew Caines Paula Buttery Assessing how accurately large language models encode and apply the common European framework of reference for languages Computers and Education: Artificial Intelligence Large language models Language learning Common European Framework of Reference for Languages |
| title | Assessing how accurately large language models encode and apply the common European framework of reference for languages |
| title_full | Assessing how accurately large language models encode and apply the common European framework of reference for languages |
| title_fullStr | Assessing how accurately large language models encode and apply the common European framework of reference for languages |
| title_full_unstemmed | Assessing how accurately large language models encode and apply the common European framework of reference for languages |
| title_short | Assessing how accurately large language models encode and apply the common European framework of reference for languages |
| title_sort | assessing how accurately large language models encode and apply the common european framework of reference for languages |
| topic | Large language models Language learning Common European Framework of Reference for Languages |
| url | http://www.sciencedirect.com/science/article/pii/S2666920X24001565 |
| work_keys_str_mv | AT lucabenedetto assessinghowaccuratelylargelanguagemodelsencodeandapplythecommoneuropeanframeworkofreferenceforlanguages AT gabriellegaudeau assessinghowaccuratelylargelanguagemodelsencodeandapplythecommoneuropeanframeworkofreferenceforlanguages AT andrewcaines assessinghowaccuratelylargelanguagemodelsencodeandapplythecommoneuropeanframeworkofreferenceforlanguages AT paulabuttery assessinghowaccuratelylargelanguagemodelsencodeandapplythecommoneuropeanframeworkofreferenceforlanguages |