Large language models meet user interfaces: The case of provisioning feedback
Incorporating Generative Artificial Intelligence (GenAI), especially Large Language Models (LLMs), into educational settings presents valuable opportunities to boost the efficiency of educators and enrich the learning experiences of students. A significant portion of the current use of LLMs by educa...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-12-01
|
| Series: | Computers and Education: Artificial Intelligence |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666920X24000924 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850250240189267968 |
|---|---|
| author | Stanislav Pozdniakov Jonathan Brazil Solmaz Abdi Aneesha Bakharia Shazia Sadiq Dragan Gašević Paul Denny Hassan Khosravi |
| author_facet | Stanislav Pozdniakov Jonathan Brazil Solmaz Abdi Aneesha Bakharia Shazia Sadiq Dragan Gašević Paul Denny Hassan Khosravi |
| author_sort | Stanislav Pozdniakov |
| collection | DOAJ |
| description | Incorporating Generative Artificial Intelligence (GenAI), especially Large Language Models (LLMs), into educational settings presents valuable opportunities to boost the efficiency of educators and enrich the learning experiences of students. A significant portion of the current use of LLMs by educators has involved using conversational user interfaces (CUIs), such as chat windows, for functions like generating educational materials or offering feedback to learners. The ability to engage in real-time conversations with LLMs, which can enhance educators' domain knowledge across various subjects, has been of high value. However, it also presents challenges to LLMs' widespread, ethical, and effective adoption. Firstly, educators must have a degree of expertise, including tool familiarity, AI literacy and prompting to effectively use CUIs, which can be a barrier to adoption. Secondly, the open-ended design of CUIs makes them exceptionally powerful, which raises ethical concerns, particularly when used for high-stakes decisions like grading. Additionally, there are risks related to privacy and intellectual property, stemming from the potential unauthorised sharing of sensitive information. Finally, CUIs are designed for short, synchronous interactions and often struggle and hallucinate when given complex, multi-step tasks (e.g., providing individual feedback based on a rubric on a large scale). To address these challenges, we explored the benefits of transitioning away from employing LLMs via CUIs to the creation of applications with user-friendly interfaces that leverage LLMs through API calls. We first propose a framework for pedagogically sound and ethically responsible incorporation of GenAI into educational tools, emphasizing a human-centred design. We then illustrate the application of our framework to the design and implementation of a novel tool called Feedback Copilot, which enables instructors to provide students with personalized qualitative feedback on their assignments in classes of any size. An evaluation involving the generation of feedback from two distinct variations of the Feedback Copilot tool, using numerically graded assignments from 338 students, demonstrates the viability and effectiveness of our approach. Our findings have significant implications for GenAI application researchers, educators seeking to leverage accessible GenAI tools, and educational technologists aiming to transcend the limitations of conversational AI interfaces, thereby charting a course for the future of GenAI in education. |
| format | Article |
| id | doaj-art-7a7f6f88a5934e2aafbf0a741edbf4bc |
| institution | OA Journals |
| issn | 2666-920X |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Computers and Education: Artificial Intelligence |
| spelling | doaj-art-7a7f6f88a5934e2aafbf0a741edbf4bc2025-08-20T01:58:16ZengElsevierComputers and Education: Artificial Intelligence2666-920X2024-12-01710028910.1016/j.caeai.2024.100289Large language models meet user interfaces: The case of provisioning feedbackStanislav Pozdniakov0Jonathan Brazil1Solmaz Abdi2Aneesha Bakharia3Shazia Sadiq4Dragan Gašević5Paul Denny6Hassan Khosravi7School of Electrical Engineering and Computer Science, The University of Queensland, St Lucia, QLD, 4072, Australia; Corresponding author.Institute for Teaching and Learning Innovation, The University of Queensland, St Lucia, QLD, 4072, AustraliaSchool of Electrical Engineering and Computer Science, The University of Queensland, St Lucia, QLD, 4072, AustraliaSchool of Electrical Engineering and Computer Science, The University of Queensland, St Lucia, QLD, 4072, AustraliaSchool of Electrical Engineering and Computer Science, The University of Queensland, St Lucia, QLD, 4072, AustraliaCentre for Learning Analytics, Faculty of Information Technology, Monash University, Melbourne, VIC, 3800, AustraliaSchool of Computer Science, University of Auckland, 38 Princes Street Auckland, 1010, New ZealandSchool of Electrical Engineering and Computer Science, The University of Queensland, St Lucia, QLD, 4072, AustraliaIncorporating Generative Artificial Intelligence (GenAI), especially Large Language Models (LLMs), into educational settings presents valuable opportunities to boost the efficiency of educators and enrich the learning experiences of students. A significant portion of the current use of LLMs by educators has involved using conversational user interfaces (CUIs), such as chat windows, for functions like generating educational materials or offering feedback to learners. The ability to engage in real-time conversations with LLMs, which can enhance educators' domain knowledge across various subjects, has been of high value. However, it also presents challenges to LLMs' widespread, ethical, and effective adoption. Firstly, educators must have a degree of expertise, including tool familiarity, AI literacy and prompting to effectively use CUIs, which can be a barrier to adoption. Secondly, the open-ended design of CUIs makes them exceptionally powerful, which raises ethical concerns, particularly when used for high-stakes decisions like grading. Additionally, there are risks related to privacy and intellectual property, stemming from the potential unauthorised sharing of sensitive information. Finally, CUIs are designed for short, synchronous interactions and often struggle and hallucinate when given complex, multi-step tasks (e.g., providing individual feedback based on a rubric on a large scale). To address these challenges, we explored the benefits of transitioning away from employing LLMs via CUIs to the creation of applications with user-friendly interfaces that leverage LLMs through API calls. We first propose a framework for pedagogically sound and ethically responsible incorporation of GenAI into educational tools, emphasizing a human-centred design. We then illustrate the application of our framework to the design and implementation of a novel tool called Feedback Copilot, which enables instructors to provide students with personalized qualitative feedback on their assignments in classes of any size. An evaluation involving the generation of feedback from two distinct variations of the Feedback Copilot tool, using numerically graded assignments from 338 students, demonstrates the viability and effectiveness of our approach. Our findings have significant implications for GenAI application researchers, educators seeking to leverage accessible GenAI tools, and educational technologists aiming to transcend the limitations of conversational AI interfaces, thereby charting a course for the future of GenAI in education.http://www.sciencedirect.com/science/article/pii/S2666920X24000924Artificial intelligenceLarge language modelsGenerative artificial intelligenceInterfacesFeedbackLearning analytics |
| spellingShingle | Stanislav Pozdniakov Jonathan Brazil Solmaz Abdi Aneesha Bakharia Shazia Sadiq Dragan Gašević Paul Denny Hassan Khosravi Large language models meet user interfaces: The case of provisioning feedback Computers and Education: Artificial Intelligence Artificial intelligence Large language models Generative artificial intelligence Interfaces Feedback Learning analytics |
| title | Large language models meet user interfaces: The case of provisioning feedback |
| title_full | Large language models meet user interfaces: The case of provisioning feedback |
| title_fullStr | Large language models meet user interfaces: The case of provisioning feedback |
| title_full_unstemmed | Large language models meet user interfaces: The case of provisioning feedback |
| title_short | Large language models meet user interfaces: The case of provisioning feedback |
| title_sort | large language models meet user interfaces the case of provisioning feedback |
| topic | Artificial intelligence Large language models Generative artificial intelligence Interfaces Feedback Learning analytics |
| url | http://www.sciencedirect.com/science/article/pii/S2666920X24000924 |
| work_keys_str_mv | AT stanislavpozdniakov largelanguagemodelsmeetuserinterfacesthecaseofprovisioningfeedback AT jonathanbrazil largelanguagemodelsmeetuserinterfacesthecaseofprovisioningfeedback AT solmazabdi largelanguagemodelsmeetuserinterfacesthecaseofprovisioningfeedback AT aneeshabakharia largelanguagemodelsmeetuserinterfacesthecaseofprovisioningfeedback AT shaziasadiq largelanguagemodelsmeetuserinterfacesthecaseofprovisioningfeedback AT dragangasevic largelanguagemodelsmeetuserinterfacesthecaseofprovisioningfeedback AT pauldenny largelanguagemodelsmeetuserinterfacesthecaseofprovisioningfeedback AT hassankhosravi largelanguagemodelsmeetuserinterfacesthecaseofprovisioningfeedback |