Enhancing student reflections with natural language processing based scaffolding: A quasi-experimental study in a large lecture course
Multiple studies have shown that scaffolding plays an important role in regulating and enhancing students' metacognitive monitoring and reflections. However, scaffolding students' reflections in large courses is a major challenge. In the current study, we explored how real-time, technology...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-06-01
|
| Series: | Computers and Education: Artificial Intelligence |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666920X25000372 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849468199731462144 |
|---|---|
| author | Muhsin Menekse Alfa Satya Putra Jiwon Kim Ahmed Ashraf Butt Mark A. McDaniel Ido Davidesco Michelle Cadieux Joe Kim Diane Litman |
| author_facet | Muhsin Menekse Alfa Satya Putra Jiwon Kim Ahmed Ashraf Butt Mark A. McDaniel Ido Davidesco Michelle Cadieux Joe Kim Diane Litman |
| author_sort | Muhsin Menekse |
| collection | DOAJ |
| description | Multiple studies have shown that scaffolding plays an important role in regulating and enhancing students' metacognitive monitoring and reflections. However, scaffolding students' reflections in large courses is a major challenge. In the current study, we explored how real-time, technology-enhanced scaffolding affects the quality of students' reflections and academic performance. Two major research questions are: RQ1) Do students in the scaffolding condition construct more specific reflections than those in the non-scaffolding condition? RQ2) How do the scaffolding feature, reflection specificity, and the number of reflections relate to students' academic performance? To address these questions, we conducted a quasi-experimental study with a large sample of undergraduate students (N = 1268) in an introductory psychology course. We designed and used a mobile application called CourseMIRROR that prompts students to reflect on what they found confusing and interesting in the lecture. The app uses Natural Language Processing (NLP) algorithms to evaluate students' reflection quality and specificity using a 4-point scale, with 1 indicating shallow reflection and 4 indicating highly relevant or specific reflection. Course sections were randomly assigned into scaffolded or non-scaffolded conditions. Students in the scaffolded condition were provided an app version with the scaffolding feature, while students in the non-scaffolded condition were provided a different version of the app without scaffolding. Regarding RQ1, we found that students in the scaffolded condition wrote significantly more specific reflections on confusing and interesting concepts. For RQ2, results showed that the number of reflections was a significant predictor of academic performance. |
| format | Article |
| id | doaj-art-be6e7ccd73db4fd48526c5e4985b9e78 |
| institution | Kabale University |
| issn | 2666-920X |
| language | English |
| publishDate | 2025-06-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Computers and Education: Artificial Intelligence |
| spelling | doaj-art-be6e7ccd73db4fd48526c5e4985b9e782025-08-20T03:25:54ZengElsevierComputers and Education: Artificial Intelligence2666-920X2025-06-01810039710.1016/j.caeai.2025.100397Enhancing student reflections with natural language processing based scaffolding: A quasi-experimental study in a large lecture courseMuhsin Menekse0Alfa Satya Putra1Jiwon Kim2Ahmed Ashraf Butt3Mark A. McDaniel4Ido Davidesco5Michelle Cadieux6Joe Kim7Diane Litman8School of Engineering Education, Purdue University, USA; Department of Curriculum and Instruction, Purdue University, USA; Corresponding author. Neil Armstrong Hall of Engineering, Room 1311, West Lafayette, IN, 47907, USA.School of Engineering Education, Purdue University, USASchool of Engineering Education, Purdue University, USAPolytechnic Institute, The University of Oklahoma, USADepartment of Psychological and Brain Sciences, Washington University in St. Louis, USALynch School of Education, Boston College, USADepartment of Psychology, Neuroscience & Behaviour, McMaster University, CanadaDepartment of Psychology, Neuroscience & Behaviour, McMaster University, CanadaDepartment of Computer Science, University of Pittsburgh, USAMultiple studies have shown that scaffolding plays an important role in regulating and enhancing students' metacognitive monitoring and reflections. However, scaffolding students' reflections in large courses is a major challenge. In the current study, we explored how real-time, technology-enhanced scaffolding affects the quality of students' reflections and academic performance. Two major research questions are: RQ1) Do students in the scaffolding condition construct more specific reflections than those in the non-scaffolding condition? RQ2) How do the scaffolding feature, reflection specificity, and the number of reflections relate to students' academic performance? To address these questions, we conducted a quasi-experimental study with a large sample of undergraduate students (N = 1268) in an introductory psychology course. We designed and used a mobile application called CourseMIRROR that prompts students to reflect on what they found confusing and interesting in the lecture. The app uses Natural Language Processing (NLP) algorithms to evaluate students' reflection quality and specificity using a 4-point scale, with 1 indicating shallow reflection and 4 indicating highly relevant or specific reflection. Course sections were randomly assigned into scaffolded or non-scaffolded conditions. Students in the scaffolded condition were provided an app version with the scaffolding feature, while students in the non-scaffolded condition were provided a different version of the app without scaffolding. Regarding RQ1, we found that students in the scaffolded condition wrote significantly more specific reflections on confusing and interesting concepts. For RQ2, results showed that the number of reflections was a significant predictor of academic performance.http://www.sciencedirect.com/science/article/pii/S2666920X25000372ScaffoldingReflectionsNatural language processingLarge-lecturesMobileTechnology |
| spellingShingle | Muhsin Menekse Alfa Satya Putra Jiwon Kim Ahmed Ashraf Butt Mark A. McDaniel Ido Davidesco Michelle Cadieux Joe Kim Diane Litman Enhancing student reflections with natural language processing based scaffolding: A quasi-experimental study in a large lecture course Computers and Education: Artificial Intelligence Scaffolding Reflections Natural language processing Large-lectures Mobile Technology |
| title | Enhancing student reflections with natural language processing based scaffolding: A quasi-experimental study in a large lecture course |
| title_full | Enhancing student reflections with natural language processing based scaffolding: A quasi-experimental study in a large lecture course |
| title_fullStr | Enhancing student reflections with natural language processing based scaffolding: A quasi-experimental study in a large lecture course |
| title_full_unstemmed | Enhancing student reflections with natural language processing based scaffolding: A quasi-experimental study in a large lecture course |
| title_short | Enhancing student reflections with natural language processing based scaffolding: A quasi-experimental study in a large lecture course |
| title_sort | enhancing student reflections with natural language processing based scaffolding a quasi experimental study in a large lecture course |
| topic | Scaffolding Reflections Natural language processing Large-lectures Mobile Technology |
| url | http://www.sciencedirect.com/science/article/pii/S2666920X25000372 |
| work_keys_str_mv | AT muhsinmenekse enhancingstudentreflectionswithnaturallanguageprocessingbasedscaffoldingaquasiexperimentalstudyinalargelecturecourse AT alfasatyaputra enhancingstudentreflectionswithnaturallanguageprocessingbasedscaffoldingaquasiexperimentalstudyinalargelecturecourse AT jiwonkim enhancingstudentreflectionswithnaturallanguageprocessingbasedscaffoldingaquasiexperimentalstudyinalargelecturecourse AT ahmedashrafbutt enhancingstudentreflectionswithnaturallanguageprocessingbasedscaffoldingaquasiexperimentalstudyinalargelecturecourse AT markamcdaniel enhancingstudentreflectionswithnaturallanguageprocessingbasedscaffoldingaquasiexperimentalstudyinalargelecturecourse AT idodavidesco enhancingstudentreflectionswithnaturallanguageprocessingbasedscaffoldingaquasiexperimentalstudyinalargelecturecourse AT michellecadieux enhancingstudentreflectionswithnaturallanguageprocessingbasedscaffoldingaquasiexperimentalstudyinalargelecturecourse AT joekim enhancingstudentreflectionswithnaturallanguageprocessingbasedscaffoldingaquasiexperimentalstudyinalargelecturecourse AT dianelitman enhancingstudentreflectionswithnaturallanguageprocessingbasedscaffoldingaquasiexperimentalstudyinalargelecturecourse |