Enhancing Critical Writing Through AI Feedback: A Randomized Control Study
This study investigates the effectiveness of artificial intelligence-generated content (AIGC) systems on undergraduate writing development through a randomized controlled trial with 259 Chinese students. Despite promising applications of AI in educational settings, empirical evidence regarding its c...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-04-01
|
| Series: | Behavioral Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-328X/15/5/600 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850257107368017920 |
|---|---|
| author | Kai Zhang |
| author_facet | Kai Zhang |
| author_sort | Kai Zhang |
| collection | DOAJ |
| description | This study investigates the effectiveness of artificial intelligence-generated content (AIGC) systems on undergraduate writing development through a randomized controlled trial with 259 Chinese students. Despite promising applications of AI in educational settings, empirical evidence regarding its comparative effectiveness in writing instruction remains limited. Using a four-week intervention comparing Qwen-powered AI feedback to traditional instructor feedback, we employed difference-in-differences (DiD) analysis and structural equation modeling to examine how technology acceptance factors influence writing outcomes. Results demonstrated significant improvements in the AIGC intervention group compared to controls (β = 0.149, <i>p</i> < 0.001), with particularly strong effects on organization (β = 0.311, <i>p</i> < 0.001) and content development (β = 0.191, <i>p</i> < 0.001). Path analysis revealed that perceived usefulness fully mediated the relationship between perceived ease of use and attitudes toward the system (β = 0.326, <i>p</i> < 0.001), with attitudes strongly predicting behavioral engagement (β = 0.431, <i>p</i> < 0.001). Contrary to traditional technology acceptance models, perceived ease of use showed no direct effect on attitudes, suggesting that students prioritize functional benefits over interface simplicity in educational technology contexts. These findings contribute to an expanded technology acceptance model for educational settings while providing evidence-based guidelines for implementing AI writing assistants in higher education. |
| format | Article |
| id | doaj-art-b3f16b6aefd24fb99c0b7fe70c8bb93e |
| institution | OA Journals |
| issn | 2076-328X |
| language | English |
| publishDate | 2025-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Behavioral Sciences |
| spelling | doaj-art-b3f16b6aefd24fb99c0b7fe70c8bb93e2025-08-20T01:56:29ZengMDPI AGBehavioral Sciences2076-328X2025-04-0115560010.3390/bs15050600Enhancing Critical Writing Through AI Feedback: A Randomized Control StudyKai Zhang0Informatization Office, Fudan University, Shanghai 200433, ChinaThis study investigates the effectiveness of artificial intelligence-generated content (AIGC) systems on undergraduate writing development through a randomized controlled trial with 259 Chinese students. Despite promising applications of AI in educational settings, empirical evidence regarding its comparative effectiveness in writing instruction remains limited. Using a four-week intervention comparing Qwen-powered AI feedback to traditional instructor feedback, we employed difference-in-differences (DiD) analysis and structural equation modeling to examine how technology acceptance factors influence writing outcomes. Results demonstrated significant improvements in the AIGC intervention group compared to controls (β = 0.149, <i>p</i> < 0.001), with particularly strong effects on organization (β = 0.311, <i>p</i> < 0.001) and content development (β = 0.191, <i>p</i> < 0.001). Path analysis revealed that perceived usefulness fully mediated the relationship between perceived ease of use and attitudes toward the system (β = 0.326, <i>p</i> < 0.001), with attitudes strongly predicting behavioral engagement (β = 0.431, <i>p</i> < 0.001). Contrary to traditional technology acceptance models, perceived ease of use showed no direct effect on attitudes, suggesting that students prioritize functional benefits over interface simplicity in educational technology contexts. These findings contribute to an expanded technology acceptance model for educational settings while providing evidence-based guidelines for implementing AI writing assistants in higher education.https://www.mdpi.com/2076-328X/15/5/600artificial intelligence-generated contentcritical writingtechnology acceptance modeleducational intervention |
| spellingShingle | Kai Zhang Enhancing Critical Writing Through AI Feedback: A Randomized Control Study Behavioral Sciences artificial intelligence-generated content critical writing technology acceptance model educational intervention |
| title | Enhancing Critical Writing Through AI Feedback: A Randomized Control Study |
| title_full | Enhancing Critical Writing Through AI Feedback: A Randomized Control Study |
| title_fullStr | Enhancing Critical Writing Through AI Feedback: A Randomized Control Study |
| title_full_unstemmed | Enhancing Critical Writing Through AI Feedback: A Randomized Control Study |
| title_short | Enhancing Critical Writing Through AI Feedback: A Randomized Control Study |
| title_sort | enhancing critical writing through ai feedback a randomized control study |
| topic | artificial intelligence-generated content critical writing technology acceptance model educational intervention |
| url | https://www.mdpi.com/2076-328X/15/5/600 |
| work_keys_str_mv | AT kaizhang enhancingcriticalwritingthroughaifeedbackarandomizedcontrolstudy |