ChatGPT, Bard, Bing Chat, and Claude generate feedback for Chinese as foreign language writing: A comparative case study
Abstract This comparative case study analyzes and evaluates the performance of four prevalent artificial intelligence (AI) models―ChatGPT, Google Bard, Microsoft Bing, and Claude―in generating feedback on Chinese as a Foreign Language writing. The study assessed the models' effectiveness, accur...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2024-09-01
|
| Series: | Future in Educational Research |
| Subjects: | |
| Online Access: | https://doi.org/10.1002/fer3.39 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832086466386198528 |
|---|---|
| author | Saleh Obaidoon Haiping Wei |
| author_facet | Saleh Obaidoon Haiping Wei |
| author_sort | Saleh Obaidoon |
| collection | DOAJ |
| description | Abstract This comparative case study analyzes and evaluates the performance of four prevalent artificial intelligence (AI) models―ChatGPT, Google Bard, Microsoft Bing, and Claude―in generating feedback on Chinese as a Foreign Language writing. The study assessed the models' effectiveness, accuracy, alignment with pedagogical principles, and cultural appropriateness through a multi‐faceted data collection process involving student article writing, chatbot feedback, and teacher evaluation. The quantitative analysis of teacher ratings indicates that Claude demonstrated the highest average alignment with human instructor scores across the four articles, followed by Google Bard. Qualitative examination reveals differences in the types of feedback provided, with models excelling at surface‐level vocabulary, grammar, and mechanics critiques but limited in providing rhetorical, pragmatic, and structural feedback compared to teachers. While showing potential benefits, judicious integration of AI writing feedback tools upholding academic integrity is advised. This paper utilizes non‐Pro subscription plans for its research, ensuring accessibility by teachers or students without any cost. The date of access for these chatbots was September 20, 2023. The AI models used include ChatGPT based on OpenAI's GPT‐3.5 architecture with a knowledge cut‐off in January 2022, without Internet browsing capabilities; Google Bard from the Gemini family, version 1.0, which integrates internet‐based search; Microsoft Copilot (Balanced mode), which evolved from Bing Chat, providing information and content generation; and Claude version 2. This approach ensures the study's findings are applicable and replicable for educators and students utilizing freely available resources. |
| format | Article |
| id | doaj-art-d0561f0055824155baff280d5823a2ff |
| institution | Kabale University |
| issn | 2835-9402 |
| language | English |
| publishDate | 2024-09-01 |
| publisher | Wiley |
| record_format | Article |
| series | Future in Educational Research |
| spelling | doaj-art-d0561f0055824155baff280d5823a2ff2025-02-06T15:35:22ZengWileyFuture in Educational Research2835-94022024-09-012318420410.1002/fer3.39ChatGPT, Bard, Bing Chat, and Claude generate feedback for Chinese as foreign language writing: A comparative case studySaleh Obaidoon0Haiping Wei1College of Chinese Language and Literature Southwest Minzu University Chengdu ChinaCollege of Chinese Language and Literature Southwest Minzu University Chengdu ChinaAbstract This comparative case study analyzes and evaluates the performance of four prevalent artificial intelligence (AI) models―ChatGPT, Google Bard, Microsoft Bing, and Claude―in generating feedback on Chinese as a Foreign Language writing. The study assessed the models' effectiveness, accuracy, alignment with pedagogical principles, and cultural appropriateness through a multi‐faceted data collection process involving student article writing, chatbot feedback, and teacher evaluation. The quantitative analysis of teacher ratings indicates that Claude demonstrated the highest average alignment with human instructor scores across the four articles, followed by Google Bard. Qualitative examination reveals differences in the types of feedback provided, with models excelling at surface‐level vocabulary, grammar, and mechanics critiques but limited in providing rhetorical, pragmatic, and structural feedback compared to teachers. While showing potential benefits, judicious integration of AI writing feedback tools upholding academic integrity is advised. This paper utilizes non‐Pro subscription plans for its research, ensuring accessibility by teachers or students without any cost. The date of access for these chatbots was September 20, 2023. The AI models used include ChatGPT based on OpenAI's GPT‐3.5 architecture with a knowledge cut‐off in January 2022, without Internet browsing capabilities; Google Bard from the Gemini family, version 1.0, which integrates internet‐based search; Microsoft Copilot (Balanced mode), which evolved from Bing Chat, providing information and content generation; and Claude version 2. This approach ensures the study's findings are applicable and replicable for educators and students utilizing freely available resources.https://doi.org/10.1002/fer3.39AI writing feedbackChatGPTChinese as foreign languagecomparative studyGoogle Bard |
| spellingShingle | Saleh Obaidoon Haiping Wei ChatGPT, Bard, Bing Chat, and Claude generate feedback for Chinese as foreign language writing: A comparative case study Future in Educational Research AI writing feedback ChatGPT Chinese as foreign language comparative study Google Bard |
| title | ChatGPT, Bard, Bing Chat, and Claude generate feedback for Chinese as foreign language writing: A comparative case study |
| title_full | ChatGPT, Bard, Bing Chat, and Claude generate feedback for Chinese as foreign language writing: A comparative case study |
| title_fullStr | ChatGPT, Bard, Bing Chat, and Claude generate feedback for Chinese as foreign language writing: A comparative case study |
| title_full_unstemmed | ChatGPT, Bard, Bing Chat, and Claude generate feedback for Chinese as foreign language writing: A comparative case study |
| title_short | ChatGPT, Bard, Bing Chat, and Claude generate feedback for Chinese as foreign language writing: A comparative case study |
| title_sort | chatgpt bard bing chat and claude generate feedback for chinese as foreign language writing a comparative case study |
| topic | AI writing feedback ChatGPT Chinese as foreign language comparative study Google Bard |
| url | https://doi.org/10.1002/fer3.39 |
| work_keys_str_mv | AT salehobaidoon chatgptbardbingchatandclaudegeneratefeedbackforchineseasforeignlanguagewritingacomparativecasestudy AT haipingwei chatgptbardbingchatandclaudegeneratefeedbackforchineseasforeignlanguagewritingacomparativecasestudy |