Evaluating ChatGPT-4's performance on oral and maxillofacial queries: Chain of Thought and standard method
ObjectivesOral and maxillofacial diseases affect approximately 3.5 billion people worldwide. With the continuous advancement of Artificial Intelligence technologies, particularly the application of generative pre-trained transformers like ChatGPT-4, there is potential to enhance public awareness of...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-02-01
|
| Series: | Frontiers in Oral Health |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/froh.2025.1541976/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1823856589734412288 |
|---|---|
| author | Kaiyuan Ji Zhihan Wu Jing Han Guangtao Zhai Guangtao Zhai Jiannan Liu |
| author_facet | Kaiyuan Ji Zhihan Wu Jing Han Guangtao Zhai Guangtao Zhai Jiannan Liu |
| author_sort | Kaiyuan Ji |
| collection | DOAJ |
| description | ObjectivesOral and maxillofacial diseases affect approximately 3.5 billion people worldwide. With the continuous advancement of Artificial Intelligence technologies, particularly the application of generative pre-trained transformers like ChatGPT-4, there is potential to enhance public awareness of the prevention and early detection of these diseases. This study evaluated the performance of ChatGPT-4 in addressing oral and maxillofacial disease questions using standard approaches and the Chain of Thought (CoT) method, aiming to gain a deeper understanding of its capabilities, potential, and limitations.Materials and methodsThree experts, drawing from their extensive experience and the most common questions in clinical settings, selected 130 open-ended questions and 1,805 multiple-choice questions from the national dental licensing examination. These questions encompass 12 areas of oral and maxillofacial surgery, including Prosthodontics, Pediatric Dentistry, Maxillofacial Tumors and Salivary Gland Diseases, and maxillofacial Infections.ResultsUsing CoT approach, ChatGPT-4 exhibited marked enhancements in accuracy, structure, completeness, professionalism, and overall impression for open-ended questions, revealing statistically significant differences compared to its performance on general oral and maxillofacial inquiries. In the realm of multiple-choice questions, the application of CoT method boosted ChatGPT-4's accuracy across all major subjects, achieving an overall accuracy increase of 3.1%.ConclusionsWhen employing ChatGPT-4 to address questions in oral and maxillofacial surgery, incorporating CoT as a querying method can enhance its performance and help the public improve their understanding and awareness of such issues. However, it is not advisable to consider it a substitute for doctors. |
| format | Article |
| id | doaj-art-6a05acc2a1754104ac309ff342c68da7 |
| institution | Kabale University |
| issn | 2673-4842 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Oral Health |
| spelling | doaj-art-6a05acc2a1754104ac309ff342c68da72025-02-12T07:25:31ZengFrontiers Media S.A.Frontiers in Oral Health2673-48422025-02-01610.3389/froh.2025.15419761541976Evaluating ChatGPT-4's performance on oral and maxillofacial queries: Chain of Thought and standard methodKaiyuan Ji0Zhihan Wu1Jing Han2Guangtao Zhai3Guangtao Zhai4Jiannan Liu5School of Communication and Electronic Engineering, East China Normal University, Shanghai, ChinaDepartment of Oral and Maxillofacial Head and Neck Oncology, Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, ChinaDepartment of Oral and Maxillofacial Head and Neck Oncology, Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, ChinaSchool of Communication and Electronic Engineering, East China Normal University, Shanghai, ChinaSchool of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, ChinaDepartment of Oral and Maxillofacial Head and Neck Oncology, Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, ChinaObjectivesOral and maxillofacial diseases affect approximately 3.5 billion people worldwide. With the continuous advancement of Artificial Intelligence technologies, particularly the application of generative pre-trained transformers like ChatGPT-4, there is potential to enhance public awareness of the prevention and early detection of these diseases. This study evaluated the performance of ChatGPT-4 in addressing oral and maxillofacial disease questions using standard approaches and the Chain of Thought (CoT) method, aiming to gain a deeper understanding of its capabilities, potential, and limitations.Materials and methodsThree experts, drawing from their extensive experience and the most common questions in clinical settings, selected 130 open-ended questions and 1,805 multiple-choice questions from the national dental licensing examination. These questions encompass 12 areas of oral and maxillofacial surgery, including Prosthodontics, Pediatric Dentistry, Maxillofacial Tumors and Salivary Gland Diseases, and maxillofacial Infections.ResultsUsing CoT approach, ChatGPT-4 exhibited marked enhancements in accuracy, structure, completeness, professionalism, and overall impression for open-ended questions, revealing statistically significant differences compared to its performance on general oral and maxillofacial inquiries. In the realm of multiple-choice questions, the application of CoT method boosted ChatGPT-4's accuracy across all major subjects, achieving an overall accuracy increase of 3.1%.ConclusionsWhen employing ChatGPT-4 to address questions in oral and maxillofacial surgery, incorporating CoT as a querying method can enhance its performance and help the public improve their understanding and awareness of such issues. However, it is not advisable to consider it a substitute for doctors.https://www.frontiersin.org/articles/10.3389/froh.2025.1541976/fullArtificial IntelligenceChain of Thoughteducation toolChatGPT-4oral and maxillofacial |
| spellingShingle | Kaiyuan Ji Zhihan Wu Jing Han Guangtao Zhai Guangtao Zhai Jiannan Liu Evaluating ChatGPT-4's performance on oral and maxillofacial queries: Chain of Thought and standard method Frontiers in Oral Health Artificial Intelligence Chain of Thought education tool ChatGPT-4 oral and maxillofacial |
| title | Evaluating ChatGPT-4's performance on oral and maxillofacial queries: Chain of Thought and standard method |
| title_full | Evaluating ChatGPT-4's performance on oral and maxillofacial queries: Chain of Thought and standard method |
| title_fullStr | Evaluating ChatGPT-4's performance on oral and maxillofacial queries: Chain of Thought and standard method |
| title_full_unstemmed | Evaluating ChatGPT-4's performance on oral and maxillofacial queries: Chain of Thought and standard method |
| title_short | Evaluating ChatGPT-4's performance on oral and maxillofacial queries: Chain of Thought and standard method |
| title_sort | evaluating chatgpt 4 s performance on oral and maxillofacial queries chain of thought and standard method |
| topic | Artificial Intelligence Chain of Thought education tool ChatGPT-4 oral and maxillofacial |
| url | https://www.frontiersin.org/articles/10.3389/froh.2025.1541976/full |
| work_keys_str_mv | AT kaiyuanji evaluatingchatgpt4sperformanceonoralandmaxillofacialquerieschainofthoughtandstandardmethod AT zhihanwu evaluatingchatgpt4sperformanceonoralandmaxillofacialquerieschainofthoughtandstandardmethod AT jinghan evaluatingchatgpt4sperformanceonoralandmaxillofacialquerieschainofthoughtandstandardmethod AT guangtaozhai evaluatingchatgpt4sperformanceonoralandmaxillofacialquerieschainofthoughtandstandardmethod AT guangtaozhai evaluatingchatgpt4sperformanceonoralandmaxillofacialquerieschainofthoughtandstandardmethod AT jiannanliu evaluatingchatgpt4sperformanceonoralandmaxillofacialquerieschainofthoughtandstandardmethod |