The utility of generative artificial intelligence Chatbot (ChatGPT) in generating teaching and learning material for anesthesiology residents
The popularization of large language chatbots such as ChatGPT has led to growing utility in various biomedical fields. It has been shown that chatbots can provide reasonably accurate responses to medical exam style questions. On the other hand, chatbots have known limitations which may hinder their...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-05-01
|
| Series: | Frontiers in Artificial Intelligence |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frai.2025.1582096/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850261621545369600 |
|---|---|
| author | Zhaosheng Jin Ramon Abola Vincent Bargnes Alexandra Tsivitis Sadiq Rahman Jonathon Schwartz Sergio D. Bergese Joy E. Schabel |
| author_facet | Zhaosheng Jin Ramon Abola Vincent Bargnes Alexandra Tsivitis Sadiq Rahman Jonathon Schwartz Sergio D. Bergese Joy E. Schabel |
| author_sort | Zhaosheng Jin |
| collection | DOAJ |
| description | The popularization of large language chatbots such as ChatGPT has led to growing utility in various biomedical fields. It has been shown that chatbots can provide reasonably accurate responses to medical exam style questions. On the other hand, chatbots have known limitations which may hinder their utility in medical education. We conducted a pragmatically designed study to evaluate the accuracy and completeness of ChatGPT generated responses to various styles of prompts, based on entry-level anesthesiology topics. Ninety-five unique prompts were constructed using topics from the Anesthesia Knowledge Test 1 (AKT-1), a standardized exam undertaken by US anesthesiology residents after 1 month of specialty training. A combination of focused and open-ended prompts was used to evaluate its ability to present and organize information. We also included prompts for journal references, lecture outlines, as well as biased (medically inaccurate) prompts. The responses were independently scored using a 3-point Likert scale, by two board-certified anesthesiologists with extensive experience in medical education. Fifty-two (55%) responses were rated as completely accurate by both evaluators. For longer responses prompts, most of the responses were also deemed complete. Notably, the chatbot frequently generated inaccurate responses when asked for specific literature references and when the input prompt contained deliberate errors (biased prompts). Another recurring observation was the conflation of adjacent concepts (e.g., a specific characteristic was attributed to the wrong drug under the same pharmacological class). Some of the inaccuracies could potentially result in significant harm if applied to clinical situations. While chatbots such as ChatGPT can generate medically accurate responses in most cases, its reliability is not yet suited for medical and clinical education. Content generated by ChatGPT and other chatbots will require validation prior to use. |
| format | Article |
| id | doaj-art-5f4a5ecf49dc49c5a16f65ac0b3450d0 |
| institution | OA Journals |
| issn | 2624-8212 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Artificial Intelligence |
| spelling | doaj-art-5f4a5ecf49dc49c5a16f65ac0b3450d02025-08-20T01:55:21ZengFrontiers Media S.A.Frontiers in Artificial Intelligence2624-82122025-05-01810.3389/frai.2025.15820961582096The utility of generative artificial intelligence Chatbot (ChatGPT) in generating teaching and learning material for anesthesiology residentsZhaosheng JinRamon AbolaVincent BargnesAlexandra TsivitisSadiq RahmanJonathon SchwartzSergio D. BergeseJoy E. SchabelThe popularization of large language chatbots such as ChatGPT has led to growing utility in various biomedical fields. It has been shown that chatbots can provide reasonably accurate responses to medical exam style questions. On the other hand, chatbots have known limitations which may hinder their utility in medical education. We conducted a pragmatically designed study to evaluate the accuracy and completeness of ChatGPT generated responses to various styles of prompts, based on entry-level anesthesiology topics. Ninety-five unique prompts were constructed using topics from the Anesthesia Knowledge Test 1 (AKT-1), a standardized exam undertaken by US anesthesiology residents after 1 month of specialty training. A combination of focused and open-ended prompts was used to evaluate its ability to present and organize information. We also included prompts for journal references, lecture outlines, as well as biased (medically inaccurate) prompts. The responses were independently scored using a 3-point Likert scale, by two board-certified anesthesiologists with extensive experience in medical education. Fifty-two (55%) responses were rated as completely accurate by both evaluators. For longer responses prompts, most of the responses were also deemed complete. Notably, the chatbot frequently generated inaccurate responses when asked for specific literature references and when the input prompt contained deliberate errors (biased prompts). Another recurring observation was the conflation of adjacent concepts (e.g., a specific characteristic was attributed to the wrong drug under the same pharmacological class). Some of the inaccuracies could potentially result in significant harm if applied to clinical situations. While chatbots such as ChatGPT can generate medically accurate responses in most cases, its reliability is not yet suited for medical and clinical education. Content generated by ChatGPT and other chatbots will require validation prior to use.https://www.frontiersin.org/articles/10.3389/frai.2025.1582096/fullartificial intelligencegraduate medical educationlarge language modelanesthesiology residencygenerative AI |
| spellingShingle | Zhaosheng Jin Ramon Abola Vincent Bargnes Alexandra Tsivitis Sadiq Rahman Jonathon Schwartz Sergio D. Bergese Joy E. Schabel The utility of generative artificial intelligence Chatbot (ChatGPT) in generating teaching and learning material for anesthesiology residents Frontiers in Artificial Intelligence artificial intelligence graduate medical education large language model anesthesiology residency generative AI |
| title | The utility of generative artificial intelligence Chatbot (ChatGPT) in generating teaching and learning material for anesthesiology residents |
| title_full | The utility of generative artificial intelligence Chatbot (ChatGPT) in generating teaching and learning material for anesthesiology residents |
| title_fullStr | The utility of generative artificial intelligence Chatbot (ChatGPT) in generating teaching and learning material for anesthesiology residents |
| title_full_unstemmed | The utility of generative artificial intelligence Chatbot (ChatGPT) in generating teaching and learning material for anesthesiology residents |
| title_short | The utility of generative artificial intelligence Chatbot (ChatGPT) in generating teaching and learning material for anesthesiology residents |
| title_sort | utility of generative artificial intelligence chatbot chatgpt in generating teaching and learning material for anesthesiology residents |
| topic | artificial intelligence graduate medical education large language model anesthesiology residency generative AI |
| url | https://www.frontiersin.org/articles/10.3389/frai.2025.1582096/full |
| work_keys_str_mv | AT zhaoshengjin theutilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT ramonabola theutilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT vincentbargnes theutilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT alexandratsivitis theutilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT sadiqrahman theutilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT jonathonschwartz theutilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT sergiodbergese theutilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT joyeschabel theutilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT zhaoshengjin utilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT ramonabola utilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT vincentbargnes utilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT alexandratsivitis utilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT sadiqrahman utilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT jonathonschwartz utilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT sergiodbergese utilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents AT joyeschabel utilityofgenerativeartificialintelligencechatbotchatgptingeneratingteachingandlearningmaterialforanesthesiologyresidents |