Assessing the Performance of ChatGPT in Medical Toxicology Through Simulated Case Scenarios
Objective: The insufficient number of medical toxicologists and poison information centers worldwide limits the accessibility of adequate medical recommendations for the management of poisoned patients. This study aimed to assess the effectiveness of Chat Generative Pretrained Transformers (GPTs) me...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Türkiye Acil Tıp Vakfı
2024-12-01
|
| Series: | Global Emergency and Critical Care |
| Subjects: | |
| Online Access: | https://globemcc.com/articles/assessing-the-performance-of-chatgpt-in-medical-toxicology-through-simulated-case-scenarios/doi/globecc.galenos.2024.06025 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850133327668838400 |
|---|---|
| author | İbrahim Altundağ Semih Korkut Ramazan Güven Aynur Şahin |
| author_facet | İbrahim Altundağ Semih Korkut Ramazan Güven Aynur Şahin |
| author_sort | İbrahim Altundağ |
| collection | DOAJ |
| description | Objective: The insufficient number of medical toxicologists and poison information centers worldwide limits the accessibility of adequate medical recommendations for the management of poisoned patients. This study aimed to assess the effectiveness of Chat Generative Pretrained Transformers (GPTs) medical recommendations in medical toxicology and evaluate its accuracy as a valuable resource when accessing medical toxicologists or poison information centers is limited.
Materials and Methods: A toxicologist created 10 different toxicology-simulated case scenarios based on the possible presentations of poisoned patients in an emergency department setting. The categories of general approach and stabilization, diagnostic activities, and medical treatments and follow-up were used to measure case assessment and ChatGPT’s medical recommendation capacity.
Results: ChatGPT-4o achieved an average success rate of 90.88% across the simulated case scenarios. ChatGPT-4o received a passing grade in 9 cases (90%) and received “improvable” in only 1 case (10%). ChatGPT-4o’s average success rate in all categories and across all cases increased from 90.88% to 97.22% with the secondary test.
Conclusion: Our study indicates that it is possible to improve the success rate of ChatGPT in providing medical toxicology recommendations. The ability to query current medical toxicology information through ChatGPT-4o demonstrates the potential of ChatGPT to serve as a next-generation poison information center. |
| format | Article |
| id | doaj-art-3d7455e190594c75adfbd2b9b6681a4a |
| institution | OA Journals |
| issn | 2822-4078 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Türkiye Acil Tıp Vakfı |
| record_format | Article |
| series | Global Emergency and Critical Care |
| spelling | doaj-art-3d7455e190594c75adfbd2b9b6681a4a2025-08-20T02:31:59ZengTürkiye Acil Tıp VakfıGlobal Emergency and Critical Care2822-40782024-12-013313213910.4274/globecc.galenos.2024.06025Assessing the Performance of ChatGPT in Medical Toxicology Through Simulated Case Scenariosİbrahim Altundağ0https://orcid.org/0000-0002-0880-7218Semih Korkut1https://orcid.org/0000-0002-5409-3586Ramazan Güven2https://orcid.org/0000-0003-4129-8985Aynur Şahin3https://orcid.org/0000-0002-2679-6276University of Health Sciences Türkiye, Haydarpaşa Numune Training and Research Hospital, Clinic of Emergency Medicine, İstanbul, TürkiyeUniversity of Health Sciences Türkiye, Başakşehir Çam and Sakura City Hospital, Clinic of Emergency Medicine, İstanbul, TürkiyeUniversity of Health Sciences Türkiye, Başakşehir Çam and Sakura City Hospital, Clinic of Emergency Medicine, İstanbul, TürkiyeUniversity of Health Sciences Türkiye, Başakşehir Çam and Sakura City Hospital, Clinic of Emergency Medicine, İstanbul, TürkiyeObjective: The insufficient number of medical toxicologists and poison information centers worldwide limits the accessibility of adequate medical recommendations for the management of poisoned patients. This study aimed to assess the effectiveness of Chat Generative Pretrained Transformers (GPTs) medical recommendations in medical toxicology and evaluate its accuracy as a valuable resource when accessing medical toxicologists or poison information centers is limited. Materials and Methods: A toxicologist created 10 different toxicology-simulated case scenarios based on the possible presentations of poisoned patients in an emergency department setting. The categories of general approach and stabilization, diagnostic activities, and medical treatments and follow-up were used to measure case assessment and ChatGPT’s medical recommendation capacity. Results: ChatGPT-4o achieved an average success rate of 90.88% across the simulated case scenarios. ChatGPT-4o received a passing grade in 9 cases (90%) and received “improvable” in only 1 case (10%). ChatGPT-4o’s average success rate in all categories and across all cases increased from 90.88% to 97.22% with the secondary test. Conclusion: Our study indicates that it is possible to improve the success rate of ChatGPT in providing medical toxicology recommendations. The ability to query current medical toxicology information through ChatGPT-4o demonstrates the potential of ChatGPT to serve as a next-generation poison information center.https://globemcc.com/articles/assessing-the-performance-of-chatgpt-in-medical-toxicology-through-simulated-case-scenarios/doi/globecc.galenos.2024.06025artificial intelligence (ai)chatgpt-4oclinical decision support systemsgenerative pretrained transformerpoison control centertoxicology |
| spellingShingle | İbrahim Altundağ Semih Korkut Ramazan Güven Aynur Şahin Assessing the Performance of ChatGPT in Medical Toxicology Through Simulated Case Scenarios Global Emergency and Critical Care artificial intelligence (ai) chatgpt-4o clinical decision support systems generative pretrained transformer poison control center toxicology |
| title | Assessing the Performance of ChatGPT in Medical Toxicology Through Simulated Case Scenarios |
| title_full | Assessing the Performance of ChatGPT in Medical Toxicology Through Simulated Case Scenarios |
| title_fullStr | Assessing the Performance of ChatGPT in Medical Toxicology Through Simulated Case Scenarios |
| title_full_unstemmed | Assessing the Performance of ChatGPT in Medical Toxicology Through Simulated Case Scenarios |
| title_short | Assessing the Performance of ChatGPT in Medical Toxicology Through Simulated Case Scenarios |
| title_sort | assessing the performance of chatgpt in medical toxicology through simulated case scenarios |
| topic | artificial intelligence (ai) chatgpt-4o clinical decision support systems generative pretrained transformer poison control center toxicology |
| url | https://globemcc.com/articles/assessing-the-performance-of-chatgpt-in-medical-toxicology-through-simulated-case-scenarios/doi/globecc.galenos.2024.06025 |
| work_keys_str_mv | AT ibrahimaltundag assessingtheperformanceofchatgptinmedicaltoxicologythroughsimulatedcasescenarios AT semihkorkut assessingtheperformanceofchatgptinmedicaltoxicologythroughsimulatedcasescenarios AT ramazanguven assessingtheperformanceofchatgptinmedicaltoxicologythroughsimulatedcasescenarios AT aynursahin assessingtheperformanceofchatgptinmedicaltoxicologythroughsimulatedcasescenarios |