ChatGPT-4 Performance on German Continuing Medical Education—Friend or Foe (Trick or Treat)? Protocol for a Randomized Controlled Trial
BackgroundThe increasing development and spread of artificial and assistive intelligence is opening up new areas of application not only in applied medicine but also in related fields such as continuing medical education (CME), which is part of the mandatory training program...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-02-01
|
| Series: | JMIR Research Protocols |
| Online Access: | https://www.researchprotocols.org/2025/1/e63887 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1825208900786323456 |
|---|---|
| author | Christian Burisch Abhav Bellary Frank Breuckmann Jan Ehlers Serge C Thal Timur Sellmann Daniel Gödde |
| author_facet | Christian Burisch Abhav Bellary Frank Breuckmann Jan Ehlers Serge C Thal Timur Sellmann Daniel Gödde |
| author_sort | Christian Burisch |
| collection | DOAJ |
| description |
BackgroundThe increasing development and spread of artificial and assistive intelligence is opening up new areas of application not only in applied medicine but also in related fields such as continuing medical education (CME), which is part of the mandatory training program for medical doctors in Germany. This study aimed to determine whether medical laypersons can successfully conduct training courses specifically for physicians with the help of a large language model (LLM) such as ChatGPT-4. This study aims to qualitatively and quantitatively investigate the impact of using artificial intelligence (AI; specifically ChatGPT) on the acquisition of credit points in German postgraduate medical education.
ObjectiveUsing this approach, we wanted to test further possible applications of AI in the postgraduate medical education setting and obtain results for practical use. Depending on the results, the potential influence of LLMs such as ChatGPT-4 on CME will be discussed, for example, as part of a SWOT (strengths, weaknesses, opportunities, threats) analysis.
MethodsWe designed a randomized controlled trial, in which adult high school students attempt to solve CME tests across six medical specialties in three study arms in total with 18 CME training courses per study arm under different interventional conditions with varying amounts of permitted use of ChatGPT-4. Sample size calculation was performed including guess probability (20% correct answers, SD=40%; confidence level of 1–α=.95/α=.05; test power of 1–β=.95; P<.05). The study was registered at open scientific framework.
ResultsAs of October 2024, the acquisition of data and students to participate in the trial is ongoing. Upon analysis of our acquired data, we predict our findings to be ready for publication as soon as early 2025.
ConclusionsWe aim to prove that the advances in AI, especially LLMs such as ChatGPT-4 have considerable effects on medical laypersons’ ability to successfully pass CME tests. The implications that this holds on how the concept of continuous medical education requires reevaluation are yet to be contemplated.
Trial RegistrationOSF Registries 10.17605/OSF.IO/MZNUF; https://osf.io/mznuf
International Registered Report Identifier (IRRID)PRR1-10.2196/63887 |
| format | Article |
| id | doaj-art-e68b379a457946f98334caeb97d0a59a |
| institution | Kabale University |
| issn | 1929-0748 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Research Protocols |
| spelling | doaj-art-e68b379a457946f98334caeb97d0a59a2025-02-06T17:31:33ZengJMIR PublicationsJMIR Research Protocols1929-07482025-02-0114e6388710.2196/63887ChatGPT-4 Performance on German Continuing Medical Education—Friend or Foe (Trick or Treat)? Protocol for a Randomized Controlled TrialChristian Burischhttps://orcid.org/0009-0009-9710-7827Abhav Bellaryhttps://orcid.org/0009-0004-8349-3468Frank Breuckmannhttps://orcid.org/0000-0001-7245-8000Jan Ehlershttps://orcid.org/0000-0001-6306-4173Serge C Thalhttps://orcid.org/0000-0002-1222-8729Timur Sellmannhttps://orcid.org/0000-0002-1471-6806Daniel Göddehttps://orcid.org/0000-0002-8430-1411 BackgroundThe increasing development and spread of artificial and assistive intelligence is opening up new areas of application not only in applied medicine but also in related fields such as continuing medical education (CME), which is part of the mandatory training program for medical doctors in Germany. This study aimed to determine whether medical laypersons can successfully conduct training courses specifically for physicians with the help of a large language model (LLM) such as ChatGPT-4. This study aims to qualitatively and quantitatively investigate the impact of using artificial intelligence (AI; specifically ChatGPT) on the acquisition of credit points in German postgraduate medical education. ObjectiveUsing this approach, we wanted to test further possible applications of AI in the postgraduate medical education setting and obtain results for practical use. Depending on the results, the potential influence of LLMs such as ChatGPT-4 on CME will be discussed, for example, as part of a SWOT (strengths, weaknesses, opportunities, threats) analysis. MethodsWe designed a randomized controlled trial, in which adult high school students attempt to solve CME tests across six medical specialties in three study arms in total with 18 CME training courses per study arm under different interventional conditions with varying amounts of permitted use of ChatGPT-4. Sample size calculation was performed including guess probability (20% correct answers, SD=40%; confidence level of 1–α=.95/α=.05; test power of 1–β=.95; P<.05). The study was registered at open scientific framework. ResultsAs of October 2024, the acquisition of data and students to participate in the trial is ongoing. Upon analysis of our acquired data, we predict our findings to be ready for publication as soon as early 2025. ConclusionsWe aim to prove that the advances in AI, especially LLMs such as ChatGPT-4 have considerable effects on medical laypersons’ ability to successfully pass CME tests. The implications that this holds on how the concept of continuous medical education requires reevaluation are yet to be contemplated. Trial RegistrationOSF Registries 10.17605/OSF.IO/MZNUF; https://osf.io/mznuf International Registered Report Identifier (IRRID)PRR1-10.2196/63887https://www.researchprotocols.org/2025/1/e63887 |
| spellingShingle | Christian Burisch Abhav Bellary Frank Breuckmann Jan Ehlers Serge C Thal Timur Sellmann Daniel Gödde ChatGPT-4 Performance on German Continuing Medical Education—Friend or Foe (Trick or Treat)? Protocol for a Randomized Controlled Trial JMIR Research Protocols |
| title | ChatGPT-4 Performance on German Continuing Medical Education—Friend or Foe (Trick or Treat)? Protocol for a Randomized Controlled Trial |
| title_full | ChatGPT-4 Performance on German Continuing Medical Education—Friend or Foe (Trick or Treat)? Protocol for a Randomized Controlled Trial |
| title_fullStr | ChatGPT-4 Performance on German Continuing Medical Education—Friend or Foe (Trick or Treat)? Protocol for a Randomized Controlled Trial |
| title_full_unstemmed | ChatGPT-4 Performance on German Continuing Medical Education—Friend or Foe (Trick or Treat)? Protocol for a Randomized Controlled Trial |
| title_short | ChatGPT-4 Performance on German Continuing Medical Education—Friend or Foe (Trick or Treat)? Protocol for a Randomized Controlled Trial |
| title_sort | chatgpt 4 performance on german continuing medical education friend or foe trick or treat protocol for a randomized controlled trial |
| url | https://www.researchprotocols.org/2025/1/e63887 |
| work_keys_str_mv | AT christianburisch chatgpt4performanceongermancontinuingmedicaleducationfriendorfoetrickortreatprotocolforarandomizedcontrolledtrial AT abhavbellary chatgpt4performanceongermancontinuingmedicaleducationfriendorfoetrickortreatprotocolforarandomizedcontrolledtrial AT frankbreuckmann chatgpt4performanceongermancontinuingmedicaleducationfriendorfoetrickortreatprotocolforarandomizedcontrolledtrial AT janehlers chatgpt4performanceongermancontinuingmedicaleducationfriendorfoetrickortreatprotocolforarandomizedcontrolledtrial AT sergecthal chatgpt4performanceongermancontinuingmedicaleducationfriendorfoetrickortreatprotocolforarandomizedcontrolledtrial AT timursellmann chatgpt4performanceongermancontinuingmedicaleducationfriendorfoetrickortreatprotocolforarandomizedcontrolledtrial AT danielgodde chatgpt4performanceongermancontinuingmedicaleducationfriendorfoetrickortreatprotocolforarandomizedcontrolledtrial |