Enhancing AI Chatbot Responses in Health Care: The SMART Prompt Structure in Head and Neck Surgery
Abstract Objective This study aims to evaluate the impact of prompt construction on the quality of artificial intelligence (AI) chatbot responses in the context of head and neck surgery. Study Design Observational and evaluative study. Setting An international collaboration involving 16 researchers...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2025-01-01
|
| Series: | OTO Open |
| Subjects: | |
| Online Access: | https://doi.org/10.1002/oto2.70075 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850066643936346112 |
|---|---|
| author | Luigi Angelo Vaira Jerome R. Lechien Vincenzo Abbate Guido Gabriele Andrea Frosolini Andrea De Vito Antonino Maniaci Miguel Mayo‐Yáñez Paolo Boscolo‐Rizzo Alberto Maria Saibene Fabio Maglitto Giovanni Salzano Gianluigi Califano Stefania Troise Carlos Miguel Chiesa‐Estomba Giacomo De Riu |
| author_facet | Luigi Angelo Vaira Jerome R. Lechien Vincenzo Abbate Guido Gabriele Andrea Frosolini Andrea De Vito Antonino Maniaci Miguel Mayo‐Yáñez Paolo Boscolo‐Rizzo Alberto Maria Saibene Fabio Maglitto Giovanni Salzano Gianluigi Califano Stefania Troise Carlos Miguel Chiesa‐Estomba Giacomo De Riu |
| author_sort | Luigi Angelo Vaira |
| collection | DOAJ |
| description | Abstract Objective This study aims to evaluate the impact of prompt construction on the quality of artificial intelligence (AI) chatbot responses in the context of head and neck surgery. Study Design Observational and evaluative study. Setting An international collaboration involving 16 researchers from 11 European centers specializing in head and neck surgery. Methods A total of 24 questions, divided into clinical scenarios, theoretical questions, and patient inquiries, were developed. These questions were entered into ChatGPT‐4o both with and without the use of a structured prompt format, known as SMART (Seeker, Mission, AI Role, Register, Targeted Question). The AI‐generated responses were evaluated by experienced head and neck surgeons using the Quality Analysis of Medical Artificial Intelligence instrument (QAMAI), which assesses accuracy, clarity, relevance, completeness, source quality, and usefulness. Results The responses generated using the SMART prompt scored significantly higher across all QAMAI dimensions compared to those without contextualized prompts. Median QAMAI scores for SMART prompts were 27.5 (interquartile range [IQR] 25‐29) versus 24 (IQR 21.8‐25) for unstructured prompts (P < .001). Clinical scenarios and patient inquiries showed the most significant improvements, while theoretical questions also benefited, but to a lesser extent. The AI's source quality improved notably with the SMART prompt, particularly in theoretical questions. Conclusion This study suggests that the structured SMART prompt format significantly enhances the quality of AI chatbot responses in head and neck surgery. This approach improves the accuracy, relevance, and completeness of AI‐generated information, underscoring the importance of well‐constructed prompts in clinical applications. Further research is warranted to explore the applicability of SMART prompts across different medical specialties and AI platforms. |
| format | Article |
| id | doaj-art-7ec48b8ad3a444c8b3768422e1e7908a |
| institution | DOAJ |
| issn | 2473-974X |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Wiley |
| record_format | Article |
| series | OTO Open |
| spelling | doaj-art-7ec48b8ad3a444c8b3768422e1e7908a2025-08-20T02:48:41ZengWileyOTO Open2473-974X2025-01-0191n/an/a10.1002/oto2.70075Enhancing AI Chatbot Responses in Health Care: The SMART Prompt Structure in Head and Neck SurgeryLuigi Angelo Vaira0Jerome R. Lechien1Vincenzo Abbate2Guido Gabriele3Andrea Frosolini4Andrea De Vito5Antonino Maniaci6Miguel Mayo‐Yáñez7Paolo Boscolo‐Rizzo8Alberto Maria Saibene9Fabio Maglitto10Giovanni Salzano11Gianluigi Califano12Stefania Troise13Carlos Miguel Chiesa‐Estomba14Giacomo De Riu15Maxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy University of Sassari Sassari ItalyDepartment of Surgery, Mons School of Medicine, UMONS Research Institute for Health Sciences and Technology University of Mons (UMons) Mons BelgiumHead and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science Federico II University of Naples Naples ItalyMaxillofacial Surgery Unit, Department of Medical Biotechnologies University of Siena Siena ItalyMaxillofacial Surgery Unit, Department of Medical Biotechnologies University of Siena Siena ItalyDepartment of Medicine, Surgery and Pharmacy University of Sassari Sassari ItalyDepartment of Medicine and Surgery University of Enna Kore Enna ItalyOtorhinolaryngology, Head and Neck Surgery Department Complexo Hospitalario Universitario A Coruña (CHUAC) A Coruña SpainSection of Otolaryngology, Department of Medical, Surgical and Health Sciences University of Trieste Trieste ItalyOtolaryngology Unit, Department of Health Sciences, Santi Paolo e Carlo Hospital University of Milan Milan ItalyMaxillo‐Facial Surgery Unit University of Bari “Aldo Moro” Bari ItalyHead and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science Federico II University of Naples Naples ItalyDepartment of Neurosciences, Reproductive and Odontostomatological Science Federico II University of Naples Naples ItalyHead and Neck Section, Department of Neurosciences, Reproductive and Odontostomatological Science Federico II University of Naples Naples ItalyDepartment of Otorhinolaryngology–Head and Neck Surgery Hospital Universitario Donostia San Sebastian SpainMaxillofacial Surgery Operative Unit, Department of Medicine, Surgery and Pharmacy University of Sassari Sassari ItalyAbstract Objective This study aims to evaluate the impact of prompt construction on the quality of artificial intelligence (AI) chatbot responses in the context of head and neck surgery. Study Design Observational and evaluative study. Setting An international collaboration involving 16 researchers from 11 European centers specializing in head and neck surgery. Methods A total of 24 questions, divided into clinical scenarios, theoretical questions, and patient inquiries, were developed. These questions were entered into ChatGPT‐4o both with and without the use of a structured prompt format, known as SMART (Seeker, Mission, AI Role, Register, Targeted Question). The AI‐generated responses were evaluated by experienced head and neck surgeons using the Quality Analysis of Medical Artificial Intelligence instrument (QAMAI), which assesses accuracy, clarity, relevance, completeness, source quality, and usefulness. Results The responses generated using the SMART prompt scored significantly higher across all QAMAI dimensions compared to those without contextualized prompts. Median QAMAI scores for SMART prompts were 27.5 (interquartile range [IQR] 25‐29) versus 24 (IQR 21.8‐25) for unstructured prompts (P < .001). Clinical scenarios and patient inquiries showed the most significant improvements, while theoretical questions also benefited, but to a lesser extent. The AI's source quality improved notably with the SMART prompt, particularly in theoretical questions. Conclusion This study suggests that the structured SMART prompt format significantly enhances the quality of AI chatbot responses in head and neck surgery. This approach improves the accuracy, relevance, and completeness of AI‐generated information, underscoring the importance of well‐constructed prompts in clinical applications. Further research is warranted to explore the applicability of SMART prompts across different medical specialties and AI platforms.https://doi.org/10.1002/oto2.70075AIartificial intelligenceChatGPTmaxillofacial surgeryotorhinolaryngologyprompt engineering |
| spellingShingle | Luigi Angelo Vaira Jerome R. Lechien Vincenzo Abbate Guido Gabriele Andrea Frosolini Andrea De Vito Antonino Maniaci Miguel Mayo‐Yáñez Paolo Boscolo‐Rizzo Alberto Maria Saibene Fabio Maglitto Giovanni Salzano Gianluigi Califano Stefania Troise Carlos Miguel Chiesa‐Estomba Giacomo De Riu Enhancing AI Chatbot Responses in Health Care: The SMART Prompt Structure in Head and Neck Surgery OTO Open AI artificial intelligence ChatGPT maxillofacial surgery otorhinolaryngology prompt engineering |
| title | Enhancing AI Chatbot Responses in Health Care: The SMART Prompt Structure in Head and Neck Surgery |
| title_full | Enhancing AI Chatbot Responses in Health Care: The SMART Prompt Structure in Head and Neck Surgery |
| title_fullStr | Enhancing AI Chatbot Responses in Health Care: The SMART Prompt Structure in Head and Neck Surgery |
| title_full_unstemmed | Enhancing AI Chatbot Responses in Health Care: The SMART Prompt Structure in Head and Neck Surgery |
| title_short | Enhancing AI Chatbot Responses in Health Care: The SMART Prompt Structure in Head and Neck Surgery |
| title_sort | enhancing ai chatbot responses in health care the smart prompt structure in head and neck surgery |
| topic | AI artificial intelligence ChatGPT maxillofacial surgery otorhinolaryngology prompt engineering |
| url | https://doi.org/10.1002/oto2.70075 |
| work_keys_str_mv | AT luigiangelovaira enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT jeromerlechien enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT vincenzoabbate enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT guidogabriele enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT andreafrosolini enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT andreadevito enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT antoninomaniaci enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT miguelmayoyanez enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT paoloboscolorizzo enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT albertomariasaibene enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT fabiomaglitto enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT giovannisalzano enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT gianluigicalifano enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT stefaniatroise enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT carlosmiguelchiesaestomba enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery AT giacomoderiu enhancingaichatbotresponsesinhealthcarethesmartpromptstructureinheadandnecksurgery |