The need for guardrails with large language models in pharmacovigilance and other medical safety critical settings
Abstract Large language models (LLMs) are useful tools with the capacity for performing specific types of knowledge work at an effective scale. However, LLM deployments in high-risk and safety-critical domains pose unique challenges, notably the issue of “hallucinations”, where LLMs can generate fab...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-07-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-025-09138-0 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849343057663623168 |
|---|---|
| author | Joe B. Hakim Jeffery L. Painter Darmendra Ramcharran Vijay Kara Greg Powell Paulina Sobczak Chiho Sato Andrew Bate Andrew Beam |
| author_facet | Joe B. Hakim Jeffery L. Painter Darmendra Ramcharran Vijay Kara Greg Powell Paulina Sobczak Chiho Sato Andrew Bate Andrew Beam |
| author_sort | Joe B. Hakim |
| collection | DOAJ |
| description | Abstract Large language models (LLMs) are useful tools with the capacity for performing specific types of knowledge work at an effective scale. However, LLM deployments in high-risk and safety-critical domains pose unique challenges, notably the issue of “hallucinations”, where LLMs can generate fabricated information. This is particularly concerning in settings such as drug safety, where inaccuracies could lead to patient harm. To mitigate these risks, we have developed and demonstrated a proof of concept suite of guardrails specifically designed to mitigate certain types of hallucinations and errors for drug safety, with potential applicability to other medical safety-critical contexts. These guardrails include mechanisms to detect anomalous documents to prevent the ingestion of inappropriate data, identify incorrect drug names or adverse event terms, and convey uncertainty in generated content. We integrated these guardrails with an LLM fine-tuned for a text-to-text task, which involves converting both structured and unstructured data within adverse event reports into natural language. This method was applied to translate individual case safety reports, demonstrating effective application in a pharmacovigilance processing task. Our guardrail framework offers a set of tools with broad applicability across various domains, ensuring LLMs can be safely used in high-risk situations by eliminating the occurrence of key errors, including the generation of incorrect pharmacovigilance-related terms, thus adhering to stringent regulatory and quality standards in medical safety-critical environments. |
| format | Article |
| id | doaj-art-1dc80227d455455e99e75d0d73be755a |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-1dc80227d455455e99e75d0d73be755a2025-08-20T03:43:11ZengNature PortfolioScientific Reports2045-23222025-07-0115111410.1038/s41598-025-09138-0The need for guardrails with large language models in pharmacovigilance and other medical safety critical settingsJoe B. Hakim0Jeffery L. Painter1Darmendra Ramcharran2Vijay Kara3Greg Powell4Paulina Sobczak5Chiho Sato6Andrew Bate7Andrew Beam8Harvard-MIT Department of Health Sciences and TechnologyGSKGSKGSKGSKGSKGSKGSKDepartment of Epidemiology, Harvard T.H. Chan School of Public HealthAbstract Large language models (LLMs) are useful tools with the capacity for performing specific types of knowledge work at an effective scale. However, LLM deployments in high-risk and safety-critical domains pose unique challenges, notably the issue of “hallucinations”, where LLMs can generate fabricated information. This is particularly concerning in settings such as drug safety, where inaccuracies could lead to patient harm. To mitigate these risks, we have developed and demonstrated a proof of concept suite of guardrails specifically designed to mitigate certain types of hallucinations and errors for drug safety, with potential applicability to other medical safety-critical contexts. These guardrails include mechanisms to detect anomalous documents to prevent the ingestion of inappropriate data, identify incorrect drug names or adverse event terms, and convey uncertainty in generated content. We integrated these guardrails with an LLM fine-tuned for a text-to-text task, which involves converting both structured and unstructured data within adverse event reports into natural language. This method was applied to translate individual case safety reports, demonstrating effective application in a pharmacovigilance processing task. Our guardrail framework offers a set of tools with broad applicability across various domains, ensuring LLMs can be safely used in high-risk situations by eliminating the occurrence of key errors, including the generation of incorrect pharmacovigilance-related terms, thus adhering to stringent regulatory and quality standards in medical safety-critical environments.https://doi.org/10.1038/s41598-025-09138-0 |
| spellingShingle | Joe B. Hakim Jeffery L. Painter Darmendra Ramcharran Vijay Kara Greg Powell Paulina Sobczak Chiho Sato Andrew Bate Andrew Beam The need for guardrails with large language models in pharmacovigilance and other medical safety critical settings Scientific Reports |
| title | The need for guardrails with large language models in pharmacovigilance and other medical safety critical settings |
| title_full | The need for guardrails with large language models in pharmacovigilance and other medical safety critical settings |
| title_fullStr | The need for guardrails with large language models in pharmacovigilance and other medical safety critical settings |
| title_full_unstemmed | The need for guardrails with large language models in pharmacovigilance and other medical safety critical settings |
| title_short | The need for guardrails with large language models in pharmacovigilance and other medical safety critical settings |
| title_sort | need for guardrails with large language models in pharmacovigilance and other medical safety critical settings |
| url | https://doi.org/10.1038/s41598-025-09138-0 |
| work_keys_str_mv | AT joebhakim theneedforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT jefferylpainter theneedforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT darmendraramcharran theneedforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT vijaykara theneedforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT gregpowell theneedforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT paulinasobczak theneedforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT chihosato theneedforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT andrewbate theneedforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT andrewbeam theneedforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT joebhakim needforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT jefferylpainter needforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT darmendraramcharran needforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT vijaykara needforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT gregpowell needforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT paulinasobczak needforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT chihosato needforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT andrewbate needforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings AT andrewbeam needforguardrailswithlargelanguagemodelsinpharmacovigilanceandothermedicalsafetycriticalsettings |