Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: a proof of concept study
Abstract In this proof of concept study, we demonstrated how Large Language Models (LLMs) can automate the conversion of unstructured case reports into clinical ratings. By leveraging instructions from a standardized clinical rating scale and evaluating the LLM’s confidence in its outputs, we aimed...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-03-01
|
| Series: | Schizophrenia |
| Online Access: | https://doi.org/10.1038/s41537-025-00601-5 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850095007597330432 |
|---|---|
| author | Sebastian Volkmer Alina Glück Andreas Meyer-Lindenberg Emanuel Schwarz Dusan Hirjak |
| author_facet | Sebastian Volkmer Alina Glück Andreas Meyer-Lindenberg Emanuel Schwarz Dusan Hirjak |
| author_sort | Sebastian Volkmer |
| collection | DOAJ |
| description | Abstract In this proof of concept study, we demonstrated how Large Language Models (LLMs) can automate the conversion of unstructured case reports into clinical ratings. By leveraging instructions from a standardized clinical rating scale and evaluating the LLM’s confidence in its outputs, we aimed to refine prompting strategies and enhance reproducibility. Using this strategy and case reports of drug-induced Parkinsonism, we showed that LLM-extracted data closely align with clinical rater manual extraction, achieving an accuracy of 90%. |
| format | Article |
| id | doaj-art-81de6d847b9f40c1a5b0ad8f09d8d005 |
| institution | DOAJ |
| issn | 2754-6993 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Schizophrenia |
| spelling | doaj-art-81de6d847b9f40c1a5b0ad8f09d8d0052025-08-20T02:41:32ZengNature PortfolioSchizophrenia2754-69932025-03-011111410.1038/s41537-025-00601-5Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: a proof of concept studySebastian Volkmer0Alina Glück1Andreas Meyer-Lindenberg2Emanuel Schwarz3Dusan Hirjak4Department of Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg UniversityDepartment of Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg UniversityDepartment of Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg UniversityDepartment of Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg UniversityDepartment of Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg UniversityAbstract In this proof of concept study, we demonstrated how Large Language Models (LLMs) can automate the conversion of unstructured case reports into clinical ratings. By leveraging instructions from a standardized clinical rating scale and evaluating the LLM’s confidence in its outputs, we aimed to refine prompting strategies and enhance reproducibility. Using this strategy and case reports of drug-induced Parkinsonism, we showed that LLM-extracted data closely align with clinical rater manual extraction, achieving an accuracy of 90%.https://doi.org/10.1038/s41537-025-00601-5 |
| spellingShingle | Sebastian Volkmer Alina Glück Andreas Meyer-Lindenberg Emanuel Schwarz Dusan Hirjak Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: a proof of concept study Schizophrenia |
| title | Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: a proof of concept study |
| title_full | Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: a proof of concept study |
| title_fullStr | Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: a proof of concept study |
| title_full_unstemmed | Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: a proof of concept study |
| title_short | Validating large language models against manual information extraction from case reports of drug-induced parkinsonism in patients with schizophrenia spectrum and mood disorders: a proof of concept study |
| title_sort | validating large language models against manual information extraction from case reports of drug induced parkinsonism in patients with schizophrenia spectrum and mood disorders a proof of concept study |
| url | https://doi.org/10.1038/s41537-025-00601-5 |
| work_keys_str_mv | AT sebastianvolkmer validatinglargelanguagemodelsagainstmanualinformationextractionfromcasereportsofdruginducedparkinsonisminpatientswithschizophreniaspectrumandmooddisordersaproofofconceptstudy AT alinagluck validatinglargelanguagemodelsagainstmanualinformationextractionfromcasereportsofdruginducedparkinsonisminpatientswithschizophreniaspectrumandmooddisordersaproofofconceptstudy AT andreasmeyerlindenberg validatinglargelanguagemodelsagainstmanualinformationextractionfromcasereportsofdruginducedparkinsonisminpatientswithschizophreniaspectrumandmooddisordersaproofofconceptstudy AT emanuelschwarz validatinglargelanguagemodelsagainstmanualinformationextractionfromcasereportsofdruginducedparkinsonisminpatientswithschizophreniaspectrumandmooddisordersaproofofconceptstudy AT dusanhirjak validatinglargelanguagemodelsagainstmanualinformationextractionfromcasereportsofdruginducedparkinsonisminpatientswithschizophreniaspectrumandmooddisordersaproofofconceptstudy |