Characterizing research domain criteria symptoms among psychiatric inpatients using large language models
We sought to characterize the ability of large language models to estimate NIMH Research Domain Criteria dimensions from narrative clinical notes of adult psychiatric inpatients, deriving estimate of overall burden of symptoms in each domain. We extracted consecutive admissions to a psychiatric inpa...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-12-01
|
| Series: | Journal of Mood and Anxiety Disorders |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2950004424000336 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846123214832402432 |
|---|---|
| author | Thomas H. McCoy Roy H. Perlis |
| author_facet | Thomas H. McCoy Roy H. Perlis |
| author_sort | Thomas H. McCoy |
| collection | DOAJ |
| description | We sought to characterize the ability of large language models to estimate NIMH Research Domain Criteria dimensions from narrative clinical notes of adult psychiatric inpatients, deriving estimate of overall burden of symptoms in each domain. We extracted consecutive admissions to a psychiatric inpatient unit between December 23, 2009 and September 27, 2015 from the electronic health records of a large academic medical center. Admission and discharge notes were scored with a HIPAA-compliant instance of a large language model (gpt-4–1106-preview). To examine convergent validity, the resulting estimates were correlated with those derived using an earlier method; for predictive validity, they were examined for association with length of hospitalization and probability of readmission. The cohort included 3619 individuals, 1779 female (49 %), 1840 male (51 %) with mean age 44 (SD=16.6). We identified modest correlations between LLM-derived RDoC scores and a previously validated scoring method, with Kendall’s tau between from.07 for arousal and 0.27 for positive and cognitive domains (p < .001 for all of these). For admission notes, greater scores on cognitive, sensorimotor, negative, and social domains were significantly associated with longer length of hospitalization in linear regression models including sociodemographic features (p < .01 for all of these); positive valence was associated with shorter hospitalization (p < .001). For discharge notes, social, arousal, and positive valence were associated with likelihood of readmission within 180 days in adjusted logistic regression models (p < .05 for social and arousal, p < .001 for positive valence). Overall, LLM-derived estimates of RDoC psychopathology demonstrated promising convergent and predictive validity, suggesting this approach may make real-world application of the RDoC framework more feasible. |
| format | Article |
| id | doaj-art-b7129e27c1504823a790fd4ac6a623a2 |
| institution | Kabale University |
| issn | 2950-0044 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Journal of Mood and Anxiety Disorders |
| spelling | doaj-art-b7129e27c1504823a790fd4ac6a623a22024-12-14T06:34:41ZengElsevierJournal of Mood and Anxiety Disorders2950-00442024-12-018100079Characterizing research domain criteria symptoms among psychiatric inpatients using large language modelsThomas H. McCoy0Roy H. Perlis1Center for Quantitative Health and Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States; Department of Psychiatry, Harvard Medical School, Boston, MA, United StatesCenter for Quantitative Health and Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States; Department of Psychiatry, Harvard Medical School, Boston, MA, United States; Correspondence to: Massachusetts General Hospital, 185 Cambridge Street, 6th Floor, Boston, MA 02114, United States.We sought to characterize the ability of large language models to estimate NIMH Research Domain Criteria dimensions from narrative clinical notes of adult psychiatric inpatients, deriving estimate of overall burden of symptoms in each domain. We extracted consecutive admissions to a psychiatric inpatient unit between December 23, 2009 and September 27, 2015 from the electronic health records of a large academic medical center. Admission and discharge notes were scored with a HIPAA-compliant instance of a large language model (gpt-4–1106-preview). To examine convergent validity, the resulting estimates were correlated with those derived using an earlier method; for predictive validity, they were examined for association with length of hospitalization and probability of readmission. The cohort included 3619 individuals, 1779 female (49 %), 1840 male (51 %) with mean age 44 (SD=16.6). We identified modest correlations between LLM-derived RDoC scores and a previously validated scoring method, with Kendall’s tau between from.07 for arousal and 0.27 for positive and cognitive domains (p < .001 for all of these). For admission notes, greater scores on cognitive, sensorimotor, negative, and social domains were significantly associated with longer length of hospitalization in linear regression models including sociodemographic features (p < .01 for all of these); positive valence was associated with shorter hospitalization (p < .001). For discharge notes, social, arousal, and positive valence were associated with likelihood of readmission within 180 days in adjusted logistic regression models (p < .05 for social and arousal, p < .001 for positive valence). Overall, LLM-derived estimates of RDoC psychopathology demonstrated promising convergent and predictive validity, suggesting this approach may make real-world application of the RDoC framework more feasible.http://www.sciencedirect.com/science/article/pii/S2950004424000336Research domain criteriaInpatientDepressionMachine learningArtificial intelligence |
| spellingShingle | Thomas H. McCoy Roy H. Perlis Characterizing research domain criteria symptoms among psychiatric inpatients using large language models Journal of Mood and Anxiety Disorders Research domain criteria Inpatient Depression Machine learning Artificial intelligence |
| title | Characterizing research domain criteria symptoms among psychiatric inpatients using large language models |
| title_full | Characterizing research domain criteria symptoms among psychiatric inpatients using large language models |
| title_fullStr | Characterizing research domain criteria symptoms among psychiatric inpatients using large language models |
| title_full_unstemmed | Characterizing research domain criteria symptoms among psychiatric inpatients using large language models |
| title_short | Characterizing research domain criteria symptoms among psychiatric inpatients using large language models |
| title_sort | characterizing research domain criteria symptoms among psychiatric inpatients using large language models |
| topic | Research domain criteria Inpatient Depression Machine learning Artificial intelligence |
| url | http://www.sciencedirect.com/science/article/pii/S2950004424000336 |
| work_keys_str_mv | AT thomashmccoy characterizingresearchdomaincriteriasymptomsamongpsychiatricinpatientsusinglargelanguagemodels AT royhperlis characterizingresearchdomaincriteriasymptomsamongpsychiatricinpatientsusinglargelanguagemodels |