Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human Performance
Abstract BackgroundThe application of large language models (LLMs) in analyzing expert textual online data is a topic of growing importance in computational linguistics and qualitative research within health care settings. ObjectiveThe objective of this study was t...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-04-01
|
| Series: | JMIR AI |
| Online Access: | https://ai.jmir.org/2025/1/e64447 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849706628402642944 |
|---|---|
| author | Arturo Castellanos Haoqiang Jiang Paulo Gomes Debra Vander Meer Alfred Castillo |
| author_facet | Arturo Castellanos Haoqiang Jiang Paulo Gomes Debra Vander Meer Alfred Castillo |
| author_sort | Arturo Castellanos |
| collection | DOAJ |
| description |
Abstract
BackgroundThe application of large language models (LLMs) in analyzing expert textual online data is a topic of growing importance in computational linguistics and qualitative research within health care settings.
ObjectiveThe objective of this study was to understand how LLMs can help analyze expert textual data. Topic modeling enables scaling the thematic analysis of content of a large corpus of data, but it still requires interpretation. We investigate the use of LLMs to help researchers scale this interpretation.
MethodsThe primary methodological phases of this project were (1) collecting data representing posts to an online nurse forum, as well as cleaning and preprocessing the data; (2) using latent Dirichlet allocation (LDA) to derive topics; (3) using human categorization for topic modeling; and (4) using LLMs to complement and scale the interpretation of thematic analysis. The purpose is to compare the outcomes of human interpretation with those derived from LLMs.
ResultsThere is substantial agreement (247/310, 80%) between LLM and human interpretation. For two-thirds of the topics, human evaluation and LLMs agree on alignment and convergence of themes. Furthermore, LLM subthemes offer depth of analysis within LDA topics, providing detailed explanations that align with and build upon established human themes. Nonetheless, LLMs identify coherence and complementarity where human evaluation does not.
ConclusionsLLMs enable the automation of the interpretation task in qualitative research. There are challenges in the use of LLMs for evaluation of the resulting themes. |
| format | Article |
| id | doaj-art-d6d06d48896e42be8bda3c5a47e40788 |
| institution | DOAJ |
| issn | 2817-1705 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR AI |
| spelling | doaj-art-d6d06d48896e42be8bda3c5a47e407882025-08-20T03:16:08ZengJMIR PublicationsJMIR AI2817-17052025-04-014e64447e6444710.2196/64447Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human PerformanceArturo Castellanoshttp://orcid.org/0000-0002-7477-7379Haoqiang Jianghttp://orcid.org/0000-0002-6788-9604Paulo Gomeshttp://orcid.org/0000-0001-7694-5500Debra Vander Meerhttp://orcid.org/0000-0002-5930-6667Alfred Castillohttp://orcid.org/0000-0002-3498-2415 Abstract BackgroundThe application of large language models (LLMs) in analyzing expert textual online data is a topic of growing importance in computational linguistics and qualitative research within health care settings. ObjectiveThe objective of this study was to understand how LLMs can help analyze expert textual data. Topic modeling enables scaling the thematic analysis of content of a large corpus of data, but it still requires interpretation. We investigate the use of LLMs to help researchers scale this interpretation. MethodsThe primary methodological phases of this project were (1) collecting data representing posts to an online nurse forum, as well as cleaning and preprocessing the data; (2) using latent Dirichlet allocation (LDA) to derive topics; (3) using human categorization for topic modeling; and (4) using LLMs to complement and scale the interpretation of thematic analysis. The purpose is to compare the outcomes of human interpretation with those derived from LLMs. ResultsThere is substantial agreement (247/310, 80%) between LLM and human interpretation. For two-thirds of the topics, human evaluation and LLMs agree on alignment and convergence of themes. Furthermore, LLM subthemes offer depth of analysis within LDA topics, providing detailed explanations that align with and build upon established human themes. Nonetheless, LLMs identify coherence and complementarity where human evaluation does not. ConclusionsLLMs enable the automation of the interpretation task in qualitative research. There are challenges in the use of LLMs for evaluation of the resulting themes.https://ai.jmir.org/2025/1/e64447 |
| spellingShingle | Arturo Castellanos Haoqiang Jiang Paulo Gomes Debra Vander Meer Alfred Castillo Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human Performance JMIR AI |
| title | Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human Performance |
| title_full | Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human Performance |
| title_fullStr | Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human Performance |
| title_full_unstemmed | Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human Performance |
| title_short | Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human Performance |
| title_sort | large language models for thematic summarization in qualitative health care research comparative analysis of model and human performance |
| url | https://ai.jmir.org/2025/1/e64447 |
| work_keys_str_mv | AT arturocastellanos largelanguagemodelsforthematicsummarizationinqualitativehealthcareresearchcomparativeanalysisofmodelandhumanperformance AT haoqiangjiang largelanguagemodelsforthematicsummarizationinqualitativehealthcareresearchcomparativeanalysisofmodelandhumanperformance AT paulogomes largelanguagemodelsforthematicsummarizationinqualitativehealthcareresearchcomparativeanalysisofmodelandhumanperformance AT debravandermeer largelanguagemodelsforthematicsummarizationinqualitativehealthcareresearchcomparativeanalysisofmodelandhumanperformance AT alfredcastillo largelanguagemodelsforthematicsummarizationinqualitativehealthcareresearchcomparativeanalysisofmodelandhumanperformance |