Medical Specialty Classification Using Large Language Models (LLMs)
This study evaluates the performance of Large Language Model (LLM)-based classifiers, including BERT, Bio-BERT, and Distil-BERT, in comparison to traditional Machine Learning algorithms to classify the medical transcription reports into various specialties. While LLMs are increasingly utilized in h...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
LibraryPress@UF
2025-05-01
|
| Series: | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| Online Access: | https://journals.flvc.org/FLAIRS/article/view/138953 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850272407107928064 |
|---|---|
| author | Surya Kathirvel Lenin Mookiah |
| author_facet | Surya Kathirvel Lenin Mookiah |
| author_sort | Surya Kathirvel |
| collection | DOAJ |
| description |
This study evaluates the performance of Large Language Model (LLM)-based classifiers, including BERT, Bio-BERT, and Distil-BERT, in comparison to traditional Machine Learning algorithms to classify the medical transcription reports into various specialties. While LLMs are increasingly utilized in healthcare applications, Naive Bayes demonstrated the highest performance, achieving an accuracy of 86.16% and an F1-score of 84.52%, outperforming all other models. Machine Learning approaches, such as Random Forest and Multi-Layer Perceptron, also exhibited strong performance, whereas BERT-based models achieved lower accuracy, around 63%. However, LLM-based models performed better in certain specialties, such as Surgery, while Machine Learning models demonstrated superior performance in areas such as Allergy/Immunology, Cardiovascular/Pulmonary, and General Medicine. Our results show that LLM-based models might offer ad-
vantages in classifying certain medical specialties (e.g., Surgery) due to the benefits of pre-trained LLM models on domain-specific knowledge bases and/or their better semantic understanding with fewer data points.
|
| format | Article |
| id | doaj-art-1bd81245d460407fb448005caa8eabec |
| institution | OA Journals |
| issn | 2334-0754 2334-0762 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | LibraryPress@UF |
| record_format | Article |
| series | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| spelling | doaj-art-1bd81245d460407fb448005caa8eabec2025-08-20T01:51:49ZengLibraryPress@UFProceedings of the International Florida Artificial Intelligence Research Society Conference2334-07542334-07622025-05-0138110.32473/flairs.38.1.138953Medical Specialty Classification Using Large Language Models (LLMs)Surya Kathirvel0Lenin MookiahData Scientist This study evaluates the performance of Large Language Model (LLM)-based classifiers, including BERT, Bio-BERT, and Distil-BERT, in comparison to traditional Machine Learning algorithms to classify the medical transcription reports into various specialties. While LLMs are increasingly utilized in healthcare applications, Naive Bayes demonstrated the highest performance, achieving an accuracy of 86.16% and an F1-score of 84.52%, outperforming all other models. Machine Learning approaches, such as Random Forest and Multi-Layer Perceptron, also exhibited strong performance, whereas BERT-based models achieved lower accuracy, around 63%. However, LLM-based models performed better in certain specialties, such as Surgery, while Machine Learning models demonstrated superior performance in areas such as Allergy/Immunology, Cardiovascular/Pulmonary, and General Medicine. Our results show that LLM-based models might offer ad- vantages in classifying certain medical specialties (e.g., Surgery) due to the benefits of pre-trained LLM models on domain-specific knowledge bases and/or their better semantic understanding with fewer data points. https://journals.flvc.org/FLAIRS/article/view/138953 |
| spellingShingle | Surya Kathirvel Lenin Mookiah Medical Specialty Classification Using Large Language Models (LLMs) Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| title | Medical Specialty Classification Using Large Language Models (LLMs) |
| title_full | Medical Specialty Classification Using Large Language Models (LLMs) |
| title_fullStr | Medical Specialty Classification Using Large Language Models (LLMs) |
| title_full_unstemmed | Medical Specialty Classification Using Large Language Models (LLMs) |
| title_short | Medical Specialty Classification Using Large Language Models (LLMs) |
| title_sort | medical specialty classification using large language models llms |
| url | https://journals.flvc.org/FLAIRS/article/view/138953 |
| work_keys_str_mv | AT suryakathirvel medicalspecialtyclassificationusinglargelanguagemodelsllms AT leninmookiah medicalspecialtyclassificationusinglargelanguagemodelsllms |