A New Adapter Tuning of Large Language Model for Chinese Medical Named Entity Recognition
Named entity recognition (NER) is a crucial step in extracting medical information from Chinese text, and fine-tuning large language models (LLMs) for this task is an effective approach. However, full parameter fine-tuning can potentially damage the model’s original parameters, resulting in catastro...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2024-12-01
|
| Series: | Applied Artificial Intelligence |
| Online Access: | https://www.tandfonline.com/doi/10.1080/08839514.2024.2385268 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850255233999962112 |
|---|---|
| author | Lu Zhou Yiheng Chen Xinmin Li Yanan Li Ning Li Xiting Wang Rui Zhang |
| author_facet | Lu Zhou Yiheng Chen Xinmin Li Yanan Li Ning Li Xiting Wang Rui Zhang |
| author_sort | Lu Zhou |
| collection | DOAJ |
| description | Named entity recognition (NER) is a crucial step in extracting medical information from Chinese text, and fine-tuning large language models (LLMs) for this task is an effective approach. However, full parameter fine-tuning can potentially damage the model’s original parameters, resulting in catastrophic forgetting. To overcome this challenge, we introduce a novel adapter-based fine-tuning approach. Our adapter is integrated into the first and last transformers of the LLM, operating in parallel to the feed-forward network (FFN), following multi-head attention. It mirrors the FFN’s structure and uses the FFN’s weights for initializing. Additionally, to further enhance performance, we incorporate prefix embeddings into the first and last transformers. Our experiments on the Chinese medical NER benchmark demonstrate that our adapter, combined with prefix embeddings, achieves the highest F1-score of 65.90%, surpassing prompt templates (21.99%), in-context learning (18.65%), P-tuning (63.03%), and the benchmark for the Chinese medical NER task (62.40%). These results indicate that our adapter effectively fine-tunes the LLM for Chinese medical NER while preserving the original parameters. |
| format | Article |
| id | doaj-art-c993833845e24e36813dfca3bed7028e |
| institution | OA Journals |
| issn | 0883-9514 1087-6545 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | Applied Artificial Intelligence |
| spelling | doaj-art-c993833845e24e36813dfca3bed7028e2025-08-20T01:56:56ZengTaylor & Francis GroupApplied Artificial Intelligence0883-95141087-65452024-12-0138110.1080/08839514.2024.2385268A New Adapter Tuning of Large Language Model for Chinese Medical Named Entity RecognitionLu Zhou0Yiheng Chen1Xinmin Li2Yanan Li3Ning Li4Xiting Wang5Rui Zhang6Traditional Chinese Medicine (Zhong Jing) School, Henan University of Chinese Medicine, Zhengzhou, ChinaTraditional Chinese Medicine (Zhong Jing) School, Henan University of Chinese Medicine, Zhengzhou, ChinaTraditional Chinese Medicine (Zhong Jing) School, Henan University of Chinese Medicine, Zhengzhou, ChinaTraditional Chinese Medicine (Zhong Jing) School, Henan University of Chinese Medicine, Zhengzhou, ChinaTraditional Chinese Medicine (Zhong Jing) School, Henan University of Chinese Medicine, Zhengzhou, ChinaAcademy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, ChinaTraditional Chinese Medicine (Zhong Jing) School, Henan University of Chinese Medicine, Zhengzhou, ChinaNamed entity recognition (NER) is a crucial step in extracting medical information from Chinese text, and fine-tuning large language models (LLMs) for this task is an effective approach. However, full parameter fine-tuning can potentially damage the model’s original parameters, resulting in catastrophic forgetting. To overcome this challenge, we introduce a novel adapter-based fine-tuning approach. Our adapter is integrated into the first and last transformers of the LLM, operating in parallel to the feed-forward network (FFN), following multi-head attention. It mirrors the FFN’s structure and uses the FFN’s weights for initializing. Additionally, to further enhance performance, we incorporate prefix embeddings into the first and last transformers. Our experiments on the Chinese medical NER benchmark demonstrate that our adapter, combined with prefix embeddings, achieves the highest F1-score of 65.90%, surpassing prompt templates (21.99%), in-context learning (18.65%), P-tuning (63.03%), and the benchmark for the Chinese medical NER task (62.40%). These results indicate that our adapter effectively fine-tunes the LLM for Chinese medical NER while preserving the original parameters.https://www.tandfonline.com/doi/10.1080/08839514.2024.2385268 |
| spellingShingle | Lu Zhou Yiheng Chen Xinmin Li Yanan Li Ning Li Xiting Wang Rui Zhang A New Adapter Tuning of Large Language Model for Chinese Medical Named Entity Recognition Applied Artificial Intelligence |
| title | A New Adapter Tuning of Large Language Model for Chinese Medical Named Entity Recognition |
| title_full | A New Adapter Tuning of Large Language Model for Chinese Medical Named Entity Recognition |
| title_fullStr | A New Adapter Tuning of Large Language Model for Chinese Medical Named Entity Recognition |
| title_full_unstemmed | A New Adapter Tuning of Large Language Model for Chinese Medical Named Entity Recognition |
| title_short | A New Adapter Tuning of Large Language Model for Chinese Medical Named Entity Recognition |
| title_sort | new adapter tuning of large language model for chinese medical named entity recognition |
| url | https://www.tandfonline.com/doi/10.1080/08839514.2024.2385268 |
| work_keys_str_mv | AT luzhou anewadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT yihengchen anewadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT xinminli anewadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT yananli anewadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT ningli anewadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT xitingwang anewadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT ruizhang anewadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT luzhou newadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT yihengchen newadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT xinminli newadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT yananli newadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT ningli newadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT xitingwang newadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition AT ruizhang newadaptertuningoflargelanguagemodelforchinesemedicalnamedentityrecognition |