Research on the Lightweight of Natural Language Understanding Model in Electric Power Field
To address the application requirements of indicator question-and-answer (Q&A) in the electricity industry, this paper researches a natural language understanding model for balancing performance, computing resource consumption, and inference time, and proposes a method to incorporate an early st...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Harbin University of Science and Technology Publications
2024-12-01
|
| Series: | Journal of Harbin University of Science and Technology |
| Subjects: | |
| Online Access: | https://hlgxb.hrbust.edu.cn/#/digest?ArticleID=2391 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849416737378795520 |
|---|---|
| author | DONG Zengbo XU Shiyu CHEN Xi XU Bo XIN Rui ZHANG Pengfei SONG Hui |
| author_facet | DONG Zengbo XU Shiyu CHEN Xi XU Bo XIN Rui ZHANG Pengfei SONG Hui |
| author_sort | DONG Zengbo |
| collection | DOAJ |
| description | To address the application requirements of indicator question-and-answer (Q&A) in the electricity industry, this paper researches a natural language understanding model for balancing performance, computing resource consumption, and inference time, and proposes a method to incorporate an early stopping mechanism in the lightweight process of knowledge distillation. When training the student model with the knowledge of the teacher model, patient early stopping mechanism is added to each layer of encoding, allowing samples of different complexity to be encoded with different layers, reducing overfitting problems that are prone to occur in the student model, and controlling the inference time of the model. On the dataset constructed by the real data in the power industry, experiments show that when the parameters of the student model are reduced to 60% of the teacher model, and the inference time is reduced by nearly 50% , while the accuracy of the model is only reduced by about 2% , maintaining high availability. Compared with the benchmark models, it achieves a better balance in performance and inference time. |
| format | Article |
| id | doaj-art-05ef79a002a841df9e46e720e39512d6 |
| institution | Kabale University |
| issn | 1007-2683 |
| language | zho |
| publishDate | 2024-12-01 |
| publisher | Harbin University of Science and Technology Publications |
| record_format | Article |
| series | Journal of Harbin University of Science and Technology |
| spelling | doaj-art-05ef79a002a841df9e46e720e39512d62025-08-20T03:33:07ZzhoHarbin University of Science and Technology PublicationsJournal of Harbin University of Science and Technology1007-26832024-12-01290614314910.15938/j.jhust.2024.06.014Research on the Lightweight of Natural Language Understanding Model in Electric Power FieldDONG Zengbo0XU Shiyu1CHEN Xi2XU Bo3XIN Rui4ZHANG Pengfei5SONG Hui6Telecommunication Branch State Grid Hebei Electric Power Co. , Ltd. , Shijiazhuang 050081 ,ChinaSchool of Computer Science and Technology, Donghua University, Shanghai 201620 ,ChinaTelecommunication Branch State Grid Hebei Electric Power Co. , Ltd. , Shijiazhuang 050081 ,ChinaSchool of Computer Science and Technology, Donghua University, Shanghai 201620 ,ChinaTelecommunication Branch State Grid Hebei Electric Power Co. , Ltd. , Shijiazhuang 050081 ,ChinaTelecommunication Branch State Grid Hebei Electric Power Co. , Ltd. , Shijiazhuang 050081 ,ChinaSchool of Computer Science and Technology, Donghua University, Shanghai 201620 ,ChinaTo address the application requirements of indicator question-and-answer (Q&A) in the electricity industry, this paper researches a natural language understanding model for balancing performance, computing resource consumption, and inference time, and proposes a method to incorporate an early stopping mechanism in the lightweight process of knowledge distillation. When training the student model with the knowledge of the teacher model, patient early stopping mechanism is added to each layer of encoding, allowing samples of different complexity to be encoded with different layers, reducing overfitting problems that are prone to occur in the student model, and controlling the inference time of the model. On the dataset constructed by the real data in the power industry, experiments show that when the parameters of the student model are reduced to 60% of the teacher model, and the inference time is reduced by nearly 50% , while the accuracy of the model is only reduced by about 2% , maintaining high availability. Compared with the benchmark models, it achieves a better balance in performance and inference time.https://hlgxb.hrbust.edu.cn/#/digest?ArticleID=2391electricity domainnatural language understanding modellightweight modelknowledge distillationearly stopping mechanism |
| spellingShingle | DONG Zengbo XU Shiyu CHEN Xi XU Bo XIN Rui ZHANG Pengfei SONG Hui Research on the Lightweight of Natural Language Understanding Model in Electric Power Field Journal of Harbin University of Science and Technology electricity domain natural language understanding model lightweight model knowledge distillation early stopping mechanism |
| title | Research on the Lightweight of Natural Language Understanding Model in Electric Power Field |
| title_full | Research on the Lightweight of Natural Language Understanding Model in Electric Power Field |
| title_fullStr | Research on the Lightweight of Natural Language Understanding Model in Electric Power Field |
| title_full_unstemmed | Research on the Lightweight of Natural Language Understanding Model in Electric Power Field |
| title_short | Research on the Lightweight of Natural Language Understanding Model in Electric Power Field |
| title_sort | research on the lightweight of natural language understanding model in electric power field |
| topic | electricity domain natural language understanding model lightweight model knowledge distillation early stopping mechanism |
| url | https://hlgxb.hrbust.edu.cn/#/digest?ArticleID=2391 |
| work_keys_str_mv | AT dongzengbo researchonthelightweightofnaturallanguageunderstandingmodelinelectricpowerfield AT xushiyu researchonthelightweightofnaturallanguageunderstandingmodelinelectricpowerfield AT chenxi researchonthelightweightofnaturallanguageunderstandingmodelinelectricpowerfield AT xubo researchonthelightweightofnaturallanguageunderstandingmodelinelectricpowerfield AT xinrui researchonthelightweightofnaturallanguageunderstandingmodelinelectricpowerfield AT zhangpengfei researchonthelightweightofnaturallanguageunderstandingmodelinelectricpowerfield AT songhui researchonthelightweightofnaturallanguageunderstandingmodelinelectricpowerfield |