LOGIC: LLM-originated guidance for internal cognitive improvement of small language models in stance detection
Stance detection is a critical task in natural language processing that determines an author’s viewpoint toward a specific target, playing a pivotal role in social science research and various applications. Traditional approaches incorporating Wikipedia-sourced data into small language models (SLMs)...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
PeerJ Inc.
2024-12-01
|
| Series: | PeerJ Computer Science |
| Subjects: | |
| Online Access: | https://peerj.com/articles/cs-2585.pdf |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850245680184950784 |
|---|---|
| author | Woojin Lee Jaewook Lee Harksoo Kim |
| author_facet | Woojin Lee Jaewook Lee Harksoo Kim |
| author_sort | Woojin Lee |
| collection | DOAJ |
| description | Stance detection is a critical task in natural language processing that determines an author’s viewpoint toward a specific target, playing a pivotal role in social science research and various applications. Traditional approaches incorporating Wikipedia-sourced data into small language models (SLMs) to compensate for limited target knowledge often suffer from inconsistencies in article quality and length due to the diverse pool of Wikipedia contributors. To address these limitations, we utilize large language models (LLMs) pretrained on expansive datasets to generate accurate and contextually relevant target knowledge. By providing concise, real-world insights tailored to the stance detection task, this approach surpasses the limitations of Wikipedia-based information. Despite their superior reasoning capabilities, LLMs are computationally intensive and challenging to deploy on smaller devices. To mitigate these drawbacks, we introduce a reasoning distillation methodology that transfers the reasoning capabilities of LLMs to more compact SLMs, enhancing their efficiency while maintaining robust performance. Our stance detection model, LOGIC (LLM-Originated Guidance for Internal Cognitive improvement of small language models in stance detection), is built on Bidirectional and Auto-Regressive Transformer (BART) and fine-tuned with auxiliary learning tasks, including reasoning distillation. By incorporating LLM-generated target knowledge into the inference process, LOGIC achieves state-of-the-art performance on the VAried Stance Topics (VAST) dataset, outperforming advanced models like GPT-3.5 Turbo and GPT-4 Turbo in stance detection tasks. |
| format | Article |
| id | doaj-art-c4c869dd3e8841dca394e8e49e3741d4 |
| institution | OA Journals |
| issn | 2376-5992 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | PeerJ Inc. |
| record_format | Article |
| series | PeerJ Computer Science |
| spelling | doaj-art-c4c869dd3e8841dca394e8e49e3741d42025-08-20T01:59:22ZengPeerJ Inc.PeerJ Computer Science2376-59922024-12-0110e258510.7717/peerj-cs.2585LOGIC: LLM-originated guidance for internal cognitive improvement of small language models in stance detectionWoojin Lee0Jaewook Lee1Harksoo Kim2Department of Artificial Intelligence, Konkuk University, Seoul, Republic of South KoreaDepartment of Artificial Intelligence, Konkuk University, Seoul, Republic of South KoreaDepartment of Computer Science and Engineering, Konkuk University, Seoul, Republic of South KoreaStance detection is a critical task in natural language processing that determines an author’s viewpoint toward a specific target, playing a pivotal role in social science research and various applications. Traditional approaches incorporating Wikipedia-sourced data into small language models (SLMs) to compensate for limited target knowledge often suffer from inconsistencies in article quality and length due to the diverse pool of Wikipedia contributors. To address these limitations, we utilize large language models (LLMs) pretrained on expansive datasets to generate accurate and contextually relevant target knowledge. By providing concise, real-world insights tailored to the stance detection task, this approach surpasses the limitations of Wikipedia-based information. Despite their superior reasoning capabilities, LLMs are computationally intensive and challenging to deploy on smaller devices. To mitigate these drawbacks, we introduce a reasoning distillation methodology that transfers the reasoning capabilities of LLMs to more compact SLMs, enhancing their efficiency while maintaining robust performance. Our stance detection model, LOGIC (LLM-Originated Guidance for Internal Cognitive improvement of small language models in stance detection), is built on Bidirectional and Auto-Regressive Transformer (BART) and fine-tuned with auxiliary learning tasks, including reasoning distillation. By incorporating LLM-generated target knowledge into the inference process, LOGIC achieves state-of-the-art performance on the VAried Stance Topics (VAST) dataset, outperforming advanced models like GPT-3.5 Turbo and GPT-4 Turbo in stance detection tasks.https://peerj.com/articles/cs-2585.pdfNatural language processingStance detectionReasoning distillation |
| spellingShingle | Woojin Lee Jaewook Lee Harksoo Kim LOGIC: LLM-originated guidance for internal cognitive improvement of small language models in stance detection PeerJ Computer Science Natural language processing Stance detection Reasoning distillation |
| title | LOGIC: LLM-originated guidance for internal cognitive improvement of small language models in stance detection |
| title_full | LOGIC: LLM-originated guidance for internal cognitive improvement of small language models in stance detection |
| title_fullStr | LOGIC: LLM-originated guidance for internal cognitive improvement of small language models in stance detection |
| title_full_unstemmed | LOGIC: LLM-originated guidance for internal cognitive improvement of small language models in stance detection |
| title_short | LOGIC: LLM-originated guidance for internal cognitive improvement of small language models in stance detection |
| title_sort | logic llm originated guidance for internal cognitive improvement of small language models in stance detection |
| topic | Natural language processing Stance detection Reasoning distillation |
| url | https://peerj.com/articles/cs-2585.pdf |
| work_keys_str_mv | AT woojinlee logicllmoriginatedguidanceforinternalcognitiveimprovementofsmalllanguagemodelsinstancedetection AT jaewooklee logicllmoriginatedguidanceforinternalcognitiveimprovementofsmalllanguagemodelsinstancedetection AT harksookim logicllmoriginatedguidanceforinternalcognitiveimprovementofsmalllanguagemodelsinstancedetection |