Frozen Weights as Prior for Parameter-Efficient Fine-Tuning
In the fields of natural language processing and computer vision, the emergence of large pre-trained models has led to the adoption of fine-tuning them for downstream tasks as an important paradigm. However, the full fine-tuning approach often comes with a hefty cost, which is not feasible for many...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10840174/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1823859587840737280 |
|---|---|
| author | Xiaolong Ma Peishun Liu Haojie Gao Zikang Yan Ningning Ma Wenqiang Liu Xuefang Wang Ruichun Tang |
| author_facet | Xiaolong Ma Peishun Liu Haojie Gao Zikang Yan Ningning Ma Wenqiang Liu Xuefang Wang Ruichun Tang |
| author_sort | Xiaolong Ma |
| collection | DOAJ |
| description | In the fields of natural language processing and computer vision, the emergence of large pre-trained models has led to the adoption of fine-tuning them for downstream tasks as an important paradigm. However, the full fine-tuning approach often comes with a hefty cost, which is not feasible for many researchers. Therefore, in recent years, numerous fine-tuning methods have been proposed to efficiently learn incremental updates of pre-trained weights in a more parameter-efficient way (e.g., employing low-rank increments or introducing adapters to modify the network architecture). However, most of these methods involve adding a set of incrementally learned parameters from scratch. From the perspective of full fine-tuning, these approaches often fail to fully exploit the connection between incremental changes during fine-tuning and the frozen weights of the pre-trained model. In order to delve into how to more effectively harness the weights of pre-trained models during the fine-tuning process to fully acquire new knowledge, we propose a novel parameter-efficient approach, which reuses the Frozen Weights as a prior (FoWA). We adapt the incremental matrix to the unitary matrix obtained from the singular value decomposition of the frozen weights, and further fine-tune the model by incorporating prior information. Through the frozen weight prior, FoWA can automatically select the appropriate rank and decouple the number of trainable parameters from the rank. We have conducted extensive experiments on various tasks, including natural language processing, question answering, natural language generation, and visual classification, demonstrating the effectiveness of FoWA. |
| format | Article |
| id | doaj-art-01bfff9597894a9f8de58ccee8a31c88 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-01bfff9597894a9f8de58ccee8a31c882025-02-11T00:01:06ZengIEEEIEEE Access2169-35362025-01-0113244112442510.1109/ACCESS.2025.352980710840174Frozen Weights as Prior for Parameter-Efficient Fine-TuningXiaolong Ma0https://orcid.org/0009-0002-4050-0854Peishun Liu1https://orcid.org/0000-0002-7746-8061Haojie Gao2Zikang Yan3Ningning Ma4Wenqiang Liu5Xuefang Wang6Ruichun Tang7School of Computer Science and Engineer, Ocean University of China, Qingdao, ChinaSchool of Computer Science and Engineer, Ocean University of China, Qingdao, ChinaSchool of Computer Science and Engineer, Ocean University of China, Qingdao, ChinaSchool of Computer Science and Engineer, Ocean University of China, Qingdao, ChinaSchool of Computer Science and Engineer, Ocean University of China, Qingdao, ChinaSchool of Computer Science and Engineer, Ocean University of China, Qingdao, ChinaSchool of Mathematical Sciences, Ocean University of China, Qingdao, ChinaSchool of Computer Science and Engineer, Ocean University of China, Qingdao, ChinaIn the fields of natural language processing and computer vision, the emergence of large pre-trained models has led to the adoption of fine-tuning them for downstream tasks as an important paradigm. However, the full fine-tuning approach often comes with a hefty cost, which is not feasible for many researchers. Therefore, in recent years, numerous fine-tuning methods have been proposed to efficiently learn incremental updates of pre-trained weights in a more parameter-efficient way (e.g., employing low-rank increments or introducing adapters to modify the network architecture). However, most of these methods involve adding a set of incrementally learned parameters from scratch. From the perspective of full fine-tuning, these approaches often fail to fully exploit the connection between incremental changes during fine-tuning and the frozen weights of the pre-trained model. In order to delve into how to more effectively harness the weights of pre-trained models during the fine-tuning process to fully acquire new knowledge, we propose a novel parameter-efficient approach, which reuses the Frozen Weights as a prior (FoWA). We adapt the incremental matrix to the unitary matrix obtained from the singular value decomposition of the frozen weights, and further fine-tune the model by incorporating prior information. Through the frozen weight prior, FoWA can automatically select the appropriate rank and decouple the number of trainable parameters from the rank. We have conducted extensive experiments on various tasks, including natural language processing, question answering, natural language generation, and visual classification, demonstrating the effectiveness of FoWA.https://ieeexplore.ieee.org/document/10840174/Fine-tuningPEFTlarge pre-trained modelstransfer learningadaptation |
| spellingShingle | Xiaolong Ma Peishun Liu Haojie Gao Zikang Yan Ningning Ma Wenqiang Liu Xuefang Wang Ruichun Tang Frozen Weights as Prior for Parameter-Efficient Fine-Tuning IEEE Access Fine-tuning PEFT large pre-trained models transfer learning adaptation |
| title | Frozen Weights as Prior for Parameter-Efficient Fine-Tuning |
| title_full | Frozen Weights as Prior for Parameter-Efficient Fine-Tuning |
| title_fullStr | Frozen Weights as Prior for Parameter-Efficient Fine-Tuning |
| title_full_unstemmed | Frozen Weights as Prior for Parameter-Efficient Fine-Tuning |
| title_short | Frozen Weights as Prior for Parameter-Efficient Fine-Tuning |
| title_sort | frozen weights as prior for parameter efficient fine tuning |
| topic | Fine-tuning PEFT large pre-trained models transfer learning adaptation |
| url | https://ieeexplore.ieee.org/document/10840174/ |
| work_keys_str_mv | AT xiaolongma frozenweightsaspriorforparameterefficientfinetuning AT peishunliu frozenweightsaspriorforparameterefficientfinetuning AT haojiegao frozenweightsaspriorforparameterefficientfinetuning AT zikangyan frozenweightsaspriorforparameterefficientfinetuning AT ningningma frozenweightsaspriorforparameterefficientfinetuning AT wenqiangliu frozenweightsaspriorforparameterefficientfinetuning AT xuefangwang frozenweightsaspriorforparameterefficientfinetuning AT ruichuntang frozenweightsaspriorforparameterefficientfinetuning |