A unified DNN weight compression framework using reweighted optimization methods
To address the large model sizes and intensive computation requirements of deep neural networks (DNNs), weight pruning techniques have been proposed and generally fall into two categories: static regularization-based pruning and dynamic regularization-based pruning. However, the static method often...
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-09-01
|
| Series: | Intelligent Systems with Applications |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2667305325000821 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849318336022708224 |
|---|---|
| author | Mengchen Fan Tianyun Zhang Xiaolong Ma Jiacheng Guo Zheng Zhan Shanglin Zhou Minghai Qin Caiwen Ding Baocheng Geng Makan Fardad Yanzhi Wang |
| author_facet | Mengchen Fan Tianyun Zhang Xiaolong Ma Jiacheng Guo Zheng Zhan Shanglin Zhou Minghai Qin Caiwen Ding Baocheng Geng Makan Fardad Yanzhi Wang |
| author_sort | Mengchen Fan |
| collection | DOAJ |
| description | To address the large model sizes and intensive computation requirements of deep neural networks (DNNs), weight pruning techniques have been proposed and generally fall into two categories: static regularization-based pruning and dynamic regularization-based pruning. However, the static method often leads to either complex operations or reduced accuracy, while the dynamic method requires extensive time to adjust parameters to maintain accuracy while achieving effective pruning. In this paper, we propose a unified robustness-aware framework for DNN weight pruning that dynamically updates regularization terms bounded by the designated constraint. This framework can generate both non-structured sparsity and different kinds of structured sparsity, and it incorporates adversarial training to enhance the robustness of the sparse model. We further extend our approach into an integrated framework capable of handling multiple DNN compression tasks. Experimental results show that our proposed method increases the compression rate – up to 630× for LeNet-5, 45× for AlexNet, 7.2× for MobileNet, 3.2× for ResNet-50 – while also reducing training time and simplifying hyperparameter tuning to a single penalty parameter. Additionally, our method improves model robustness by 5.07% for ResNet-18 and 3.34% for VGG-16 under a 16×pruning rate, outperforming the state-of-the-art ADMM-based hard constraint method. |
| format | Article |
| id | doaj-art-5018871cd3fd472fa2eedf1f87e22e70 |
| institution | Kabale University |
| issn | 2667-3053 |
| language | English |
| publishDate | 2025-09-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Intelligent Systems with Applications |
| spelling | doaj-art-5018871cd3fd472fa2eedf1f87e22e702025-08-20T03:50:53ZengElsevierIntelligent Systems with Applications2667-30532025-09-012720055610.1016/j.iswa.2025.200556A unified DNN weight compression framework using reweighted optimization methodsMengchen Fan0Tianyun Zhang1Xiaolong Ma2Jiacheng Guo3Zheng Zhan4Shanglin Zhou5Minghai Qin6Caiwen Ding7Baocheng Geng8Makan Fardad9Yanzhi Wang10The University of Alabama at Birmingham, USACleveland State University, USA; Corresponding author.Clemson University, USACleveland State University, USANortheastern University, USAUniversity of Connecticut, USAWestern Digital Research, USAUniversity of Connecticut, USAThe University of Alabama at Birmingham, USASyracuse University, USANortheastern University, USATo address the large model sizes and intensive computation requirements of deep neural networks (DNNs), weight pruning techniques have been proposed and generally fall into two categories: static regularization-based pruning and dynamic regularization-based pruning. However, the static method often leads to either complex operations or reduced accuracy, while the dynamic method requires extensive time to adjust parameters to maintain accuracy while achieving effective pruning. In this paper, we propose a unified robustness-aware framework for DNN weight pruning that dynamically updates regularization terms bounded by the designated constraint. This framework can generate both non-structured sparsity and different kinds of structured sparsity, and it incorporates adversarial training to enhance the robustness of the sparse model. We further extend our approach into an integrated framework capable of handling multiple DNN compression tasks. Experimental results show that our proposed method increases the compression rate – up to 630× for LeNet-5, 45× for AlexNet, 7.2× for MobileNet, 3.2× for ResNet-50 – while also reducing training time and simplifying hyperparameter tuning to a single penalty parameter. Additionally, our method improves model robustness by 5.07% for ResNet-18 and 3.34% for VGG-16 under a 16×pruning rate, outperforming the state-of-the-art ADMM-based hard constraint method.http://www.sciencedirect.com/science/article/pii/S2667305325000821Deep neural networksWeight pruningModel compressionAdversarial training |
| spellingShingle | Mengchen Fan Tianyun Zhang Xiaolong Ma Jiacheng Guo Zheng Zhan Shanglin Zhou Minghai Qin Caiwen Ding Baocheng Geng Makan Fardad Yanzhi Wang A unified DNN weight compression framework using reweighted optimization methods Intelligent Systems with Applications Deep neural networks Weight pruning Model compression Adversarial training |
| title | A unified DNN weight compression framework using reweighted optimization methods |
| title_full | A unified DNN weight compression framework using reweighted optimization methods |
| title_fullStr | A unified DNN weight compression framework using reweighted optimization methods |
| title_full_unstemmed | A unified DNN weight compression framework using reweighted optimization methods |
| title_short | A unified DNN weight compression framework using reweighted optimization methods |
| title_sort | unified dnn weight compression framework using reweighted optimization methods |
| topic | Deep neural networks Weight pruning Model compression Adversarial training |
| url | http://www.sciencedirect.com/science/article/pii/S2667305325000821 |
| work_keys_str_mv | AT mengchenfan aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT tianyunzhang aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT xiaolongma aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT jiachengguo aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT zhengzhan aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT shanglinzhou aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT minghaiqin aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT caiwending aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT baochenggeng aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT makanfardad aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT yanzhiwang aunifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT mengchenfan unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT tianyunzhang unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT xiaolongma unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT jiachengguo unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT zhengzhan unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT shanglinzhou unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT minghaiqin unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT caiwending unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT baochenggeng unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT makanfardad unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods AT yanzhiwang unifieddnnweightcompressionframeworkusingreweightedoptimizationmethods |