Unit-Centric Regularization for Efficient Deep Neural Networks
Deep neural networks excel by learning hierarchical representations, often requiring architectural enhancements like increased width, normalization layers, or skip connections, each adding complexity and computational cost. This paper proposes Jumpstart, a novel regularization technique that enables...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/11087585/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849729279496028160 |
|---|---|
| author | Carles R. Riera Molina Eloi Puertas Oriol Pujol |
| author_facet | Carles R. Riera Molina Eloi Puertas Oriol Pujol |
| author_sort | Carles R. Riera Molina |
| collection | DOAJ |
| description | Deep neural networks excel by learning hierarchical representations, often requiring architectural enhancements like increased width, normalization layers, or skip connections, each adding complexity and computational cost. This paper proposes Jumpstart, a novel regularization technique that enables the use of simpler architectures by promoting efficient utilization of both network units and data points. The method penalizes units that become inactive (dead) or operate strictly in the linear regime, as well as data points whose activations within a layer are uniformly zero or strictly positive. This strategy enables the training of plain ReLU networks without relying on overparameterization, specialized initialization, normalization layers, or architectural modifications like skip connections. As a result, it promotes more efficient use of units and data, maintaining performance while avoiding waste of computational resources during both training and inference. On the ImageNet benchmark, it matches the top-1 accuracy of a standard ResNet50 with Batch Normalization and skip connections. On UCI tabular datasets, it consistently outperforms batch normalization and often surpasses residual connections. The method is evaluated using four global metrics: Dead Units, Linear Units, Trainability, and Convergence. Jumpstart significantly reduces the presence of inactive and linear units (0.07 and 0.12, respectively), outperforming most baselines and achieves superior trainability (1.0) and convergence (-0.03). These results demonstrate that simpler, regularized networks can maintain competitive accuracy while significantly lowering architectural complexity and computational burden. Jumpstart offers a sustainable and effective alternative to conventional deep learning design strategies, facilitating efficient training without compromising performance. |
| format | Article |
| id | doaj-art-e5c9aab4e0fb45008d21815dabbe6298 |
| institution | DOAJ |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-e5c9aab4e0fb45008d21815dabbe62982025-08-20T03:09:16ZengIEEEIEEE Access2169-35362025-01-011312955112957210.1109/ACCESS.2025.359131311087585Unit-Centric Regularization for Efficient Deep Neural NetworksCarles R. Riera Molina0https://orcid.org/0000-0002-4011-3486Eloi Puertas1https://orcid.org/0000-0001-6292-6448Oriol Pujol2https://orcid.org/0000-0001-7573-009XDepartament de Matemà tiques i Informà tica, Universitat de Barcelona, Barcelona, SpainDepartament de Matemà tiques i Informà tica, Universitat de Barcelona, Barcelona, SpainDepartament de Matemà tiques i Informà tica, Universitat de Barcelona, Barcelona, SpainDeep neural networks excel by learning hierarchical representations, often requiring architectural enhancements like increased width, normalization layers, or skip connections, each adding complexity and computational cost. This paper proposes Jumpstart, a novel regularization technique that enables the use of simpler architectures by promoting efficient utilization of both network units and data points. The method penalizes units that become inactive (dead) or operate strictly in the linear regime, as well as data points whose activations within a layer are uniformly zero or strictly positive. This strategy enables the training of plain ReLU networks without relying on overparameterization, specialized initialization, normalization layers, or architectural modifications like skip connections. As a result, it promotes more efficient use of units and data, maintaining performance while avoiding waste of computational resources during both training and inference. On the ImageNet benchmark, it matches the top-1 accuracy of a standard ResNet50 with Batch Normalization and skip connections. On UCI tabular datasets, it consistently outperforms batch normalization and often surpasses residual connections. The method is evaluated using four global metrics: Dead Units, Linear Units, Trainability, and Convergence. Jumpstart significantly reduces the presence of inactive and linear units (0.07 and 0.12, respectively), outperforming most baselines and achieves superior trainability (1.0) and convergence (-0.03). These results demonstrate that simpler, regularized networks can maintain competitive accuracy while significantly lowering architectural complexity and computational burden. Jumpstart offers a sustainable and effective alternative to conventional deep learning design strategies, facilitating efficient training without compromising performance.https://ieeexplore.ieee.org/document/11087585/Optimization constraintsdying neuronsregularizationdeep learning |
| spellingShingle | Carles R. Riera Molina Eloi Puertas Oriol Pujol Unit-Centric Regularization for Efficient Deep Neural Networks IEEE Access Optimization constraints dying neurons regularization deep learning |
| title | Unit-Centric Regularization for Efficient Deep Neural Networks |
| title_full | Unit-Centric Regularization for Efficient Deep Neural Networks |
| title_fullStr | Unit-Centric Regularization for Efficient Deep Neural Networks |
| title_full_unstemmed | Unit-Centric Regularization for Efficient Deep Neural Networks |
| title_short | Unit-Centric Regularization for Efficient Deep Neural Networks |
| title_sort | unit centric regularization for efficient deep neural networks |
| topic | Optimization constraints dying neurons regularization deep learning |
| url | https://ieeexplore.ieee.org/document/11087585/ |
| work_keys_str_mv | AT carlesrrieramolina unitcentricregularizationforefficientdeepneuralnetworks AT eloipuertas unitcentricregularizationforefficientdeepneuralnetworks AT oriolpujol unitcentricregularizationforefficientdeepneuralnetworks |