Deep representation learning using layer-wise VICReg losses
Abstract This paper presents a layer-wise training procedure of neural networks by minimizing a Variance-Invariance-Covariance Regularization (VICReg) loss at each layer. The procedure is beneficial when annotated data are scarce but enough unlabeled data are present. Being able to update the parame...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-07-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-025-08504-2 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849332633102712832 |
|---|---|
| author | Joy Datta Rawhatur Rabbi Puja Saha Aniqua Nusrat Zereen M. Abdullah-Al-Wadud Jia Uddin |
| author_facet | Joy Datta Rawhatur Rabbi Puja Saha Aniqua Nusrat Zereen M. Abdullah-Al-Wadud Jia Uddin |
| author_sort | Joy Datta |
| collection | DOAJ |
| description | Abstract This paper presents a layer-wise training procedure of neural networks by minimizing a Variance-Invariance-Covariance Regularization (VICReg) loss at each layer. The procedure is beneficial when annotated data are scarce but enough unlabeled data are present. Being able to update the parameters locally at each layer also handles problems such as vanishing gradient and initialization sensitivity in backpropagation. The procedure utilizes two forward passes instead of one forward and one backward pass as done in backpropagation, where one forward pass works on original data and the other on an augmented version of the data. It is shown that this procedure can construct more compact but informative spaces progressively at each layer. The architecture of the model is selected to be pyramidal, enabling effective feature extraction. In addition, we optimize weights for variance, invariance, and covariance terms of the loss function so that the model can capture higher-level semantic information optimally. After training the model, we assess its learned representations by measuring clustering quality metrics and performance on classification tasks utilizing a few labeled data. To evaluate the proposed approach, we do several experiments with different datasets: MNIST, EMNIST, Fashion MNIST, and CIFAR-100. The experimental results show that the training procedure enhances the classification accuracy of Deep Neural Networks (DNNs) trained on MNIST, EMNIST, Fashion MNIST, and CIFAR-100 by approximately 7%, 16%, 1%, and 7% respectively compared to the baseline models of similar architectures. |
| format | Article |
| id | doaj-art-d1d07e5bb3c74057a0b0c807850df232 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-d1d07e5bb3c74057a0b0c807850df2322025-08-20T03:46:08ZengNature PortfolioScientific Reports2045-23222025-07-0115112110.1038/s41598-025-08504-2Deep representation learning using layer-wise VICReg lossesJoy Datta0Rawhatur Rabbi1Puja Saha2Aniqua Nusrat Zereen3M. Abdullah-Al-Wadud4Jia Uddin5Department of Computer Science and Engineering, School of Data and Sciences, Brac UniversityDepartment of Computer Science and Engineering, School of Data and Sciences, Brac UniversityDepartment of Computer Science and Engineering, Rajshahi University of Engineering and TechnologyFaculty of Information and Communication Technology, Mahidol UniversityDepartment of Software Engineering, College of Computer and Information Sciences, King Saud UniversityAI and Big Data Department, Endicott College, Woosong UniversityAbstract This paper presents a layer-wise training procedure of neural networks by minimizing a Variance-Invariance-Covariance Regularization (VICReg) loss at each layer. The procedure is beneficial when annotated data are scarce but enough unlabeled data are present. Being able to update the parameters locally at each layer also handles problems such as vanishing gradient and initialization sensitivity in backpropagation. The procedure utilizes two forward passes instead of one forward and one backward pass as done in backpropagation, where one forward pass works on original data and the other on an augmented version of the data. It is shown that this procedure can construct more compact but informative spaces progressively at each layer. The architecture of the model is selected to be pyramidal, enabling effective feature extraction. In addition, we optimize weights for variance, invariance, and covariance terms of the loss function so that the model can capture higher-level semantic information optimally. After training the model, we assess its learned representations by measuring clustering quality metrics and performance on classification tasks utilizing a few labeled data. To evaluate the proposed approach, we do several experiments with different datasets: MNIST, EMNIST, Fashion MNIST, and CIFAR-100. The experimental results show that the training procedure enhances the classification accuracy of Deep Neural Networks (DNNs) trained on MNIST, EMNIST, Fashion MNIST, and CIFAR-100 by approximately 7%, 16%, 1%, and 7% respectively compared to the baseline models of similar architectures.https://doi.org/10.1038/s41598-025-08504-2BackpropagationForward-forward algorithmLayer-wise trainingNeural networksVICReg |
| spellingShingle | Joy Datta Rawhatur Rabbi Puja Saha Aniqua Nusrat Zereen M. Abdullah-Al-Wadud Jia Uddin Deep representation learning using layer-wise VICReg losses Scientific Reports Backpropagation Forward-forward algorithm Layer-wise training Neural networks VICReg |
| title | Deep representation learning using layer-wise VICReg losses |
| title_full | Deep representation learning using layer-wise VICReg losses |
| title_fullStr | Deep representation learning using layer-wise VICReg losses |
| title_full_unstemmed | Deep representation learning using layer-wise VICReg losses |
| title_short | Deep representation learning using layer-wise VICReg losses |
| title_sort | deep representation learning using layer wise vicreg losses |
| topic | Backpropagation Forward-forward algorithm Layer-wise training Neural networks VICReg |
| url | https://doi.org/10.1038/s41598-025-08504-2 |
| work_keys_str_mv | AT joydatta deeprepresentationlearningusinglayerwisevicreglosses AT rawhaturrabbi deeprepresentationlearningusinglayerwisevicreglosses AT pujasaha deeprepresentationlearningusinglayerwisevicreglosses AT aniquanusratzereen deeprepresentationlearningusinglayerwisevicreglosses AT mabdullahalwadud deeprepresentationlearningusinglayerwisevicreglosses AT jiauddin deeprepresentationlearningusinglayerwisevicreglosses |