Uncertainty-based quantization method for stable training of binary neural networks
Binary neural networks (BNNs) have gained attention due to their computational efficiency. However, training BNNs has proven to be challenging. Existing algorithms either fail to produce stable and high-quality results or are overly complex for practical use. In this paper, we introduce a novel quan...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Samara National Research University
2024-08-01
|
| Series: | Компьютерная оптика |
| Subjects: | |
| Online Access: | https://www.computeroptics.ru/eng/KO/Annot/KO48-4/480412e.html |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1823863783027638272 |
|---|---|
| author | A.V. Trusov D.N. Putintsev E.E. Limonova |
| author_facet | A.V. Trusov D.N. Putintsev E.E. Limonova |
| author_sort | A.V. Trusov |
| collection | DOAJ |
| description | Binary neural networks (BNNs) have gained attention due to their computational efficiency. However, training BNNs has proven to be challenging. Existing algorithms either fail to produce stable and high-quality results or are overly complex for practical use. In this paper, we introduce a novel quantizer called UBQ (Uncertainty-based quantizer) for BNNs, which combines the advantages of existing methods, resulting in stable training and high-quality BNNs even with a low number of trainable parameters. We also propose a training method involving gradual network freezing and batch normalization replacement, facilitating a smooth transition from training mode to execution mode for BNNs.
To evaluate UBQ, we conducted experiments on the MNIST and CIFAR-10 datasets and compared our method to existing algorithms. The results demonstrate that UBQ outperforms previous methods for smaller networks and achieves comparable results for larger networks. |
| format | Article |
| id | doaj-art-a668750103804f948a38419c2b782848 |
| institution | Kabale University |
| issn | 0134-2452 2412-6179 |
| language | English |
| publishDate | 2024-08-01 |
| publisher | Samara National Research University |
| record_format | Article |
| series | Компьютерная оптика |
| spelling | doaj-art-a668750103804f948a38419c2b7828482025-02-09T09:55:37ZengSamara National Research UniversityКомпьютерная оптика0134-24522412-61792024-08-0148457358110.18287/2412-6179-CO-1427Uncertainty-based quantization method for stable training of binary neural networksA.V. Trusov0 D.N. Putintsev1E.E. Limonova2Moscow Institute of Physics and Technology (National Research University); Federal Research Center “Computer Science and Control” of Russian Academy of Sciences; LLC “Smart Engines Service”Federal Research Center “Computer Science and Control” of Russian Academy of Sciences; LLC “Smart Engines Service”Federal Research Center “Computer Science and Control” of Russian Academy of Sciences; LLC “Smart Engines Service”Binary neural networks (BNNs) have gained attention due to their computational efficiency. However, training BNNs has proven to be challenging. Existing algorithms either fail to produce stable and high-quality results or are overly complex for practical use. In this paper, we introduce a novel quantizer called UBQ (Uncertainty-based quantizer) for BNNs, which combines the advantages of existing methods, resulting in stable training and high-quality BNNs even with a low number of trainable parameters. We also propose a training method involving gradual network freezing and batch normalization replacement, facilitating a smooth transition from training mode to execution mode for BNNs. To evaluate UBQ, we conducted experiments on the MNIST and CIFAR-10 datasets and compared our method to existing algorithms. The results demonstrate that UBQ outperforms previous methods for smaller networks and achieves comparable results for larger networks.https://www.computeroptics.ru/eng/KO/Annot/KO48-4/480412e.htmlbinary networksneural networks trainingquantizationgradient estimationapproximation |
| spellingShingle | A.V. Trusov D.N. Putintsev E.E. Limonova Uncertainty-based quantization method for stable training of binary neural networks Компьютерная оптика binary networks neural networks training quantization gradient estimation approximation |
| title | Uncertainty-based quantization method for stable training of binary neural networks |
| title_full | Uncertainty-based quantization method for stable training of binary neural networks |
| title_fullStr | Uncertainty-based quantization method for stable training of binary neural networks |
| title_full_unstemmed | Uncertainty-based quantization method for stable training of binary neural networks |
| title_short | Uncertainty-based quantization method for stable training of binary neural networks |
| title_sort | uncertainty based quantization method for stable training of binary neural networks |
| topic | binary networks neural networks training quantization gradient estimation approximation |
| url | https://www.computeroptics.ru/eng/KO/Annot/KO48-4/480412e.html |
| work_keys_str_mv | AT avtrusov uncertaintybasedquantizationmethodforstabletrainingofbinaryneuralnetworks AT dnputintsev uncertaintybasedquantizationmethodforstabletrainingofbinaryneuralnetworks AT eelimonova uncertaintybasedquantizationmethodforstabletrainingofbinaryneuralnetworks |