Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural Networks
Neural networks have become a cornerstone of computer vision applications, with tasks ranging from image classification to object detection. However, challenges such as hyperparameter optimization (HPO) and model compression remain critical for improving performance and deploying models on resource-...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-02-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/4/1852 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850081161546563584 |
|---|---|
| author | A. Naumov A. Melnikov M. Perelshtein Ar. Melnikov V. Abronin F. Oksanichenko |
| author_facet | A. Naumov A. Melnikov M. Perelshtein Ar. Melnikov V. Abronin F. Oksanichenko |
| author_sort | A. Naumov |
| collection | DOAJ |
| description | Neural networks have become a cornerstone of computer vision applications, with tasks ranging from image classification to object detection. However, challenges such as hyperparameter optimization (HPO) and model compression remain critical for improving performance and deploying models on resource-constrained devices. In this work, we address these challenges using Tensor Network-based methods. For HPO, we propose and evaluate the TetraOpt algorithm against various optimization algorithms. These evaluations were conducted on subsets of the NATS-Bench dataset, including CIFAR-10, CIFAR-100, and ImageNet subsets. TetraOpt consistently demonstrated superior performance, effectively exploring the global optimization space and identifying configurations with higher accuracies. For model compression, we introduce a novel iterative method that combines CP, SVD, and Tucker tensor decompositions. Applied to ResNet-18 and ResNet-152, we evaluated our method on the CIFAR-10 and Tiny ImageNet datasets. Our method achieved compression ratios of up to <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>14.5</mn><mo>×</mo></mrow></semantics></math></inline-formula> for ResNet-18 and <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>2.5</mn><mo>×</mo></mrow></semantics></math></inline-formula> for ResNet-152. Additionally, the inference time for processing an image on a CPU remained largely unaffected, demonstrating the practicality of the method. |
| format | Article |
| id | doaj-art-e4634d42db29463ab54a17c63d8a58cf |
| institution | DOAJ |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-e4634d42db29463ab54a17c63d8a58cf2025-08-20T02:44:47ZengMDPI AGApplied Sciences2076-34172025-02-01154185210.3390/app15041852Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural NetworksA. Naumov0A. Melnikov1M. Perelshtein2Ar. Melnikov3V. Abronin4F. Oksanichenko5Terra Quantum AG, Kornhausstrasse 25, 9000 St. Gallen, SwitzerlandTerra Quantum AG, Kornhausstrasse 25, 9000 St. Gallen, SwitzerlandTerra Quantum AG, Kornhausstrasse 25, 9000 St. Gallen, SwitzerlandTerra Quantum AG, Kornhausstrasse 25, 9000 St. Gallen, SwitzerlandTerra Quantum AG, Kornhausstrasse 25, 9000 St. Gallen, SwitzerlandTerra Quantum AG, Kornhausstrasse 25, 9000 St. Gallen, SwitzerlandNeural networks have become a cornerstone of computer vision applications, with tasks ranging from image classification to object detection. However, challenges such as hyperparameter optimization (HPO) and model compression remain critical for improving performance and deploying models on resource-constrained devices. In this work, we address these challenges using Tensor Network-based methods. For HPO, we propose and evaluate the TetraOpt algorithm against various optimization algorithms. These evaluations were conducted on subsets of the NATS-Bench dataset, including CIFAR-10, CIFAR-100, and ImageNet subsets. TetraOpt consistently demonstrated superior performance, effectively exploring the global optimization space and identifying configurations with higher accuracies. For model compression, we introduce a novel iterative method that combines CP, SVD, and Tucker tensor decompositions. Applied to ResNet-18 and ResNet-152, we evaluated our method on the CIFAR-10 and Tiny ImageNet datasets. Our method achieved compression ratios of up to <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>14.5</mn><mo>×</mo></mrow></semantics></math></inline-formula> for ResNet-18 and <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>2.5</mn><mo>×</mo></mrow></semantics></math></inline-formula> for ResNet-152. Additionally, the inference time for processing an image on a CPU remained largely unaffected, demonstrating the practicality of the method.https://www.mdpi.com/2076-3417/15/4/1852tensortrain optimisationhyperparameter optimisationimage classificationcomputer vision and pattern recognitionconvolutional neural networksmodel compression |
| spellingShingle | A. Naumov A. Melnikov M. Perelshtein Ar. Melnikov V. Abronin F. Oksanichenko Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural Networks Applied Sciences tensortrain optimisation hyperparameter optimisation image classification computer vision and pattern recognition convolutional neural networks model compression |
| title | Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural Networks |
| title_full | Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural Networks |
| title_fullStr | Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural Networks |
| title_full_unstemmed | Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural Networks |
| title_short | Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural Networks |
| title_sort | tensor network methods for hyperparameter optimization and compression of convolutional neural networks |
| topic | tensortrain optimisation hyperparameter optimisation image classification computer vision and pattern recognition convolutional neural networks model compression |
| url | https://www.mdpi.com/2076-3417/15/4/1852 |
| work_keys_str_mv | AT anaumov tensornetworkmethodsforhyperparameteroptimizationandcompressionofconvolutionalneuralnetworks AT amelnikov tensornetworkmethodsforhyperparameteroptimizationandcompressionofconvolutionalneuralnetworks AT mperelshtein tensornetworkmethodsforhyperparameteroptimizationandcompressionofconvolutionalneuralnetworks AT armelnikov tensornetworkmethodsforhyperparameteroptimizationandcompressionofconvolutionalneuralnetworks AT vabronin tensornetworkmethodsforhyperparameteroptimizationandcompressionofconvolutionalneuralnetworks AT foksanichenko tensornetworkmethodsforhyperparameteroptimizationandcompressionofconvolutionalneuralnetworks |