A refined lion optimizer for deep learning
Abstract Optimization algorithms play a fundamental role in training neural networks. The optimizer focuses on the updating weights of momentum and velocity on learning rates and losses, furthermore the complexity of the optimizer and the quantity of updated parameters are considered. The Lion optim...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-07-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-025-07112-4 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849238309570609152 |
|---|---|
| author | Jian Rong Chenhao Ma Qinghui Zhang Yong Cao Weili Kou |
| author_facet | Jian Rong Chenhao Ma Qinghui Zhang Yong Cao Weili Kou |
| author_sort | Jian Rong |
| collection | DOAJ |
| description | Abstract Optimization algorithms play a fundamental role in training neural networks. The optimizer focuses on the updating weights of momentum and velocity on learning rates and losses, furthermore the complexity of the optimizer and the quantity of updated parameters are considered. The Lion optimizer, proposed by Google, is an excellent optimizer known for its faster training speed and more efficient memory usage. However, due to the discreteness of the sign function, the optimizer’s parameter updates may fail to adapt dynamically with momentum in some models, leading to non-convergence issues. In this paper, a Refined Lion Optimizer(RLion) introduce a novel update rule for the optimizer by leveraging a non-linear continuous bounded function that maps the product of momentum and a scaling factor. This design enables the optimization parameters to adaptively adjust according to both the magnitude of momentum and the scaling factor. Theoretical analysis demonstrates that the RLion is able to smooth out the fluctuations, converge faster and more reliable. The FasterNet, EfficientNetV2 and the YOLOV8 on ImageNet1k dataset are trained without warm up for classification by leveraging the RLion optimizer. The YOLOV8, YOLOV11 on VOC2012 data and Object detection with Vision Transformers on Caltech 101 dataset are trained for Object detection. The DeepLabV3+ on instance-level human parsing dataset, TwinLiteNet on BDD100K, UNet on part of the CARLA self-driving dataset are trained for semantic segmentation. Compared to the AdamW and Lion optimizer, the loss and accuracy present the RLion can promote the classification validation accuracy about $$0 \sim +20\%$$ higher than AdamW on many models even the learning rate is as high as AdamW. For object detection and semantic segmentation, RLion achieves or approaches the performance of the adamw, but overcomes the Lion gradient’s explosive or disappear susceptibility. The RLion has better convergence performance and versatility. |
| format | Article |
| id | doaj-art-7df1cb6f8a50438ca9a4a4c0ccf79249 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-7df1cb6f8a50438ca9a4a4c0ccf792492025-08-20T04:01:41ZengNature PortfolioScientific Reports2045-23222025-07-0115111710.1038/s41598-025-07112-4A refined lion optimizer for deep learningJian Rong0Chenhao Ma1Qinghui Zhang2Yong Cao3Weili Kou4College of Big Data and Intelligence, Southwest Forestry UniversityCollege of Big Data and Intelligence, Southwest Forestry UniversityCollege of Big Data and Intelligence, Southwest Forestry UniversityCollege of Big Data and Intelligence, Southwest Forestry UniversityCollege of Big Data and Intelligence, Southwest Forestry UniversityAbstract Optimization algorithms play a fundamental role in training neural networks. The optimizer focuses on the updating weights of momentum and velocity on learning rates and losses, furthermore the complexity of the optimizer and the quantity of updated parameters are considered. The Lion optimizer, proposed by Google, is an excellent optimizer known for its faster training speed and more efficient memory usage. However, due to the discreteness of the sign function, the optimizer’s parameter updates may fail to adapt dynamically with momentum in some models, leading to non-convergence issues. In this paper, a Refined Lion Optimizer(RLion) introduce a novel update rule for the optimizer by leveraging a non-linear continuous bounded function that maps the product of momentum and a scaling factor. This design enables the optimization parameters to adaptively adjust according to both the magnitude of momentum and the scaling factor. Theoretical analysis demonstrates that the RLion is able to smooth out the fluctuations, converge faster and more reliable. The FasterNet, EfficientNetV2 and the YOLOV8 on ImageNet1k dataset are trained without warm up for classification by leveraging the RLion optimizer. The YOLOV8, YOLOV11 on VOC2012 data and Object detection with Vision Transformers on Caltech 101 dataset are trained for Object detection. The DeepLabV3+ on instance-level human parsing dataset, TwinLiteNet on BDD100K, UNet on part of the CARLA self-driving dataset are trained for semantic segmentation. Compared to the AdamW and Lion optimizer, the loss and accuracy present the RLion can promote the classification validation accuracy about $$0 \sim +20\%$$ higher than AdamW on many models even the learning rate is as high as AdamW. For object detection and semantic segmentation, RLion achieves or approaches the performance of the adamw, but overcomes the Lion gradient’s explosive or disappear susceptibility. The RLion has better convergence performance and versatility.https://doi.org/10.1038/s41598-025-07112-4Lion optimierRefined lion optimizerarctanscaleable factor $$\alpha$$update rule |
| spellingShingle | Jian Rong Chenhao Ma Qinghui Zhang Yong Cao Weili Kou A refined lion optimizer for deep learning Scientific Reports Lion optimier Refined lion optimizer arctan scaleable factor $$\alpha$$ update rule |
| title | A refined lion optimizer for deep learning |
| title_full | A refined lion optimizer for deep learning |
| title_fullStr | A refined lion optimizer for deep learning |
| title_full_unstemmed | A refined lion optimizer for deep learning |
| title_short | A refined lion optimizer for deep learning |
| title_sort | refined lion optimizer for deep learning |
| topic | Lion optimier Refined lion optimizer arctan scaleable factor $$\alpha$$ update rule |
| url | https://doi.org/10.1038/s41598-025-07112-4 |
| work_keys_str_mv | AT jianrong arefinedlionoptimizerfordeeplearning AT chenhaoma arefinedlionoptimizerfordeeplearning AT qinghuizhang arefinedlionoptimizerfordeeplearning AT yongcao arefinedlionoptimizerfordeeplearning AT weilikou arefinedlionoptimizerfordeeplearning AT jianrong refinedlionoptimizerfordeeplearning AT chenhaoma refinedlionoptimizerfordeeplearning AT qinghuizhang refinedlionoptimizerfordeeplearning AT yongcao refinedlionoptimizerfordeeplearning AT weilikou refinedlionoptimizerfordeeplearning |