A Novel Sine Step Size for Warm-Restart Stochastic Gradient Descent
This paper proposes a novel sine step size for warm-restart stochastic gradient descent (SGD). For the SGD based on the new proposed step size, we establish convergence rates for smooth non-convex functions with and without the Polyak–Łojasiewicz (PL) condition. To assess the effectiveness of the ne...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Axioms |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2075-1680/13/12/857 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850086476616826880 |
|---|---|
| author | Mahsa Soheil Shamaee Sajad Fathi Hafshejani |
| author_facet | Mahsa Soheil Shamaee Sajad Fathi Hafshejani |
| author_sort | Mahsa Soheil Shamaee |
| collection | DOAJ |
| description | This paper proposes a novel sine step size for warm-restart stochastic gradient descent (SGD). For the SGD based on the new proposed step size, we establish convergence rates for smooth non-convex functions with and without the Polyak–Łojasiewicz (PL) condition. To assess the effectiveness of the new step size, we implemented it across several datasets, including FashionMNIST, CIFAR10, and CIFAR100. This implementation was compared against eight distinct existing methods. The experimental results demonstrate that the proposed sine step size improves the test accuracy of the CIFAR100 dataset by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>1.14</mn><mo>%</mo></mrow></semantics></math></inline-formula>. This improvement highlights the efficiency of the new step size when compared to eight other popular step size methods. |
| format | Article |
| id | doaj-art-37e25f9b4aeb48639181b09ebbc0cb47 |
| institution | DOAJ |
| issn | 2075-1680 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Axioms |
| spelling | doaj-art-37e25f9b4aeb48639181b09ebbc0cb472025-08-20T02:43:28ZengMDPI AGAxioms2075-16802024-12-01131285710.3390/axioms13120857A Novel Sine Step Size for Warm-Restart Stochastic Gradient DescentMahsa Soheil Shamaee0Sajad Fathi Hafshejani1Department of Computer Science, Faculty of Mathematical Science, University of Kashan, Kashan 8731753153, IranDepartment of Math and Computer Science, University of Lethbridge, Lethbridge, AB T1K 3M4, CanadaThis paper proposes a novel sine step size for warm-restart stochastic gradient descent (SGD). For the SGD based on the new proposed step size, we establish convergence rates for smooth non-convex functions with and without the Polyak–Łojasiewicz (PL) condition. To assess the effectiveness of the new step size, we implemented it across several datasets, including FashionMNIST, CIFAR10, and CIFAR100. This implementation was compared against eight distinct existing methods. The experimental results demonstrate that the proposed sine step size improves the test accuracy of the CIFAR100 dataset by <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mn>1.14</mn><mo>%</mo></mrow></semantics></math></inline-formula>. This improvement highlights the efficiency of the new step size when compared to eight other popular step size methods.https://www.mdpi.com/2075-1680/13/12/857stochastic gradient descentdecay step sizeconvergence rate |
| spellingShingle | Mahsa Soheil Shamaee Sajad Fathi Hafshejani A Novel Sine Step Size for Warm-Restart Stochastic Gradient Descent Axioms stochastic gradient descent decay step size convergence rate |
| title | A Novel Sine Step Size for Warm-Restart Stochastic Gradient Descent |
| title_full | A Novel Sine Step Size for Warm-Restart Stochastic Gradient Descent |
| title_fullStr | A Novel Sine Step Size for Warm-Restart Stochastic Gradient Descent |
| title_full_unstemmed | A Novel Sine Step Size for Warm-Restart Stochastic Gradient Descent |
| title_short | A Novel Sine Step Size for Warm-Restart Stochastic Gradient Descent |
| title_sort | novel sine step size for warm restart stochastic gradient descent |
| topic | stochastic gradient descent decay step size convergence rate |
| url | https://www.mdpi.com/2075-1680/13/12/857 |
| work_keys_str_mv | AT mahsasoheilshamaee anovelsinestepsizeforwarmrestartstochasticgradientdescent AT sajadfathihafshejani anovelsinestepsizeforwarmrestartstochasticgradientdescent AT mahsasoheilshamaee novelsinestepsizeforwarmrestartstochasticgradientdescent AT sajadfathihafshejani novelsinestepsizeforwarmrestartstochasticgradientdescent |