Survey on Replay-Based Continual Learning and Empirical Validation on Feasibility in Diverse Edge Devices Using a Representative Method
The goal of on-device continual learning is to enable models to adapt to streaming data without forgetting previously acquired knowledge, even with limited computational resources and memory constraints. Recent research has demonstrated that weighted regularization-based methods are constrained by i...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-07-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/13/14/2257 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849406872763760640 |
|---|---|
| author | Heon-Sung Park Hyeon-Chang Chu Min-Kyung Sung Chaewoon Kim Jeongwon Lee Dae-Won Kim Jaesung Lee |
| author_facet | Heon-Sung Park Hyeon-Chang Chu Min-Kyung Sung Chaewoon Kim Jeongwon Lee Dae-Won Kim Jaesung Lee |
| author_sort | Heon-Sung Park |
| collection | DOAJ |
| description | The goal of on-device continual learning is to enable models to adapt to streaming data without forgetting previously acquired knowledge, even with limited computational resources and memory constraints. Recent research has demonstrated that weighted regularization-based methods are constrained by indirect knowledge preservation and sensitive hyperparameter settings, and dynamic architecture methods are ill-suited for on-device environments due to increased resource consumption as the structure scales. In order to compensate for these limitations, replay-based continuous learning, which maintains a compact structure and stable performance, is gaining attention. The limitations of replay-based continuous learning are (1) the limited amount of historical training data that can be stored due to limited memory capacity, and (2) the computational resources of on-device systems are significantly lower than those of servers or cloud infrastructures. Consequently, designing strategies that balance the preservation of past knowledge with rapid and cost-effective updates of model parameters has become a critical consideration in on-device continual learning. This paper presents an empirical survey of replay-based continual learning studies, considering the nearest class mean classifier with replay-based sparse weight updates as a representative method for validating the feasibility of diverse edge devices. Our empirical comparison of standard benchmarks, including CIFAR-10, CIFAR-100, and TinyImageNet, deployed on devices such as Jetson Nano and Raspberry Pi, showed that the proposed representative method achieved reasonable accuracy under limited buffer sizes compared with existing replay-based techniques. A significant reduction in training time and resource consumption was observed, thereby supporting the feasibility of replay-based on-device continual learning in practice. |
| format | Article |
| id | doaj-art-e4f675e58eb942e9a04f552805446606 |
| institution | Kabale University |
| issn | 2227-7390 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj-art-e4f675e58eb942e9a04f5528054466062025-08-20T03:36:14ZengMDPI AGMathematics2227-73902025-07-011314225710.3390/math13142257Survey on Replay-Based Continual Learning and Empirical Validation on Feasibility in Diverse Edge Devices Using a Representative MethodHeon-Sung Park0Hyeon-Chang Chu1Min-Kyung Sung2Chaewoon Kim3Jeongwon Lee4Dae-Won Kim5Jaesung Lee6School of Computer Science and Engineering, Chung-Ang University, Dongjak-gu, Seoul 06974, Republic of KoreaSchool of Computer Science and Engineering, Chung-Ang University, Dongjak-gu, Seoul 06974, Republic of KoreaDepartment of Artificial Intelligence, Chung-Ang University, Dongjak-gu, Seoul 06974, Republic of KoreaDepartment of Artificial Intelligence, Chung-Ang University, Dongjak-gu, Seoul 06974, Republic of KoreaDepartment of Artificial Intelligence, Chung-Ang University, Dongjak-gu, Seoul 06974, Republic of KoreaSchool of Computer Science and Engineering, Chung-Ang University, Dongjak-gu, Seoul 06974, Republic of KoreaDepartment of Artificial Intelligence, Chung-Ang University, Dongjak-gu, Seoul 06974, Republic of KoreaThe goal of on-device continual learning is to enable models to adapt to streaming data without forgetting previously acquired knowledge, even with limited computational resources and memory constraints. Recent research has demonstrated that weighted regularization-based methods are constrained by indirect knowledge preservation and sensitive hyperparameter settings, and dynamic architecture methods are ill-suited for on-device environments due to increased resource consumption as the structure scales. In order to compensate for these limitations, replay-based continuous learning, which maintains a compact structure and stable performance, is gaining attention. The limitations of replay-based continuous learning are (1) the limited amount of historical training data that can be stored due to limited memory capacity, and (2) the computational resources of on-device systems are significantly lower than those of servers or cloud infrastructures. Consequently, designing strategies that balance the preservation of past knowledge with rapid and cost-effective updates of model parameters has become a critical consideration in on-device continual learning. This paper presents an empirical survey of replay-based continual learning studies, considering the nearest class mean classifier with replay-based sparse weight updates as a representative method for validating the feasibility of diverse edge devices. Our empirical comparison of standard benchmarks, including CIFAR-10, CIFAR-100, and TinyImageNet, deployed on devices such as Jetson Nano and Raspberry Pi, showed that the proposed representative method achieved reasonable accuracy under limited buffer sizes compared with existing replay-based techniques. A significant reduction in training time and resource consumption was observed, thereby supporting the feasibility of replay-based on-device continual learning in practice.https://www.mdpi.com/2227-7390/13/14/2257continual learningreplaysparse updateon-deviceedge device |
| spellingShingle | Heon-Sung Park Hyeon-Chang Chu Min-Kyung Sung Chaewoon Kim Jeongwon Lee Dae-Won Kim Jaesung Lee Survey on Replay-Based Continual Learning and Empirical Validation on Feasibility in Diverse Edge Devices Using a Representative Method Mathematics continual learning replay sparse update on-device edge device |
| title | Survey on Replay-Based Continual Learning and Empirical Validation on Feasibility in Diverse Edge Devices Using a Representative Method |
| title_full | Survey on Replay-Based Continual Learning and Empirical Validation on Feasibility in Diverse Edge Devices Using a Representative Method |
| title_fullStr | Survey on Replay-Based Continual Learning and Empirical Validation on Feasibility in Diverse Edge Devices Using a Representative Method |
| title_full_unstemmed | Survey on Replay-Based Continual Learning and Empirical Validation on Feasibility in Diverse Edge Devices Using a Representative Method |
| title_short | Survey on Replay-Based Continual Learning and Empirical Validation on Feasibility in Diverse Edge Devices Using a Representative Method |
| title_sort | survey on replay based continual learning and empirical validation on feasibility in diverse edge devices using a representative method |
| topic | continual learning replay sparse update on-device edge device |
| url | https://www.mdpi.com/2227-7390/13/14/2257 |
| work_keys_str_mv | AT heonsungpark surveyonreplaybasedcontinuallearningandempiricalvalidationonfeasibilityindiverseedgedevicesusingarepresentativemethod AT hyeonchangchu surveyonreplaybasedcontinuallearningandempiricalvalidationonfeasibilityindiverseedgedevicesusingarepresentativemethod AT minkyungsung surveyonreplaybasedcontinuallearningandempiricalvalidationonfeasibilityindiverseedgedevicesusingarepresentativemethod AT chaewoonkim surveyonreplaybasedcontinuallearningandempiricalvalidationonfeasibilityindiverseedgedevicesusingarepresentativemethod AT jeongwonlee surveyonreplaybasedcontinuallearningandempiricalvalidationonfeasibilityindiverseedgedevicesusingarepresentativemethod AT daewonkim surveyonreplaybasedcontinuallearningandempiricalvalidationonfeasibilityindiverseedgedevicesusingarepresentativemethod AT jaesunglee surveyonreplaybasedcontinuallearningandempiricalvalidationonfeasibilityindiverseedgedevicesusingarepresentativemethod |