Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze
In this study, we investigate the adaptability of artificial agents within a noisy T-maze that use Markov decision processes (MDPs) and successor feature (SF) and predecessor feature (PF) learning algorithms. Our focus is on quantifying how varying the hyperparameters, specifically the reward learni...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-10-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/24/19/6419 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850284200598437888 |
|---|---|
| author | Incheol Seo Hyunsu Lee |
| author_facet | Incheol Seo Hyunsu Lee |
| author_sort | Incheol Seo |
| collection | DOAJ |
| description | In this study, we investigate the adaptability of artificial agents within a noisy T-maze that use Markov decision processes (MDPs) and successor feature (SF) and predecessor feature (PF) learning algorithms. Our focus is on quantifying how varying the hyperparameters, specifically the reward learning rate (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>α</mi><mi>r</mi></msub></semantics></math></inline-formula>) and the eligibility trace decay rate (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>λ</mi></semantics></math></inline-formula>), can enhance their adaptability. Adaptation is evaluated by analyzing the hyperparameters of cumulative reward, step length, adaptation rate, and adaptation step length and the relationships between them using Spearman’s correlation tests and linear regression. Our findings reveal that an <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>α</mi><mi>r</mi></msub></semantics></math></inline-formula> of 0.9 consistently yields superior adaptation across all metrics at a noise level of 0.05. However, the optimal setting for <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>λ</mi></semantics></math></inline-formula> varies by metric and context. In discussing these results, we emphasize the critical role of hyperparameter optimization in refining the performance and transfer learning efficacy of learning algorithms. This research advances our understanding of the functionality of PF and SF algorithms, particularly in navigating the inherent uncertainty of transfer learning tasks. By offering insights into the optimal hyperparameter configurations, this study contributes to the development of more adaptive and robust learning algorithms, paving the way for future explorations in artificial intelligence and neuroscience. |
| format | Article |
| id | doaj-art-77473eb5bfab470fa6f5648faf36aac1 |
| institution | OA Journals |
| issn | 1424-8220 |
| language | English |
| publishDate | 2024-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj-art-77473eb5bfab470fa6f5648faf36aac12025-08-20T01:47:37ZengMDPI AGSensors1424-82202024-10-012419641910.3390/s24196419Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-MazeIncheol Seo0Hyunsu Lee1Department of Immunology, Kyungpook National University School of Medicine, Daegu 41944, Republic of KoreaDepartment of Physiology, Pusan National University School of Medicine, Yangsan 50612, Republic of KoreaIn this study, we investigate the adaptability of artificial agents within a noisy T-maze that use Markov decision processes (MDPs) and successor feature (SF) and predecessor feature (PF) learning algorithms. Our focus is on quantifying how varying the hyperparameters, specifically the reward learning rate (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>α</mi><mi>r</mi></msub></semantics></math></inline-formula>) and the eligibility trace decay rate (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>λ</mi></semantics></math></inline-formula>), can enhance their adaptability. Adaptation is evaluated by analyzing the hyperparameters of cumulative reward, step length, adaptation rate, and adaptation step length and the relationships between them using Spearman’s correlation tests and linear regression. Our findings reveal that an <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>α</mi><mi>r</mi></msub></semantics></math></inline-formula> of 0.9 consistently yields superior adaptation across all metrics at a noise level of 0.05. However, the optimal setting for <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>λ</mi></semantics></math></inline-formula> varies by metric and context. In discussing these results, we emphasize the critical role of hyperparameter optimization in refining the performance and transfer learning efficacy of learning algorithms. This research advances our understanding of the functionality of PF and SF algorithms, particularly in navigating the inherent uncertainty of transfer learning tasks. By offering insights into the optimal hyperparameter configurations, this study contributes to the development of more adaptive and robust learning algorithms, paving the way for future explorations in artificial intelligence and neuroscience.https://www.mdpi.com/1424-8220/24/19/6419reinforcement learningT-maze transfer learninghyperparameter tuningrobustnesspredecessor featuresnoisy observation |
| spellingShingle | Incheol Seo Hyunsu Lee Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze Sensors reinforcement learning T-maze transfer learning hyperparameter tuning robustness predecessor features noisy observation |
| title | Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze |
| title_full | Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze |
| title_fullStr | Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze |
| title_full_unstemmed | Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze |
| title_short | Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze |
| title_sort | investigating transfer learning in noisy environments a study of predecessor and successor features in spatial learning using a t maze |
| topic | reinforcement learning T-maze transfer learning hyperparameter tuning robustness predecessor features noisy observation |
| url | https://www.mdpi.com/1424-8220/24/19/6419 |
| work_keys_str_mv | AT incheolseo investigatingtransferlearninginnoisyenvironmentsastudyofpredecessorandsuccessorfeaturesinspatiallearningusingatmaze AT hyunsulee investigatingtransferlearninginnoisyenvironmentsastudyofpredecessorandsuccessorfeaturesinspatiallearningusingatmaze |