Memory–Non-Linearity Trade-Off in Distance-Based Delay Networks
The performance of echo state networks (ESNs) in temporal pattern learning tasks depends both on their memory capacity (MC) and their non-linear processing. It has been shown that linear memory capacity is maximized when ESN neurons have linear activation, and that a trade-off between non-linearity...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Biomimetics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2313-7673/9/12/755 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850059192376754176 |
|---|---|
| author | Stefan Iacob Joni Dambre |
| author_facet | Stefan Iacob Joni Dambre |
| author_sort | Stefan Iacob |
| collection | DOAJ |
| description | The performance of echo state networks (ESNs) in temporal pattern learning tasks depends both on their memory capacity (MC) and their non-linear processing. It has been shown that linear memory capacity is maximized when ESN neurons have linear activation, and that a trade-off between non-linearity and linear memory capacity is required for temporal pattern learning tasks. The more recent distance-based delay networks (DDNs) have shown improved memory capacity over ESNs in several benchmark temporal pattern learning tasks. However, it has not thus far been studied whether this increased memory capacity comes at the cost of reduced non-linear processing. In this paper, we advance the hypothesis that DDNs in fact achieve a better trade-off between linear MC and non-linearity than ESNs, by showing that DDNs can have strong non-linearity with large memory spans. We tested this hypothesis using the NARMA-30 task and the bitwise delayed XOR task, two commonly used reservoir benchmark tasks that require a high degree of both non-linearity and memory. |
| format | Article |
| id | doaj-art-20dbcaac15954b74a95cbf384d4b971f |
| institution | DOAJ |
| issn | 2313-7673 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Biomimetics |
| spelling | doaj-art-20dbcaac15954b74a95cbf384d4b971f2025-08-20T02:50:56ZengMDPI AGBiomimetics2313-76732024-12-0191275510.3390/biomimetics9120755Memory–Non-Linearity Trade-Off in Distance-Based Delay NetworksStefan Iacob0Joni Dambre1IDLab-AIRO, Faculty of Engineering and Architecture, Ghent University, 9052 Ghent, BelgiumIDLab-AIRO, Faculty of Engineering and Architecture, Ghent University, 9052 Ghent, BelgiumThe performance of echo state networks (ESNs) in temporal pattern learning tasks depends both on their memory capacity (MC) and their non-linear processing. It has been shown that linear memory capacity is maximized when ESN neurons have linear activation, and that a trade-off between non-linearity and linear memory capacity is required for temporal pattern learning tasks. The more recent distance-based delay networks (DDNs) have shown improved memory capacity over ESNs in several benchmark temporal pattern learning tasks. However, it has not thus far been studied whether this increased memory capacity comes at the cost of reduced non-linear processing. In this paper, we advance the hypothesis that DDNs in fact achieve a better trade-off between linear MC and non-linearity than ESNs, by showing that DDNs can have strong non-linearity with large memory spans. We tested this hypothesis using the NARMA-30 task and the bitwise delayed XOR task, two commonly used reservoir benchmark tasks that require a high degree of both non-linearity and memory.https://www.mdpi.com/2313-7673/9/12/755information processing capacitymemory capacitymemory–non-linearity trade-offreservoir computingdistance-based delay networksecho state networks |
| spellingShingle | Stefan Iacob Joni Dambre Memory–Non-Linearity Trade-Off in Distance-Based Delay Networks Biomimetics information processing capacity memory capacity memory–non-linearity trade-off reservoir computing distance-based delay networks echo state networks |
| title | Memory–Non-Linearity Trade-Off in Distance-Based Delay Networks |
| title_full | Memory–Non-Linearity Trade-Off in Distance-Based Delay Networks |
| title_fullStr | Memory–Non-Linearity Trade-Off in Distance-Based Delay Networks |
| title_full_unstemmed | Memory–Non-Linearity Trade-Off in Distance-Based Delay Networks |
| title_short | Memory–Non-Linearity Trade-Off in Distance-Based Delay Networks |
| title_sort | memory non linearity trade off in distance based delay networks |
| topic | information processing capacity memory capacity memory–non-linearity trade-off reservoir computing distance-based delay networks echo state networks |
| url | https://www.mdpi.com/2313-7673/9/12/755 |
| work_keys_str_mv | AT stefaniacob memorynonlinearitytradeoffindistancebaseddelaynetworks AT jonidambre memorynonlinearitytradeoffindistancebaseddelaynetworks |