A Step Towards Neuroplasticity: Capsule Networks with Self-Building Skip Connections
<b>Background:</b> Integrating nonlinear behavior into the architecture of artificial neural networks is regarded as essential requirement to constitute their effectual learning capacity for solving complex tasks. This claim seems to be true for moderate-sized networks, i.e., with a lowe...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | AI |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2673-2688/6/1/1 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832589391248228352 |
|---|---|
| author | Nikolai A. K. Steur Friedhelm Schwenker |
| author_facet | Nikolai A. K. Steur Friedhelm Schwenker |
| author_sort | Nikolai A. K. Steur |
| collection | DOAJ |
| description | <b>Background:</b> Integrating nonlinear behavior into the architecture of artificial neural networks is regarded as essential requirement to constitute their effectual learning capacity for solving complex tasks. This claim seems to be true for moderate-sized networks, i.e., with a lower double-digit number of layers. However, going deeper with neural networks regularly turns into destructive tendencies of gradual performance degeneration during training. To circumvent this degradation problem, the prominent neural architectures <i>Residual Network</i> and <i>Highway Network</i> establish skip connections with additive identity mappings between layers. <b>Methods:</b> In this work, we unify the mechanics of both architectures into Capsule Networks (CapsNet)s by showing their inherent ability to learn skip connections. As a necessary precondition, we introduce the concept of Adaptive Nonlinearity Gates (ANG)s which dynamically steer and limit the usage of nonlinear processing. We propose practical methods for the realization of ANGs including biased batch normalization, the Doubly-Parametric ReLU (D-PReLU) activation function, and Gated Routing (GR) dedicated to extremely deep CapsNets. <b>Results:</b> Our comprehensive empirical study using MNIST substantiates the effectiveness of our developed methods and delivers valuable insights for the training of very deep nets of any kind. The final experiments on Fashion-MNIST and SVHN demonstrate the potential of pure capsule-driven networks with GR. |
| format | Article |
| id | doaj-art-3bfb0cef522d43e9bf04fbfb17c1a922 |
| institution | Kabale University |
| issn | 2673-2688 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | AI |
| spelling | doaj-art-3bfb0cef522d43e9bf04fbfb17c1a9222025-01-24T13:17:21ZengMDPI AGAI2673-26882024-12-0161110.3390/ai6010001A Step Towards Neuroplasticity: Capsule Networks with Self-Building Skip ConnectionsNikolai A. K. Steur0Friedhelm Schwenker1Institute of Neural Information Processing, Ulm University, James-Franck-Ring, 89081 Ulm, Baden-Württemberg, GermanyInstitute of Neural Information Processing, Ulm University, James-Franck-Ring, 89081 Ulm, Baden-Württemberg, Germany<b>Background:</b> Integrating nonlinear behavior into the architecture of artificial neural networks is regarded as essential requirement to constitute their effectual learning capacity for solving complex tasks. This claim seems to be true for moderate-sized networks, i.e., with a lower double-digit number of layers. However, going deeper with neural networks regularly turns into destructive tendencies of gradual performance degeneration during training. To circumvent this degradation problem, the prominent neural architectures <i>Residual Network</i> and <i>Highway Network</i> establish skip connections with additive identity mappings between layers. <b>Methods:</b> In this work, we unify the mechanics of both architectures into Capsule Networks (CapsNet)s by showing their inherent ability to learn skip connections. As a necessary precondition, we introduce the concept of Adaptive Nonlinearity Gates (ANG)s which dynamically steer and limit the usage of nonlinear processing. We propose practical methods for the realization of ANGs including biased batch normalization, the Doubly-Parametric ReLU (D-PReLU) activation function, and Gated Routing (GR) dedicated to extremely deep CapsNets. <b>Results:</b> Our comprehensive empirical study using MNIST substantiates the effectiveness of our developed methods and delivers valuable insights for the training of very deep nets of any kind. The final experiments on Fashion-MNIST and SVHN demonstrate the potential of pure capsule-driven networks with GR.https://www.mdpi.com/2673-2688/6/1/1batch normalizationcapsule networkdegradation probleminformation diffusionparametric activation functionresidual learning |
| spellingShingle | Nikolai A. K. Steur Friedhelm Schwenker A Step Towards Neuroplasticity: Capsule Networks with Self-Building Skip Connections AI batch normalization capsule network degradation problem information diffusion parametric activation function residual learning |
| title | A Step Towards Neuroplasticity: Capsule Networks with Self-Building Skip Connections |
| title_full | A Step Towards Neuroplasticity: Capsule Networks with Self-Building Skip Connections |
| title_fullStr | A Step Towards Neuroplasticity: Capsule Networks with Self-Building Skip Connections |
| title_full_unstemmed | A Step Towards Neuroplasticity: Capsule Networks with Self-Building Skip Connections |
| title_short | A Step Towards Neuroplasticity: Capsule Networks with Self-Building Skip Connections |
| title_sort | step towards neuroplasticity capsule networks with self building skip connections |
| topic | batch normalization capsule network degradation problem information diffusion parametric activation function residual learning |
| url | https://www.mdpi.com/2673-2688/6/1/1 |
| work_keys_str_mv | AT nikolaiaksteur asteptowardsneuroplasticitycapsulenetworkswithselfbuildingskipconnections AT friedhelmschwenker asteptowardsneuroplasticitycapsulenetworkswithselfbuildingskipconnections AT nikolaiaksteur steptowardsneuroplasticitycapsulenetworkswithselfbuildingskipconnections AT friedhelmschwenker steptowardsneuroplasticitycapsulenetworkswithselfbuildingskipconnections |