Coding schemes in neural networks learning classification tasks
Abstract Neural networks posses the crucial ability to generate meaningful representations of task-dependent features. Indeed, with appropriate scaling, supervised learning in neural networks can result in strong, task-dependent feature learning. However, the nature of the emergent representations i...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-04-01
|
| Series: | Nature Communications |
| Online Access: | https://doi.org/10.1038/s41467-025-58276-6 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849726146932899840 |
|---|---|
| author | Alexander van Meegen Haim Sompolinsky |
| author_facet | Alexander van Meegen Haim Sompolinsky |
| author_sort | Alexander van Meegen |
| collection | DOAJ |
| description | Abstract Neural networks posses the crucial ability to generate meaningful representations of task-dependent features. Indeed, with appropriate scaling, supervised learning in neural networks can result in strong, task-dependent feature learning. However, the nature of the emergent representations is still unclear. To understand the effect of learning on representations, we investigate fully-connected, wide neural networks learning classification tasks using the Bayesian framework where learning shapes the posterior distribution of the network weights. Consistent with previous findings, our analysis of the feature learning regime (also known as ‘non-lazy’ regime) shows that the networks acquire strong, data-dependent features, denoted as coding schemes, where neuronal responses to each input are dominated by its class membership. Surprisingly, the nature of the coding schemes depends crucially on the neuronal nonlinearity. In linear networks, an analog coding scheme of the task emerges; in nonlinear networks, strong spontaneous symmetry breaking leads to either redundant or sparse coding schemes. Our findings highlight how network properties such as scaling of weights and neuronal nonlinearity can profoundly influence the emergent representations. |
| format | Article |
| id | doaj-art-1dd85e43228c4dd692c39dd5ee69208d |
| institution | DOAJ |
| issn | 2041-1723 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Nature Communications |
| spelling | doaj-art-1dd85e43228c4dd692c39dd5ee69208d2025-08-20T03:10:17ZengNature PortfolioNature Communications2041-17232025-04-0116111210.1038/s41467-025-58276-6Coding schemes in neural networks learning classification tasksAlexander van Meegen0Haim Sompolinsky1Center for Brain Science, Harvard UniversityCenter for Brain Science, Harvard UniversityAbstract Neural networks posses the crucial ability to generate meaningful representations of task-dependent features. Indeed, with appropriate scaling, supervised learning in neural networks can result in strong, task-dependent feature learning. However, the nature of the emergent representations is still unclear. To understand the effect of learning on representations, we investigate fully-connected, wide neural networks learning classification tasks using the Bayesian framework where learning shapes the posterior distribution of the network weights. Consistent with previous findings, our analysis of the feature learning regime (also known as ‘non-lazy’ regime) shows that the networks acquire strong, data-dependent features, denoted as coding schemes, where neuronal responses to each input are dominated by its class membership. Surprisingly, the nature of the coding schemes depends crucially on the neuronal nonlinearity. In linear networks, an analog coding scheme of the task emerges; in nonlinear networks, strong spontaneous symmetry breaking leads to either redundant or sparse coding schemes. Our findings highlight how network properties such as scaling of weights and neuronal nonlinearity can profoundly influence the emergent representations.https://doi.org/10.1038/s41467-025-58276-6 |
| spellingShingle | Alexander van Meegen Haim Sompolinsky Coding schemes in neural networks learning classification tasks Nature Communications |
| title | Coding schemes in neural networks learning classification tasks |
| title_full | Coding schemes in neural networks learning classification tasks |
| title_fullStr | Coding schemes in neural networks learning classification tasks |
| title_full_unstemmed | Coding schemes in neural networks learning classification tasks |
| title_short | Coding schemes in neural networks learning classification tasks |
| title_sort | coding schemes in neural networks learning classification tasks |
| url | https://doi.org/10.1038/s41467-025-58276-6 |
| work_keys_str_mv | AT alexandervanmeegen codingschemesinneuralnetworkslearningclassificationtasks AT haimsompolinsky codingschemesinneuralnetworkslearningclassificationtasks |