Effective theory of collective deep learning
Unraveling the emergence of collective learning in systems of coupled artificial neural networks points to broader implications for physics, machine learning, neuroscience, and society. Here we introduce a minimal model of interacting deep neural nets that condenses several recent decentralized algo...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Physical Society
2024-11-01
|
| Series: | Physical Review Research |
| Online Access: | http://doi.org/10.1103/PhysRevResearch.6.L042040 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850191706267320320 |
|---|---|
| author | Lluís Arola-Fernández Lucas Lacasa |
| author_facet | Lluís Arola-Fernández Lucas Lacasa |
| author_sort | Lluís Arola-Fernández |
| collection | DOAJ |
| description | Unraveling the emergence of collective learning in systems of coupled artificial neural networks points to broader implications for physics, machine learning, neuroscience, and society. Here we introduce a minimal model of interacting deep neural nets that condenses several recent decentralized algorithms by considering a competition between two terms: the local learning dynamics in the parameters of each neural network unit, and a diffusive coupling among units that tends to homogenize the parameters of the ensemble. We derive an effective theory for linear networks to show that the coarse-grained behavior of our system is equivalent to a deformed Ginzburg-Landau model with quenched disorder. This framework predicts depth-dependent disorder-order-disorder phase transitions in the parameters' solutions that reveal a depth-delayed onset of a collective learning phase and a low-rank microscopic learning path. We validate the theory in coupled ensembles of realistic neural networks trained on MNIST and CIFAR-10 datasets under privacy constraints. Interestingly, experiments confirm that individual networks–trained on private data–can fully generalize to unseen data classes when the collective learning phase emerges. Our work elucidates the physics of collective learning and contributes to the mechanistic interpretability of deep learning in decentralized settings. |
| format | Article |
| id | doaj-art-88b3ce3a2fc94b578546a8849ce3c0a3 |
| institution | OA Journals |
| issn | 2643-1564 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | American Physical Society |
| record_format | Article |
| series | Physical Review Research |
| spelling | doaj-art-88b3ce3a2fc94b578546a8849ce3c0a32025-08-20T02:14:49ZengAmerican Physical SocietyPhysical Review Research2643-15642024-11-0164L04204010.1103/PhysRevResearch.6.L042040Effective theory of collective deep learningLluís Arola-FernándezLucas LacasaUnraveling the emergence of collective learning in systems of coupled artificial neural networks points to broader implications for physics, machine learning, neuroscience, and society. Here we introduce a minimal model of interacting deep neural nets that condenses several recent decentralized algorithms by considering a competition between two terms: the local learning dynamics in the parameters of each neural network unit, and a diffusive coupling among units that tends to homogenize the parameters of the ensemble. We derive an effective theory for linear networks to show that the coarse-grained behavior of our system is equivalent to a deformed Ginzburg-Landau model with quenched disorder. This framework predicts depth-dependent disorder-order-disorder phase transitions in the parameters' solutions that reveal a depth-delayed onset of a collective learning phase and a low-rank microscopic learning path. We validate the theory in coupled ensembles of realistic neural networks trained on MNIST and CIFAR-10 datasets under privacy constraints. Interestingly, experiments confirm that individual networks–trained on private data–can fully generalize to unseen data classes when the collective learning phase emerges. Our work elucidates the physics of collective learning and contributes to the mechanistic interpretability of deep learning in decentralized settings.http://doi.org/10.1103/PhysRevResearch.6.L042040 |
| spellingShingle | Lluís Arola-Fernández Lucas Lacasa Effective theory of collective deep learning Physical Review Research |
| title | Effective theory of collective deep learning |
| title_full | Effective theory of collective deep learning |
| title_fullStr | Effective theory of collective deep learning |
| title_full_unstemmed | Effective theory of collective deep learning |
| title_short | Effective theory of collective deep learning |
| title_sort | effective theory of collective deep learning |
| url | http://doi.org/10.1103/PhysRevResearch.6.L042040 |
| work_keys_str_mv | AT lluisarolafernandez effectivetheoryofcollectivedeeplearning AT lucaslacasa effectivetheoryofcollectivedeeplearning |