A large-scale examination of inductive biases shaping high-level visual representation in brains and machines
Abstract The rapid release of high-performing computer vision models offers new potential to study the impact of different inductive biases on the emergent brain alignment of learned representations. Here, we perform controlled comparisons among a curated set of 224 diverse models to test the impact...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2024-10-01
|
| Series: | Nature Communications |
| Online Access: | https://doi.org/10.1038/s41467-024-53147-y |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850179228309389312 |
|---|---|
| author | Colin Conwell Jacob S. Prince Kendrick N. Kay George A. Alvarez Talia Konkle |
| author_facet | Colin Conwell Jacob S. Prince Kendrick N. Kay George A. Alvarez Talia Konkle |
| author_sort | Colin Conwell |
| collection | DOAJ |
| description | Abstract The rapid release of high-performing computer vision models offers new potential to study the impact of different inductive biases on the emergent brain alignment of learned representations. Here, we perform controlled comparisons among a curated set of 224 diverse models to test the impact of specific model properties on visual brain predictivity – a process requiring over 1.8 billion regressions and 50.3 thousand representational similarity analyses. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and task objectives (e.g. purely visual contrastive learning versus vision- language alignment) achieve near equivalent brain predictivity, when other factors are held constant. Instead, variation across visual training diets yields the largest, most consistent effect on brain predictivity. Many models achieve similarly high brain predictivity, despite clear variation in their underlying representations – suggesting that standard methods used to link models to brains may be too flexible. Broadly, these findings challenge common assumptions about the factors underlying emergent brain alignment, and outline how we can leverage controlled model comparison to probe the common computational principles underlying biological and artificial visual systems. |
| format | Article |
| id | doaj-art-4a9d9673eef74db08c31173cc312dbd5 |
| institution | OA Journals |
| issn | 2041-1723 |
| language | English |
| publishDate | 2024-10-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Nature Communications |
| spelling | doaj-art-4a9d9673eef74db08c31173cc312dbd52025-08-20T02:18:33ZengNature PortfolioNature Communications2041-17232024-10-0115111810.1038/s41467-024-53147-yA large-scale examination of inductive biases shaping high-level visual representation in brains and machinesColin Conwell0Jacob S. Prince1Kendrick N. Kay2George A. Alvarez3Talia Konkle4Department of Psychology, Harvard UniversityDepartment of Psychology, Harvard UniversityCenter for Magnetic Resonance Research, Department of Radiology, University of MinnesotaDepartment of Psychology, Harvard UniversityDepartment of Psychology, Harvard UniversityAbstract The rapid release of high-performing computer vision models offers new potential to study the impact of different inductive biases on the emergent brain alignment of learned representations. Here, we perform controlled comparisons among a curated set of 224 diverse models to test the impact of specific model properties on visual brain predictivity – a process requiring over 1.8 billion regressions and 50.3 thousand representational similarity analyses. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and task objectives (e.g. purely visual contrastive learning versus vision- language alignment) achieve near equivalent brain predictivity, when other factors are held constant. Instead, variation across visual training diets yields the largest, most consistent effect on brain predictivity. Many models achieve similarly high brain predictivity, despite clear variation in their underlying representations – suggesting that standard methods used to link models to brains may be too flexible. Broadly, these findings challenge common assumptions about the factors underlying emergent brain alignment, and outline how we can leverage controlled model comparison to probe the common computational principles underlying biological and artificial visual systems.https://doi.org/10.1038/s41467-024-53147-y |
| spellingShingle | Colin Conwell Jacob S. Prince Kendrick N. Kay George A. Alvarez Talia Konkle A large-scale examination of inductive biases shaping high-level visual representation in brains and machines Nature Communications |
| title | A large-scale examination of inductive biases shaping high-level visual representation in brains and machines |
| title_full | A large-scale examination of inductive biases shaping high-level visual representation in brains and machines |
| title_fullStr | A large-scale examination of inductive biases shaping high-level visual representation in brains and machines |
| title_full_unstemmed | A large-scale examination of inductive biases shaping high-level visual representation in brains and machines |
| title_short | A large-scale examination of inductive biases shaping high-level visual representation in brains and machines |
| title_sort | large scale examination of inductive biases shaping high level visual representation in brains and machines |
| url | https://doi.org/10.1038/s41467-024-53147-y |
| work_keys_str_mv | AT colinconwell alargescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines AT jacobsprince alargescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines AT kendricknkay alargescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines AT georgeaalvarez alargescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines AT taliakonkle alargescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines AT colinconwell largescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines AT jacobsprince largescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines AT kendricknkay largescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines AT georgeaalvarez largescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines AT taliakonkle largescaleexaminationofinductivebiasesshapinghighlevelvisualrepresentationinbrainsandmachines |