Deep supervised, but not unsupervised, models may explain IT cortical representation.
Inferior temporal (IT) cortex in human and nonhuman primates serves visual object recognition. Computational object-vision models, although continually improving, do not yet reach human performance. It is unclear to what extent the internal representations of computational models can explain the IT...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2014-11-01
|
| Series: | PLoS Computational Biology |
| Online Access: | https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1003915&type=printable |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849726590763663360 |
|---|---|
| author | Seyed-Mahdi Khaligh-Razavi Nikolaus Kriegeskorte |
| author_facet | Seyed-Mahdi Khaligh-Razavi Nikolaus Kriegeskorte |
| author_sort | Seyed-Mahdi Khaligh-Razavi |
| collection | DOAJ |
| description | Inferior temporal (IT) cortex in human and nonhuman primates serves visual object recognition. Computational object-vision models, although continually improving, do not yet reach human performance. It is unclear to what extent the internal representations of computational models can explain the IT representation. Here we investigate a wide range of computational model representations (37 in total), testing their categorization performance and their ability to account for the IT representational geometry. The models include well-known neuroscientific object-recognition models (e.g. HMAX, VisNet) along with several models from computer vision (e.g. SIFT, GIST, self-similarity features, and a deep convolutional neural network). We compared the representational dissimilarity matrices (RDMs) of the model representations with the RDMs obtained from human IT (measured with fMRI) and monkey IT (measured with cell recording) for the same set of stimuli (not used in training the models). Better performing models were more similar to IT in that they showed greater clustering of representational patterns by category. In addition, better performing models also more strongly resembled IT in terms of their within-category representational dissimilarities. Representational geometries were significantly correlated between IT and many of the models. However, the categorical clustering observed in IT was largely unexplained by the unsupervised models. The deep convolutional network, which was trained by supervision with over a million category-labeled images, reached the highest categorization performance and also best explained IT, although it did not fully explain the IT data. Combining the features of this model with appropriate weights and adding linear combinations that maximize the margin between animate and inanimate objects and between faces and other objects yielded a representation that fully explained our IT data. Overall, our results suggest that explaining IT requires computational features trained through supervised learning to emphasize the behaviorally important categorical divisions prominently reflected in IT. |
| format | Article |
| id | doaj-art-55a1278ee6e742e3b2d53f0bf4c39051 |
| institution | DOAJ |
| issn | 1553-734X 1553-7358 |
| language | English |
| publishDate | 2014-11-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Computational Biology |
| spelling | doaj-art-55a1278ee6e742e3b2d53f0bf4c390512025-08-20T03:10:07ZengPublic Library of Science (PLoS)PLoS Computational Biology1553-734X1553-73582014-11-011011e100391510.1371/journal.pcbi.1003915Deep supervised, but not unsupervised, models may explain IT cortical representation.Seyed-Mahdi Khaligh-RazaviNikolaus KriegeskorteInferior temporal (IT) cortex in human and nonhuman primates serves visual object recognition. Computational object-vision models, although continually improving, do not yet reach human performance. It is unclear to what extent the internal representations of computational models can explain the IT representation. Here we investigate a wide range of computational model representations (37 in total), testing their categorization performance and their ability to account for the IT representational geometry. The models include well-known neuroscientific object-recognition models (e.g. HMAX, VisNet) along with several models from computer vision (e.g. SIFT, GIST, self-similarity features, and a deep convolutional neural network). We compared the representational dissimilarity matrices (RDMs) of the model representations with the RDMs obtained from human IT (measured with fMRI) and monkey IT (measured with cell recording) for the same set of stimuli (not used in training the models). Better performing models were more similar to IT in that they showed greater clustering of representational patterns by category. In addition, better performing models also more strongly resembled IT in terms of their within-category representational dissimilarities. Representational geometries were significantly correlated between IT and many of the models. However, the categorical clustering observed in IT was largely unexplained by the unsupervised models. The deep convolutional network, which was trained by supervision with over a million category-labeled images, reached the highest categorization performance and also best explained IT, although it did not fully explain the IT data. Combining the features of this model with appropriate weights and adding linear combinations that maximize the margin between animate and inanimate objects and between faces and other objects yielded a representation that fully explained our IT data. Overall, our results suggest that explaining IT requires computational features trained through supervised learning to emphasize the behaviorally important categorical divisions prominently reflected in IT.https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1003915&type=printable |
| spellingShingle | Seyed-Mahdi Khaligh-Razavi Nikolaus Kriegeskorte Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Computational Biology |
| title | Deep supervised, but not unsupervised, models may explain IT cortical representation. |
| title_full | Deep supervised, but not unsupervised, models may explain IT cortical representation. |
| title_fullStr | Deep supervised, but not unsupervised, models may explain IT cortical representation. |
| title_full_unstemmed | Deep supervised, but not unsupervised, models may explain IT cortical representation. |
| title_short | Deep supervised, but not unsupervised, models may explain IT cortical representation. |
| title_sort | deep supervised but not unsupervised models may explain it cortical representation |
| url | https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1003915&type=printable |
| work_keys_str_mv | AT seyedmahdikhalighrazavi deepsupervisedbutnotunsupervisedmodelsmayexplainitcorticalrepresentation AT nikolauskriegeskorte deepsupervisedbutnotunsupervisedmodelsmayexplainitcorticalrepresentation |