Brain-model neural similarity reveals abstractive summarization performance
Abstract Deep language models (DLMs) have exhibited remarkable language understanding and generation capabilities, prompting researchers to explore the similarities between their internal mechanisms and human language cognitive processing. This study investigated the representational similarity (RS)...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-01-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-024-84530-w |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841559604242153472 |
|---|---|
| author | Zhejun Zhang Shaoting Guo Wenqing Zhou Yingying Luo Yingqi Zhu Lin Zhang Lei Li |
| author_facet | Zhejun Zhang Shaoting Guo Wenqing Zhou Yingying Luo Yingqi Zhu Lin Zhang Lei Li |
| author_sort | Zhejun Zhang |
| collection | DOAJ |
| description | Abstract Deep language models (DLMs) have exhibited remarkable language understanding and generation capabilities, prompting researchers to explore the similarities between their internal mechanisms and human language cognitive processing. This study investigated the representational similarity (RS) between the abstractive summarization (ABS) models and the human brain and its correlation to the performance of ABS tasks. Specifically, representational similarity analysis (RSA) was used to measure the similarity between the representational patterns (RPs) of the BART, PEGASUS, and T5 models’ hidden layers and the human brain’s language RPs under different spatiotemporal conditions. Layer-wise ablation manipulation, including attention ablation and noise addition was employed to examine the hidden layers’ effect on model performance. The results demonstrate that as the depth of hidden layers increases, the models’ text encoding becomes increasingly similar to the human brain’s language RPs. Manipulating deeper layers leads to more substantial decline in summarization performance compared to shallower layers, highlighting the crucial role of deeper layers in integrating essential information. Notably, the study confirms the hypothesis that the hidden layers exhibiting higher similarity to human brain activity play a more critical role in model performance, with their correlations reaching statistical significance even after controlling for perplexity. These findings deepen our understanding of the cognitive mechanisms underlying language representations in DLMs and their neural correlates, potentially providing insights for optimizing and improving language models by aligning them with the human brain’s language-processing mechanisms. |
| format | Article |
| id | doaj-art-797d370e12a648a5bedad13c678dd8a5 |
| institution | Kabale University |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-797d370e12a648a5bedad13c678dd8a52025-01-05T12:21:33ZengNature PortfolioScientific Reports2045-23222025-01-0115111910.1038/s41598-024-84530-wBrain-model neural similarity reveals abstractive summarization performanceZhejun Zhang0Shaoting Guo1Wenqing Zhou2Yingying Luo3Yingqi Zhu4Lin Zhang5Lei Li6School of Artificial Intelligence, Beijing University of Posts and TelecommunicationsSchool of Artificial Intelligence, Beijing University of Posts and TelecommunicationsSchool of Artificial Intelligence, Beijing University of Posts and TelecommunicationsSchool of Artificial Intelligence, Beijing University of Posts and TelecommunicationsSchool of Artificial Intelligence, Beijing University of Posts and TelecommunicationsSchool of Artificial Intelligence, Beijing University of Posts and TelecommunicationsSchool of Artificial Intelligence, Beijing University of Posts and TelecommunicationsAbstract Deep language models (DLMs) have exhibited remarkable language understanding and generation capabilities, prompting researchers to explore the similarities between their internal mechanisms and human language cognitive processing. This study investigated the representational similarity (RS) between the abstractive summarization (ABS) models and the human brain and its correlation to the performance of ABS tasks. Specifically, representational similarity analysis (RSA) was used to measure the similarity between the representational patterns (RPs) of the BART, PEGASUS, and T5 models’ hidden layers and the human brain’s language RPs under different spatiotemporal conditions. Layer-wise ablation manipulation, including attention ablation and noise addition was employed to examine the hidden layers’ effect on model performance. The results demonstrate that as the depth of hidden layers increases, the models’ text encoding becomes increasingly similar to the human brain’s language RPs. Manipulating deeper layers leads to more substantial decline in summarization performance compared to shallower layers, highlighting the crucial role of deeper layers in integrating essential information. Notably, the study confirms the hypothesis that the hidden layers exhibiting higher similarity to human brain activity play a more critical role in model performance, with their correlations reaching statistical significance even after controlling for perplexity. These findings deepen our understanding of the cognitive mechanisms underlying language representations in DLMs and their neural correlates, potentially providing insights for optimizing and improving language models by aligning them with the human brain’s language-processing mechanisms.https://doi.org/10.1038/s41598-024-84530-wDeep language modelsRepresentational similarity analysisAbstractive summarizationElectroencephalographyNeural correlates |
| spellingShingle | Zhejun Zhang Shaoting Guo Wenqing Zhou Yingying Luo Yingqi Zhu Lin Zhang Lei Li Brain-model neural similarity reveals abstractive summarization performance Scientific Reports Deep language models Representational similarity analysis Abstractive summarization Electroencephalography Neural correlates |
| title | Brain-model neural similarity reveals abstractive summarization performance |
| title_full | Brain-model neural similarity reveals abstractive summarization performance |
| title_fullStr | Brain-model neural similarity reveals abstractive summarization performance |
| title_full_unstemmed | Brain-model neural similarity reveals abstractive summarization performance |
| title_short | Brain-model neural similarity reveals abstractive summarization performance |
| title_sort | brain model neural similarity reveals abstractive summarization performance |
| topic | Deep language models Representational similarity analysis Abstractive summarization Electroencephalography Neural correlates |
| url | https://doi.org/10.1038/s41598-024-84530-w |
| work_keys_str_mv | AT zhejunzhang brainmodelneuralsimilarityrevealsabstractivesummarizationperformance AT shaotingguo brainmodelneuralsimilarityrevealsabstractivesummarizationperformance AT wenqingzhou brainmodelneuralsimilarityrevealsabstractivesummarizationperformance AT yingyingluo brainmodelneuralsimilarityrevealsabstractivesummarizationperformance AT yingqizhu brainmodelneuralsimilarityrevealsabstractivesummarizationperformance AT linzhang brainmodelneuralsimilarityrevealsabstractivesummarizationperformance AT leili brainmodelneuralsimilarityrevealsabstractivesummarizationperformance |