Concatenated Attention: A Novel Method for Regulating Information Structure Based on Sensors
This paper addresses the challenges of limited training data and suboptimal environmental conditions in image processing tasks, such as underwater imaging with poor lighting and distortion. Neural networks, including Convolutional Neural Networks (CNNs) and Transformers, have advanced image analysis...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-01-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/2/523 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832589283136897024 |
|---|---|
| author | Zeyu Zhang Tianqi Chen Yuki Todo |
| author_facet | Zeyu Zhang Tianqi Chen Yuki Todo |
| author_sort | Zeyu Zhang |
| collection | DOAJ |
| description | This paper addresses the challenges of limited training data and suboptimal environmental conditions in image processing tasks, such as underwater imaging with poor lighting and distortion. Neural networks, including Convolutional Neural Networks (CNNs) and Transformers, have advanced image analysis but remain constrained by computational demands and insufficient data. To overcome these limitations, we propose a novel split-and-concatenate method for self-attention mechanisms. By splitting Query and Key matrices into submatrices, performing cross-multiplications, and applying weighted summation, the method optimizes intermediate variables without increasing computational costs. Experiments on a real-world crack dataset demonstrate its effectiveness in improving network performance. |
| format | Article |
| id | doaj-art-ad701b6993da414eb9c1b562de569fd3 |
| institution | Kabale University |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-ad701b6993da414eb9c1b562de569fd32025-01-24T13:19:41ZengMDPI AGApplied Sciences2076-34172025-01-0115252310.3390/app15020523Concatenated Attention: A Novel Method for Regulating Information Structure Based on SensorsZeyu Zhang0Tianqi Chen1Yuki Todo2Division of Electrical Engineering and Computer Science, Kanazawa University, Kanazawa 9201192, JapanDivision of Electrical Engineering and Computer Science, Kanazawa University, Kanazawa 9201192, JapanFaculty of Electrical, Information and Communication Engineering, Kanazawa University, Kanazawa 9201192, JapanThis paper addresses the challenges of limited training data and suboptimal environmental conditions in image processing tasks, such as underwater imaging with poor lighting and distortion. Neural networks, including Convolutional Neural Networks (CNNs) and Transformers, have advanced image analysis but remain constrained by computational demands and insufficient data. To overcome these limitations, we propose a novel split-and-concatenate method for self-attention mechanisms. By splitting Query and Key matrices into submatrices, performing cross-multiplications, and applying weighted summation, the method optimizes intermediate variables without increasing computational costs. Experiments on a real-world crack dataset demonstrate its effectiveness in improving network performance.https://www.mdpi.com/2076-3417/15/2/523self-attentiontransformerneural networksensor |
| spellingShingle | Zeyu Zhang Tianqi Chen Yuki Todo Concatenated Attention: A Novel Method for Regulating Information Structure Based on Sensors Applied Sciences self-attention transformer neural network sensor |
| title | Concatenated Attention: A Novel Method for Regulating Information Structure Based on Sensors |
| title_full | Concatenated Attention: A Novel Method for Regulating Information Structure Based on Sensors |
| title_fullStr | Concatenated Attention: A Novel Method for Regulating Information Structure Based on Sensors |
| title_full_unstemmed | Concatenated Attention: A Novel Method for Regulating Information Structure Based on Sensors |
| title_short | Concatenated Attention: A Novel Method for Regulating Information Structure Based on Sensors |
| title_sort | concatenated attention a novel method for regulating information structure based on sensors |
| topic | self-attention transformer neural network sensor |
| url | https://www.mdpi.com/2076-3417/15/2/523 |
| work_keys_str_mv | AT zeyuzhang concatenatedattentionanovelmethodforregulatinginformationstructurebasedonsensors AT tianqichen concatenatedattentionanovelmethodforregulatinginformationstructurebasedonsensors AT yukitodo concatenatedattentionanovelmethodforregulatinginformationstructurebasedonsensors |