HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion
The purpose of infrared and visible image fusion is to combine the advantages of both and generate a fused image that contains target information and has rich details and contrast. However, existing fusion algorithms often overlook the importance of incorporating both local and global feature extrac...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/24/23/7729 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850060047692857344 |
|---|---|
| author | Wenqing Wang Lingzhou Li Yifei Yang Han Liu Runyuan Guo |
| author_facet | Wenqing Wang Lingzhou Li Yifei Yang Han Liu Runyuan Guo |
| author_sort | Wenqing Wang |
| collection | DOAJ |
| description | The purpose of infrared and visible image fusion is to combine the advantages of both and generate a fused image that contains target information and has rich details and contrast. However, existing fusion algorithms often overlook the importance of incorporating both local and global feature extraction, leading to missing key information in the fused image. To address these challenges, this paper proposes a dual-branch fusion network combining convolutional neural network (CNN) and Transformer, which enhances the feature extraction capability and motivates the fused image to contain more information. Firstly, a local feature extraction module with CNN as the core is constructed. Specifically, the residual gradient module is used to enhance the ability of the network to extract texture information. Also, jump links and coordinate attention are used in order to relate shallow features to deeper ones. In addition, a global feature extraction module based on Transformer is constructed. Through the powerful ability of Transformer, the global context information of the image can be captured and the global features are fully extracted. The effectiveness of the proposed method in this paper is verified on different experimental datasets, and it is better than most of the current advanced fusion algorithms. |
| format | Article |
| id | doaj-art-d6eec3fe07ae4f178e1820caaebb95ae |
| institution | DOAJ |
| issn | 1424-8220 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj-art-d6eec3fe07ae4f178e1820caaebb95ae2025-08-20T02:50:41ZengMDPI AGSensors1424-82202024-12-012423772910.3390/s24237729HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image FusionWenqing Wang0Lingzhou Li1Yifei Yang2Han Liu3Runyuan Guo4School of Automation and Information Engineering, Xi’an University of Technology, Xi’an 710048, ChinaSchool of Automation and Information Engineering, Xi’an University of Technology, Xi’an 710048, ChinaSchool of Automation and Information Engineering, Xi’an University of Technology, Xi’an 710048, ChinaSchool of Automation and Information Engineering, Xi’an University of Technology, Xi’an 710048, ChinaSchool of Automation and Information Engineering, Xi’an University of Technology, Xi’an 710048, ChinaThe purpose of infrared and visible image fusion is to combine the advantages of both and generate a fused image that contains target information and has rich details and contrast. However, existing fusion algorithms often overlook the importance of incorporating both local and global feature extraction, leading to missing key information in the fused image. To address these challenges, this paper proposes a dual-branch fusion network combining convolutional neural network (CNN) and Transformer, which enhances the feature extraction capability and motivates the fused image to contain more information. Firstly, a local feature extraction module with CNN as the core is constructed. Specifically, the residual gradient module is used to enhance the ability of the network to extract texture information. Also, jump links and coordinate attention are used in order to relate shallow features to deeper ones. In addition, a global feature extraction module based on Transformer is constructed. Through the powerful ability of Transformer, the global context information of the image can be captured and the global features are fully extracted. The effectiveness of the proposed method in this paper is verified on different experimental datasets, and it is better than most of the current advanced fusion algorithms.https://www.mdpi.com/1424-8220/24/23/7729image fusioninfrared imagevisible imageTransformerCNN |
| spellingShingle | Wenqing Wang Lingzhou Li Yifei Yang Han Liu Runyuan Guo HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion Sensors image fusion infrared image visible image Transformer CNN |
| title | HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion |
| title_full | HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion |
| title_fullStr | HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion |
| title_full_unstemmed | HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion |
| title_short | HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion |
| title_sort | hdctfusion hybrid dual branch network based on cnn and transformer for infrared and visible image fusion |
| topic | image fusion infrared image visible image Transformer CNN |
| url | https://www.mdpi.com/1424-8220/24/23/7729 |
| work_keys_str_mv | AT wenqingwang hdctfusionhybriddualbranchnetworkbasedoncnnandtransformerforinfraredandvisibleimagefusion AT lingzhouli hdctfusionhybriddualbranchnetworkbasedoncnnandtransformerforinfraredandvisibleimagefusion AT yifeiyang hdctfusionhybriddualbranchnetworkbasedoncnnandtransformerforinfraredandvisibleimagefusion AT hanliu hdctfusionhybriddualbranchnetworkbasedoncnnandtransformerforinfraredandvisibleimagefusion AT runyuanguo hdctfusionhybriddualbranchnetworkbasedoncnnandtransformerforinfraredandvisibleimagefusion |