A Novel Dual-Branch Global and Local Feature Extraction Network for SAR and Optical Image Registration
The registration of synthetic aperture radar (SAR) and optical images is significant in obtaining their complementary information, which is a key prerequisite for image fusion. However, the inherent differences between the two modalities pose a challenge to the existing deep-learning algorithms that...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10614787/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849340558286258176 |
|---|---|
| author | Xuanran Zhao Yan Wu Xin Hu Zhikang Li Ming Li |
| author_facet | Xuanran Zhao Yan Wu Xin Hu Zhikang Li Ming Li |
| author_sort | Xuanran Zhao |
| collection | DOAJ |
| description | The registration of synthetic aperture radar (SAR) and optical images is significant in obtaining their complementary information, which is a key prerequisite for image fusion. However, the inherent differences between the two modalities pose a challenge to the existing deep-learning algorithms that only depend on local features. To address this problem, we propose a global and local feature extraction network (GLFE-Net) for SAR and optical image registration. Beyond merely extracting local features to generate feature descriptors, more importantly, the network also extracts the global feature to better mine the common structural features between SAR and optical images. First, considering the ability of the transformer for long-range dependence and the advantages of convolutional neural network for local feature extraction, GLFE-Net designs two branches: one uses an attention-discarded global feature extraction transformer (ADG Transformer) to extract global features from the entire SAR and optical images, and another uses a partial convolution unit with parameter-free attention mechanism (PCU-AM) to extract local features from the patches around the keypoints. Then, the output features of the two branches are fused to obtain the final feature descriptor with more comprehensive features. Second, in order to establish accurate keypoints correspondence in the matching phase, hard-mean loss function is proposed to optimize GLFE-Net, which jointly utilizes hard negative samples and remaining negative samples to guide descriptors in learning more discriminant features to better distinguish positive and negative samples in SAR and optical images. Finally, the experimental results on the publicly available OS dataset and WHU-SEN-City demonstrate that our proposed GLFE-Net is superior to existing state-of-the-art methods with average root-mean-square error (aRMSE) reduced by 0.14. |
| format | Article |
| id | doaj-art-c0584994e3f44c009699aea07bd7d960 |
| institution | Kabale University |
| issn | 1939-1404 2151-1535 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj-art-c0584994e3f44c009699aea07bd7d9602025-08-20T03:43:52ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing1939-14042151-15352024-01-0117176371765010.1109/JSTARS.2024.343568410614787A Novel Dual-Branch Global and Local Feature Extraction Network for SAR and Optical Image RegistrationXuanran Zhao0https://orcid.org/0009-0000-1773-4754Yan Wu1https://orcid.org/0000-0001-7502-2341Xin Hu2https://orcid.org/0000-0003-4012-684XZhikang Li3Ming Li4https://orcid.org/0000-0002-4706-5173School of Electronic Engineering, Remote Sensing Image Processing and Fusion Group, Xidian University, Xi'an, ChinaSchool of Electronic Engineering, Remote Sensing Image Processing and Fusion Group, Xidian University, Xi'an, ChinaSchool of Electronic Engineering, Remote Sensing Image Processing and Fusion Group, Xidian University, Xi'an, ChinaSchool of Electronic Engineering, Remote Sensing Image Processing and Fusion Group, Xidian University, Xi'an, ChinaNational Key Lab of Radar Signal Processing, Xidian University, Xi'an, ChinaThe registration of synthetic aperture radar (SAR) and optical images is significant in obtaining their complementary information, which is a key prerequisite for image fusion. However, the inherent differences between the two modalities pose a challenge to the existing deep-learning algorithms that only depend on local features. To address this problem, we propose a global and local feature extraction network (GLFE-Net) for SAR and optical image registration. Beyond merely extracting local features to generate feature descriptors, more importantly, the network also extracts the global feature to better mine the common structural features between SAR and optical images. First, considering the ability of the transformer for long-range dependence and the advantages of convolutional neural network for local feature extraction, GLFE-Net designs two branches: one uses an attention-discarded global feature extraction transformer (ADG Transformer) to extract global features from the entire SAR and optical images, and another uses a partial convolution unit with parameter-free attention mechanism (PCU-AM) to extract local features from the patches around the keypoints. Then, the output features of the two branches are fused to obtain the final feature descriptor with more comprehensive features. Second, in order to establish accurate keypoints correspondence in the matching phase, hard-mean loss function is proposed to optimize GLFE-Net, which jointly utilizes hard negative samples and remaining negative samples to guide descriptors in learning more discriminant features to better distinguish positive and negative samples in SAR and optical images. Finally, the experimental results on the publicly available OS dataset and WHU-SEN-City demonstrate that our proposed GLFE-Net is superior to existing state-of-the-art methods with average root-mean-square error (aRMSE) reduced by 0.14.https://ieeexplore.ieee.org/document/10614787/Global and local feature extraction network (GLFE-Net)hard-mean loss functionregistrationSAR and optical images |
| spellingShingle | Xuanran Zhao Yan Wu Xin Hu Zhikang Li Ming Li A Novel Dual-Branch Global and Local Feature Extraction Network for SAR and Optical Image Registration IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Global and local feature extraction network (GLFE-Net) hard-mean loss function registration SAR and optical images |
| title | A Novel Dual-Branch Global and Local Feature Extraction Network for SAR and Optical Image Registration |
| title_full | A Novel Dual-Branch Global and Local Feature Extraction Network for SAR and Optical Image Registration |
| title_fullStr | A Novel Dual-Branch Global and Local Feature Extraction Network for SAR and Optical Image Registration |
| title_full_unstemmed | A Novel Dual-Branch Global and Local Feature Extraction Network for SAR and Optical Image Registration |
| title_short | A Novel Dual-Branch Global and Local Feature Extraction Network for SAR and Optical Image Registration |
| title_sort | novel dual branch global and local feature extraction network for sar and optical image registration |
| topic | Global and local feature extraction network (GLFE-Net) hard-mean loss function registration SAR and optical images |
| url | https://ieeexplore.ieee.org/document/10614787/ |
| work_keys_str_mv | AT xuanranzhao anoveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration AT yanwu anoveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration AT xinhu anoveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration AT zhikangli anoveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration AT mingli anoveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration AT xuanranzhao noveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration AT yanwu noveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration AT xinhu noveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration AT zhikangli noveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration AT mingli noveldualbranchglobalandlocalfeatureextractionnetworkforsarandopticalimageregistration |