DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images
Recently, the classification of multimodal remote sensing images (RSIs) has garnered significant attention due to its ability to provide rich information for various scenes on Earth. Compared to traditional feature fusion methods used for the classification of multimodal RSIs, neural architecture se...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10904332/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849344639151112192 |
|---|---|
| author | Shiyang Feng Zhaowei Li Bo Zhang Tao Chen Bin Wang |
| author_facet | Shiyang Feng Zhaowei Li Bo Zhang Tao Chen Bin Wang |
| author_sort | Shiyang Feng |
| collection | DOAJ |
| description | Recently, the classification of multimodal remote sensing images (RSIs) has garnered significant attention due to its ability to provide rich information for various scenes on Earth. Compared to traditional feature fusion methods used for the classification of multimodal RSIs, neural architecture search (NAS) is capable of identifying the optimal network structure for multimodal RSIs and downstream tasks. However, due to the diverse spatial resolutions, complex channel dimensions, and drastic foreground scale variations of multimodal RSIs, challenges arise when employing NAS methods for precise classification: 1) Due to the complementary and redundant nature between different modalities in RSIs, determining the features within each modality for fusion becomes quite challenging; 2) the design of fusion operators does not take into account the spatial positions and channel relationships between different modalities of RSIs, making it difficult for the fused features to match downstream tasks. To address these issues, we propose a dual-stage feature fusion framework based on NAS, termed DSF2-NAS, for the classification of multimodal RSIs. It primarily consists of two components: the feature candidate search (FCS) module and the fusion operator search (FOS) module, which execute sequentially. In the FCS module, a feature distance-based regularization approach is proposed to ensure fusion using multimodal features with the highest complementarity. Meanwhile, in the FOS module, a series of fusion operators are designed, which are based on spatial positions, channel relationships, and self-attention mechanisms, aiming to better integrate multimodal features with complex spatial and channel information. The proposed method has been evaluated on various datasets of multimodal RSIs, and experimental results consistently show that this method achieves state-of-the-art performance across multiple classification metrics. |
| format | Article |
| id | doaj-art-628fc53e73d646cca8f6352b19e09e45 |
| institution | Kabale University |
| issn | 1939-1404 2151-1535 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj-art-628fc53e73d646cca8f6352b19e09e452025-08-20T03:42:37ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing1939-14042151-15352025-01-01187207722010.1109/JSTARS.2025.354583110904332DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing ImagesShiyang Feng0https://orcid.org/0009-0001-6675-2948Zhaowei Li1Bo Zhang2https://orcid.org/0000-0001-8052-782XTao Chen3https://orcid.org/0000-0002-0779-9818Bin Wang4https://orcid.org/0000-0003-4748-6426Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai, ChinaKey Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai, ChinaMultimodal Foundation Data Group, Shanghai Artificial Intelligence Laboratory, Shanghai, ChinaKey Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai, ChinaKey Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai, ChinaRecently, the classification of multimodal remote sensing images (RSIs) has garnered significant attention due to its ability to provide rich information for various scenes on Earth. Compared to traditional feature fusion methods used for the classification of multimodal RSIs, neural architecture search (NAS) is capable of identifying the optimal network structure for multimodal RSIs and downstream tasks. However, due to the diverse spatial resolutions, complex channel dimensions, and drastic foreground scale variations of multimodal RSIs, challenges arise when employing NAS methods for precise classification: 1) Due to the complementary and redundant nature between different modalities in RSIs, determining the features within each modality for fusion becomes quite challenging; 2) the design of fusion operators does not take into account the spatial positions and channel relationships between different modalities of RSIs, making it difficult for the fused features to match downstream tasks. To address these issues, we propose a dual-stage feature fusion framework based on NAS, termed DSF2-NAS, for the classification of multimodal RSIs. It primarily consists of two components: the feature candidate search (FCS) module and the fusion operator search (FOS) module, which execute sequentially. In the FCS module, a feature distance-based regularization approach is proposed to ensure fusion using multimodal features with the highest complementarity. Meanwhile, in the FOS module, a series of fusion operators are designed, which are based on spatial positions, channel relationships, and self-attention mechanisms, aiming to better integrate multimodal features with complex spatial and channel information. The proposed method has been evaluated on various datasets of multimodal RSIs, and experimental results consistently show that this method achieves state-of-the-art performance across multiple classification metrics.https://ieeexplore.ieee.org/document/10904332/Classificationmultimodalneural architecture search (NAS)remote sensing images (RSIs) |
| spellingShingle | Shiyang Feng Zhaowei Li Bo Zhang Tao Chen Bin Wang DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Classification multimodal neural architecture search (NAS) remote sensing images (RSIs) |
| title | DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images |
| title_full | DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images |
| title_fullStr | DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images |
| title_full_unstemmed | DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images |
| title_short | DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images |
| title_sort | dsf2 nas dual stage feature fusion via network architecture search for classification of multimodal remote sensing images |
| topic | Classification multimodal neural architecture search (NAS) remote sensing images (RSIs) |
| url | https://ieeexplore.ieee.org/document/10904332/ |
| work_keys_str_mv | AT shiyangfeng dsf2nasdualstagefeaturefusionvianetworkarchitecturesearchforclassificationofmultimodalremotesensingimages AT zhaoweili dsf2nasdualstagefeaturefusionvianetworkarchitecturesearchforclassificationofmultimodalremotesensingimages AT bozhang dsf2nasdualstagefeaturefusionvianetworkarchitecturesearchforclassificationofmultimodalremotesensingimages AT taochen dsf2nasdualstagefeaturefusionvianetworkarchitecturesearchforclassificationofmultimodalremotesensingimages AT binwang dsf2nasdualstagefeaturefusionvianetworkarchitecturesearchforclassificationofmultimodalremotesensingimages |