Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer
Hyperspectral pansharpening is to fuse a high spatial resolution panchromatic image (PAN) with a low spatial resolution hyperspectral image (LR-HSI) and generate high resolution hyperspectral image (HR-HSI). However, most existing deep learning-based pansharpening methods have some issues, such as s...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10545580/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846169467256569856 |
|---|---|
| author | Hengyou Wang Jie Zhang Lian-Zhi Huo |
| author_facet | Hengyou Wang Jie Zhang Lian-Zhi Huo |
| author_sort | Hengyou Wang |
| collection | DOAJ |
| description | Hyperspectral pansharpening is to fuse a high spatial resolution panchromatic image (PAN) with a low spatial resolution hyperspectral image (LR-HSI) and generate high resolution hyperspectral image (HR-HSI). However, most existing deep learning-based pansharpening methods have some issues, such as spectral distortion and insufficient spatial texture enhancement. In this work, we propose a novel multiscale pansharpening network based on the Dual Gaussian-Laplacian Pyramid (DGLP) and Transformer, named MDTP-Net. Specifically, the DGLP module is designed to obtain feature maps at multilevel scales, which effectively learn global spectral information and spatial detail texture information. Then, we design a corresponding Transformer module for each scale feature and utilize the multihead attention mechanism to guide the extraction of spatial information from LR-HSI and PAN images. This enhances the stability of pansharpening and improves the fusion of spectral with spatial information across feature spaces. In addition, the feature extractors are inserted to connect DGLP and Transformer, making the spatial feature map smoother and richer in channel and texture features. The feature fusion and multiscale feature connection blocks are used to connect multiscale information together to generate HR-HSI images with more comprehensive spatial and spectral features. Finally, extensive experiments on three classic hyperspectral datasets are conducted. The experimental results demonstrate that our proposed MDTP-Net outperforms conventional methods and existing deep learning-based methods. |
| format | Article |
| id | doaj-art-28e0d874cfb24235b4850b84aeb7809d |
| institution | Kabale University |
| issn | 1939-1404 2151-1535 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj-art-28e0d874cfb24235b4850b84aeb7809d2024-11-13T00:00:15ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing1939-14042151-15352024-01-0117197861979710.1109/JSTARS.2024.340828010545580Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and TransformerHengyou Wang0https://orcid.org/0000-0001-6693-0161Jie Zhang1https://orcid.org/0009-0006-7029-6830Lian-Zhi Huo2https://orcid.org/0000-0001-6705-6453School of Science, Beijing University of Civil Engineering and Architecture, Beijing, ChinaSchool of Science, Beijing University of Civil Engineering and Architecture, Beijing, ChinaAerospace Information Research Institute, Chinese Academy of Sciences, Beijing, ChinaHyperspectral pansharpening is to fuse a high spatial resolution panchromatic image (PAN) with a low spatial resolution hyperspectral image (LR-HSI) and generate high resolution hyperspectral image (HR-HSI). However, most existing deep learning-based pansharpening methods have some issues, such as spectral distortion and insufficient spatial texture enhancement. In this work, we propose a novel multiscale pansharpening network based on the Dual Gaussian-Laplacian Pyramid (DGLP) and Transformer, named MDTP-Net. Specifically, the DGLP module is designed to obtain feature maps at multilevel scales, which effectively learn global spectral information and spatial detail texture information. Then, we design a corresponding Transformer module for each scale feature and utilize the multihead attention mechanism to guide the extraction of spatial information from LR-HSI and PAN images. This enhances the stability of pansharpening and improves the fusion of spectral with spatial information across feature spaces. In addition, the feature extractors are inserted to connect DGLP and Transformer, making the spatial feature map smoother and richer in channel and texture features. The feature fusion and multiscale feature connection blocks are used to connect multiscale information together to generate HR-HSI images with more comprehensive spatial and spectral features. Finally, extensive experiments on three classic hyperspectral datasets are conducted. The experimental results demonstrate that our proposed MDTP-Net outperforms conventional methods and existing deep learning-based methods.https://ieeexplore.ieee.org/document/10545580/Dual Gaussian-Laplacian pyramid (DGLP)hyperspectral pansharpeningmultiscale fusionTransformer |
| spellingShingle | Hengyou Wang Jie Zhang Lian-Zhi Huo Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Dual Gaussian-Laplacian pyramid (DGLP) hyperspectral pansharpening multiscale fusion Transformer |
| title | Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer |
| title_full | Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer |
| title_fullStr | Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer |
| title_full_unstemmed | Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer |
| title_short | Multiscale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer |
| title_sort | multiscale hyperspectral pansharpening network based on dual pyramid and transformer |
| topic | Dual Gaussian-Laplacian pyramid (DGLP) hyperspectral pansharpening multiscale fusion Transformer |
| url | https://ieeexplore.ieee.org/document/10545580/ |
| work_keys_str_mv | AT hengyouwang multiscalehyperspectralpansharpeningnetworkbasedondualpyramidandtransformer AT jiezhang multiscalehyperspectralpansharpeningnetworkbasedondualpyramidandtransformer AT lianzhihuo multiscalehyperspectralpansharpeningnetworkbasedondualpyramidandtransformer |