KANDiff: Kolmogorov–Arnold Network and Diffusion Model-Based Network for Hyperspectral and Multispectral Image Fusion
In recent years, the fusion of hyperspectral and multispectral images in remote sensing image processing still faces challenges, primarily due to their complexity and multimodal characteristics. Diffusion models, known for their stable training process and exceptional image generation capabilities,...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-01-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/17/1/145 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841548998831243264 |
|---|---|
| author | Wei Li Lu Li Man Peng Ran Tao |
| author_facet | Wei Li Lu Li Man Peng Ran Tao |
| author_sort | Wei Li |
| collection | DOAJ |
| description | In recent years, the fusion of hyperspectral and multispectral images in remote sensing image processing still faces challenges, primarily due to their complexity and multimodal characteristics. Diffusion models, known for their stable training process and exceptional image generation capabilities, have shown good application potential in this field. However, when dealing with multimodal data, it may prove challenging for the models to fully capture the intricate relationships between the modalities, which may result in incomplete information integration and a small amount of remaining noise in the generated images. To address these problems, we propose a new model, KanDiff, for hyperspectral and multispectral image fusion. To address the differences in modal information between multispectral and hyperspectral images, KANDiff incorporates Kolmogorov–Arnold Networks (KAN) to guide the inputs. It helps the model understand the complex relationships between the modalities by replacing the fixed activation function in the traditional MLP with a learnable activation function. Furthermore, the image generated by the diffusion model may exhibit a small amount of the remaining noise. Convolutional Neural Networks (CNNs) effectively extract local features through their convolutional layers and achieve noise suppression via layer-by-layer feature representation. Therefore, the MergeCNN module is further introduced to enhance the fusion effect, resulting in smoother and more accurate outcomes. Experimental results on the public CAVE and Harvard datasets indicate that KanDiff has achieved improvements over current high-performance methods across several metrics, particularly showing significant enhancements in the peak signal-to-noise ratio (PSNR) and the structural similarity index measure (SSIM), thus demonstrating superior performance. Additionally, we have created an image fusion dataset of the lunar surface, and KANDiff exhibits robust performance on this dataset as well. This work introduces an effective solution for addressing the challenges posed by missing high-resolution hyperspectral images (HRHS) data, which is essential for tasks including landing site selection and resource exploration within the realm of deep space exploration. |
| format | Article |
| id | doaj-art-740d4185015043ce83c022c36f67e90e |
| institution | Kabale University |
| issn | 2072-4292 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj-art-740d4185015043ce83c022c36f67e90e2025-01-10T13:20:23ZengMDPI AGRemote Sensing2072-42922025-01-0117114510.3390/rs17010145KANDiff: Kolmogorov–Arnold Network and Diffusion Model-Based Network for Hyperspectral and Multispectral Image FusionWei Li0Lu Li1Man Peng2Ran Tao3School of Automation, Beijing Information Science and Technology University, Beijing 100192, ChinaSchool of Automation, Beijing Information Science and Technology University, Beijing 100192, ChinaAerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100045, ChinaSchool of Information and Electronics, Beijing Institute of Technology, Beijing 100081, ChinaIn recent years, the fusion of hyperspectral and multispectral images in remote sensing image processing still faces challenges, primarily due to their complexity and multimodal characteristics. Diffusion models, known for their stable training process and exceptional image generation capabilities, have shown good application potential in this field. However, when dealing with multimodal data, it may prove challenging for the models to fully capture the intricate relationships between the modalities, which may result in incomplete information integration and a small amount of remaining noise in the generated images. To address these problems, we propose a new model, KanDiff, for hyperspectral and multispectral image fusion. To address the differences in modal information between multispectral and hyperspectral images, KANDiff incorporates Kolmogorov–Arnold Networks (KAN) to guide the inputs. It helps the model understand the complex relationships between the modalities by replacing the fixed activation function in the traditional MLP with a learnable activation function. Furthermore, the image generated by the diffusion model may exhibit a small amount of the remaining noise. Convolutional Neural Networks (CNNs) effectively extract local features through their convolutional layers and achieve noise suppression via layer-by-layer feature representation. Therefore, the MergeCNN module is further introduced to enhance the fusion effect, resulting in smoother and more accurate outcomes. Experimental results on the public CAVE and Harvard datasets indicate that KanDiff has achieved improvements over current high-performance methods across several metrics, particularly showing significant enhancements in the peak signal-to-noise ratio (PSNR) and the structural similarity index measure (SSIM), thus demonstrating superior performance. Additionally, we have created an image fusion dataset of the lunar surface, and KANDiff exhibits robust performance on this dataset as well. This work introduces an effective solution for addressing the challenges posed by missing high-resolution hyperspectral images (HRHS) data, which is essential for tasks including landing site selection and resource exploration within the realm of deep space exploration.https://www.mdpi.com/2072-4292/17/1/145image fusionhyperspectralmultispectralremote sensing imagesdiffusion models |
| spellingShingle | Wei Li Lu Li Man Peng Ran Tao KANDiff: Kolmogorov–Arnold Network and Diffusion Model-Based Network for Hyperspectral and Multispectral Image Fusion Remote Sensing image fusion hyperspectral multispectral remote sensing images diffusion models |
| title | KANDiff: Kolmogorov–Arnold Network and Diffusion Model-Based Network for Hyperspectral and Multispectral Image Fusion |
| title_full | KANDiff: Kolmogorov–Arnold Network and Diffusion Model-Based Network for Hyperspectral and Multispectral Image Fusion |
| title_fullStr | KANDiff: Kolmogorov–Arnold Network and Diffusion Model-Based Network for Hyperspectral and Multispectral Image Fusion |
| title_full_unstemmed | KANDiff: Kolmogorov–Arnold Network and Diffusion Model-Based Network for Hyperspectral and Multispectral Image Fusion |
| title_short | KANDiff: Kolmogorov–Arnold Network and Diffusion Model-Based Network for Hyperspectral and Multispectral Image Fusion |
| title_sort | kandiff kolmogorov arnold network and diffusion model based network for hyperspectral and multispectral image fusion |
| topic | image fusion hyperspectral multispectral remote sensing images diffusion models |
| url | https://www.mdpi.com/2072-4292/17/1/145 |
| work_keys_str_mv | AT weili kandiffkolmogorovarnoldnetworkanddiffusionmodelbasednetworkforhyperspectralandmultispectralimagefusion AT luli kandiffkolmogorovarnoldnetworkanddiffusionmodelbasednetworkforhyperspectralandmultispectralimagefusion AT manpeng kandiffkolmogorovarnoldnetworkanddiffusionmodelbasednetworkforhyperspectralandmultispectralimagefusion AT rantao kandiffkolmogorovarnoldnetworkanddiffusionmodelbasednetworkforhyperspectralandmultispectralimagefusion |