High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet
To solve the problem of poor quality in ghost imaging via sparsity constraints (GISC) multispectral image reconstruction with correlation operations and compressed sensing algorithms under low sampling rate detection conditions, we propose an end-to-end deep-learning-based method. Based on the U-Net...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Photonics Journal |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10132552/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849415935246467072 |
|---|---|
| author | Tao Hu Jianxia Chen Shu Wang Jianrong Wu Ziyan Chen Zhifu Tian Ruipeng Ma Di Wu |
| author_facet | Tao Hu Jianxia Chen Shu Wang Jianrong Wu Ziyan Chen Zhifu Tian Ruipeng Ma Di Wu |
| author_sort | Tao Hu |
| collection | DOAJ |
| description | To solve the problem of poor quality in ghost imaging via sparsity constraints (GISC) multispectral image reconstruction with correlation operations and compressed sensing algorithms under low sampling rate detection conditions, we propose an end-to-end deep-learning-based method. Based on the U-Net, Res2Net-SE-Conv is employed instead of convolutional blocks to extract local and global image features at a more fine-grained level while adaptively adjusting the channel feature response. The two-dimensional contextual transformer is constructed to fully use contextual correlation information to enhance the effectiveness of feature representations. We employ the two-dimensional contextual transformer in the decoder part, dubbed CoT-Unet, to reconstruct the desired 3D cube. The results show that compared with U-Net, TSA-Net based on spatial-spectral self-attention, the PSNR of reconstructed images by the CoT-Unet is improved by 5 dB and 3 dB, respectively, SSIM is improved by 0.23 and 0.07, and SAM is decreased by 0.06 and 0.58. Compared with conventional algorithms such as DGI and CS, our method significantly improves the quality of reconstructed images. Furthermore, the comparison results at 10%, 20%, and 30% sampling rates show that our approach has the best quality in reconstructing GISC multispectral images at low sampling rates. |
| format | Article |
| id | doaj-art-6bac2419b3a546b4ac8c6b9d02ccf097 |
| institution | Kabale University |
| issn | 1943-0655 |
| language | English |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Photonics Journal |
| spelling | doaj-art-6bac2419b3a546b4ac8c6b9d02ccf0972025-08-20T03:33:21ZengIEEEIEEE Photonics Journal1943-06552023-01-0115311010.1109/JPHOT.2023.327938610132552High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-UnetTao Hu0https://orcid.org/0000-0003-3902-0378Jianxia Chen1https://orcid.org/0000-0001-6036-0698Shu Wang2Jianrong Wu3https://orcid.org/0000-0003-1556-7758Ziyan Chen4https://orcid.org/0000-0003-1203-7858Zhifu Tian5Ruipeng Ma6Di Wu7School of Institute of Data and Target Engineering, PLA Strategic Support Force Information Engineering University, Zhengzhou, ChinaSchool of Institute of Data and Target Engineering, PLA Strategic Support Force Information Engineering University, Zhengzhou, ChinaSchool of Institute of Data and Target Engineering, PLA Strategic Support Force Information Engineering University, Zhengzhou, ChinaKey Laboratory for Quantum Optics of CAS, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, ChinaKey Laboratory for Quantum Optics of CAS, Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, ChinaSchool of Institute of Data and Target Engineering, PLA Strategic Support Force Information Engineering University, Zhengzhou, ChinaSchool of Cyber Science and Engineering, Zhengzhou University, Zhengzhou, ChinaSchool of Institute of Data and Target Engineering, PLA Strategic Support Force Information Engineering University, Zhengzhou, ChinaTo solve the problem of poor quality in ghost imaging via sparsity constraints (GISC) multispectral image reconstruction with correlation operations and compressed sensing algorithms under low sampling rate detection conditions, we propose an end-to-end deep-learning-based method. Based on the U-Net, Res2Net-SE-Conv is employed instead of convolutional blocks to extract local and global image features at a more fine-grained level while adaptively adjusting the channel feature response. The two-dimensional contextual transformer is constructed to fully use contextual correlation information to enhance the effectiveness of feature representations. We employ the two-dimensional contextual transformer in the decoder part, dubbed CoT-Unet, to reconstruct the desired 3D cube. The results show that compared with U-Net, TSA-Net based on spatial-spectral self-attention, the PSNR of reconstructed images by the CoT-Unet is improved by 5 dB and 3 dB, respectively, SSIM is improved by 0.23 and 0.07, and SAM is decreased by 0.06 and 0.58. Compared with conventional algorithms such as DGI and CS, our method significantly improves the quality of reconstructed images. Furthermore, the comparison results at 10%, 20%, and 30% sampling rates show that our approach has the best quality in reconstructing GISC multispectral images at low sampling rates.https://ieeexplore.ieee.org/document/10132552/Multispectral image reconstructionconvoluti onal neural networktransformerself-attention mechanismghost imaging |
| spellingShingle | Tao Hu Jianxia Chen Shu Wang Jianrong Wu Ziyan Chen Zhifu Tian Ruipeng Ma Di Wu High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet IEEE Photonics Journal Multispectral image reconstruction convoluti onal neural network transformer self-attention mechanism ghost imaging |
| title | High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet |
| title_full | High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet |
| title_fullStr | High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet |
| title_full_unstemmed | High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet |
| title_short | High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet |
| title_sort | high quality multispectral image reconstruction for the spectral camera based on ghost imaging via sparsity constraints using cot unet |
| topic | Multispectral image reconstruction convoluti onal neural network transformer self-attention mechanism ghost imaging |
| url | https://ieeexplore.ieee.org/document/10132552/ |
| work_keys_str_mv | AT taohu highqualitymultispectralimagereconstructionforthespectralcamerabasedonghostimagingviasparsityconstraintsusingcotunet AT jianxiachen highqualitymultispectralimagereconstructionforthespectralcamerabasedonghostimagingviasparsityconstraintsusingcotunet AT shuwang highqualitymultispectralimagereconstructionforthespectralcamerabasedonghostimagingviasparsityconstraintsusingcotunet AT jianrongwu highqualitymultispectralimagereconstructionforthespectralcamerabasedonghostimagingviasparsityconstraintsusingcotunet AT ziyanchen highqualitymultispectralimagereconstructionforthespectralcamerabasedonghostimagingviasparsityconstraintsusingcotunet AT zhifutian highqualitymultispectralimagereconstructionforthespectralcamerabasedonghostimagingviasparsityconstraintsusingcotunet AT ruipengma highqualitymultispectralimagereconstructionforthespectralcamerabasedonghostimagingviasparsityconstraintsusingcotunet AT diwu highqualitymultispectralimagereconstructionforthespectralcamerabasedonghostimagingviasparsityconstraintsusingcotunet |