PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors
Deep learning-based image signal processor (ISP) models for mobile cameras can generate high-quality images that rival those of professional DSLR cameras. However, their computational demands often make them unsuitable for mobile settings. Additionally, modern mobile cameras employ non-Bayer color f...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10114931/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850066382868185088 |
|---|---|
| author | Minhyeok Cho Haechang Lee Hyunwoo Je Kijeong Kim Dongil Ryu Albert No |
| author_facet | Minhyeok Cho Haechang Lee Hyunwoo Je Kijeong Kim Dongil Ryu Albert No |
| author_sort | Minhyeok Cho |
| collection | DOAJ |
| description | Deep learning-based image signal processor (ISP) models for mobile cameras can generate high-quality images that rival those of professional DSLR cameras. However, their computational demands often make them unsuitable for mobile settings. Additionally, modern mobile cameras employ non-Bayer color filter arrays (CFA) such as Quad Bayer, Nona Bayer, and <inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> Bayer to enhance image quality, yet most existing deep learning-based ISP (or demosaicing) models focus primarily on standard Bayer CFAs. In this study, we present PyNET-<inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula>, a lightweight demosaicing model specifically designed for <inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> Bayer CFA patterns, which is derived from the original PyNET. We also propose a knowledge distillation method called progressive distillation to train the reduced network more effectively. Consequently, PyNET-<inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> contains less than 2.5% of the parameters of the original PyNET while preserving its performance. Experiments using <inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> images captured by a prototype <inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> camera sensor show that PyNET-<inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> outperforms existing conventional algorithms in terms of texture and edge reconstruction, despite its significantly reduced parameter count. Code and partial datasets can be found at <uri>https://github.com/Minhyeok01/PyNET-QxQ</uri>. |
| format | Article |
| id | doaj-art-9e2fe9950c4d475693b9042f412a593c |
| institution | DOAJ |
| issn | 2169-3536 |
| language | English |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-9e2fe9950c4d475693b9042f412a593c2025-08-20T02:48:46ZengIEEEIEEE Access2169-35362023-01-0111448954491010.1109/ACCESS.2023.327266510114931PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image SensorsMinhyeok Cho0https://orcid.org/0000-0002-2230-4473Haechang Lee1Hyunwoo Je2https://orcid.org/0000-0001-8014-0393Kijeong Kim3Dongil Ryu4Albert No5https://orcid.org/0000-0002-6346-4182Department of Electronic and Electrical Engineering, Hongik University, Seoul, South KoreaSK hynix, Icheon, South KoreaSK hynix, Icheon, South KoreaSK hynix, Icheon, South KoreaSK hynix, Icheon, South KoreaDepartment of Electronic and Electrical Engineering, Hongik University, Seoul, South KoreaDeep learning-based image signal processor (ISP) models for mobile cameras can generate high-quality images that rival those of professional DSLR cameras. However, their computational demands often make them unsuitable for mobile settings. Additionally, modern mobile cameras employ non-Bayer color filter arrays (CFA) such as Quad Bayer, Nona Bayer, and <inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> Bayer to enhance image quality, yet most existing deep learning-based ISP (or demosaicing) models focus primarily on standard Bayer CFAs. In this study, we present PyNET-<inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula>, a lightweight demosaicing model specifically designed for <inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> Bayer CFA patterns, which is derived from the original PyNET. We also propose a knowledge distillation method called progressive distillation to train the reduced network more effectively. Consequently, PyNET-<inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> contains less than 2.5% of the parameters of the original PyNET while preserving its performance. Experiments using <inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> images captured by a prototype <inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> camera sensor show that PyNET-<inline-formula> <tex-math notation="LaTeX">$\text{Q}\times \text{Q}$ </tex-math></inline-formula> outperforms existing conventional algorithms in terms of texture and edge reconstruction, despite its significantly reduced parameter count. Code and partial datasets can be found at <uri>https://github.com/Minhyeok01/PyNET-QxQ</uri>.https://ieeexplore.ieee.org/document/10114931/Bayer filtercolor filter array (CFA)demosaicingimage signal processor (ISP)knowledge distillationnon-Bayer CFA |
| spellingShingle | Minhyeok Cho Haechang Lee Hyunwoo Je Kijeong Kim Dongil Ryu Albert No PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors IEEE Access Bayer filter color filter array (CFA) demosaicing image signal processor (ISP) knowledge distillation non-Bayer CFA |
| title | PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors |
| title_full | PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors |
| title_fullStr | PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors |
| title_full_unstemmed | PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors |
| title_short | PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors |
| title_sort | pynet q x00d7 q an efficient pynet variant for q x00d7 q bayer pattern demosaicing in cmos image sensors |
| topic | Bayer filter color filter array (CFA) demosaicing image signal processor (ISP) knowledge distillation non-Bayer CFA |
| url | https://ieeexplore.ieee.org/document/10114931/ |
| work_keys_str_mv | AT minhyeokcho pynetqx00d7qanefficientpynetvariantforqx00d7qbayerpatterndemosaicingincmosimagesensors AT haechanglee pynetqx00d7qanefficientpynetvariantforqx00d7qbayerpatterndemosaicingincmosimagesensors AT hyunwooje pynetqx00d7qanefficientpynetvariantforqx00d7qbayerpatterndemosaicingincmosimagesensors AT kijeongkim pynetqx00d7qanefficientpynetvariantforqx00d7qbayerpatterndemosaicingincmosimagesensors AT dongilryu pynetqx00d7qanefficientpynetvariantforqx00d7qbayerpatterndemosaicingincmosimagesensors AT albertno pynetqx00d7qanefficientpynetvariantforqx00d7qbayerpatterndemosaicingincmosimagesensors |