Fusion-Based Damage Segmentation for Multimodal Building Façade Images from an End-to-End Perspective
Multimodal image data have found widespread applications in visual-based building façade damage detection in recent years, offering comprehensive inspection of façade surfaces with the assistance of drones and infrared thermography. However, the comprehensive integration of such complementary data h...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Buildings |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2075-5309/15/1/63 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841549385417097216 |
|---|---|
| author | Pujin Wang Jiehui Wang Qiong Liu Lin Fang Jie Xiao |
| author_facet | Pujin Wang Jiehui Wang Qiong Liu Lin Fang Jie Xiao |
| author_sort | Pujin Wang |
| collection | DOAJ |
| description | Multimodal image data have found widespread applications in visual-based building façade damage detection in recent years, offering comprehensive inspection of façade surfaces with the assistance of drones and infrared thermography. However, the comprehensive integration of such complementary data has been hindered by low levels of automation due to the absence of properly developed methods, resulting in high cost and low efficiency. Thus, this paper proposes an automatic end-to-end building façade damage detection method by integrating multimodal image registration, infrared–visible image fusion (IVIF), and damage segmentation. An infrared and visible image dataset consisting of 1761 pairs encompassing 4 main types of façade damage has been constructed for processing and training. A novel infrared–visible image registration method using main orientation assignment for feature point extraction is developed, reaching a high RMSE of 14.35 to align the multimodal images. Then, a deep learning-based infrared–visible image fusion (IVIF) network is trained to preserve damage characteristics between the modalities. For damage detection, a relatively high mean average precision (mAP) result of 85.4% is achieved by comparing four instance segmentation models, affirming the effective utilization of IVIF results. |
| format | Article |
| id | doaj-art-dea1da22213c43bbaafc48e21c3af341 |
| institution | Kabale University |
| issn | 2075-5309 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Buildings |
| spelling | doaj-art-dea1da22213c43bbaafc48e21c3af3412025-01-10T13:15:55ZengMDPI AGBuildings2075-53092024-12-011516310.3390/buildings15010063Fusion-Based Damage Segmentation for Multimodal Building Façade Images from an End-to-End PerspectivePujin Wang0Jiehui Wang1Qiong Liu2Lin Fang3Jie Xiao4College of Civil Engineering, Tongji University, Shanghai 200092, ChinaCollege of Civil and Transportation Engineering, Shenzhen University, Shenzhen 518060, ChinaSchool of Environment and Architecture, University of Shanghai for Science and Technology, Shanghai 200093, ChinaShanghai Housing Quality Inspection Station Co., Ltd., Shanghai 200061, ChinaSchool of Civil and Transportation Engineering, Guangdong University of Technology, Guangzhou 510006, ChinaMultimodal image data have found widespread applications in visual-based building façade damage detection in recent years, offering comprehensive inspection of façade surfaces with the assistance of drones and infrared thermography. However, the comprehensive integration of such complementary data has been hindered by low levels of automation due to the absence of properly developed methods, resulting in high cost and low efficiency. Thus, this paper proposes an automatic end-to-end building façade damage detection method by integrating multimodal image registration, infrared–visible image fusion (IVIF), and damage segmentation. An infrared and visible image dataset consisting of 1761 pairs encompassing 4 main types of façade damage has been constructed for processing and training. A novel infrared–visible image registration method using main orientation assignment for feature point extraction is developed, reaching a high RMSE of 14.35 to align the multimodal images. Then, a deep learning-based infrared–visible image fusion (IVIF) network is trained to preserve damage characteristics between the modalities. For damage detection, a relatively high mean average precision (mAP) result of 85.4% is achieved by comparing four instance segmentation models, affirming the effective utilization of IVIF results.https://www.mdpi.com/2075-5309/15/1/63building façadedamage detectionmultimodal image registrationmultimodal image fusioninstance segmentation |
| spellingShingle | Pujin Wang Jiehui Wang Qiong Liu Lin Fang Jie Xiao Fusion-Based Damage Segmentation for Multimodal Building Façade Images from an End-to-End Perspective Buildings building façade damage detection multimodal image registration multimodal image fusion instance segmentation |
| title | Fusion-Based Damage Segmentation for Multimodal Building Façade Images from an End-to-End Perspective |
| title_full | Fusion-Based Damage Segmentation for Multimodal Building Façade Images from an End-to-End Perspective |
| title_fullStr | Fusion-Based Damage Segmentation for Multimodal Building Façade Images from an End-to-End Perspective |
| title_full_unstemmed | Fusion-Based Damage Segmentation for Multimodal Building Façade Images from an End-to-End Perspective |
| title_short | Fusion-Based Damage Segmentation for Multimodal Building Façade Images from an End-to-End Perspective |
| title_sort | fusion based damage segmentation for multimodal building facade images from an end to end perspective |
| topic | building façade damage detection multimodal image registration multimodal image fusion instance segmentation |
| url | https://www.mdpi.com/2075-5309/15/1/63 |
| work_keys_str_mv | AT pujinwang fusionbaseddamagesegmentationformultimodalbuildingfacadeimagesfromanendtoendperspective AT jiehuiwang fusionbaseddamagesegmentationformultimodalbuildingfacadeimagesfromanendtoendperspective AT qiongliu fusionbaseddamagesegmentationformultimodalbuildingfacadeimagesfromanendtoendperspective AT linfang fusionbaseddamagesegmentationformultimodalbuildingfacadeimagesfromanendtoendperspective AT jiexiao fusionbaseddamagesegmentationformultimodalbuildingfacadeimagesfromanendtoendperspective |