Human-Annotated Label Noise and Their Impact on ConvNets for Remote Sensing Image Scene Classification
Human-labeled training datasets are essential for convolutional neural networks (ConvNets) in satellite image scene classification. Annotation errors are unavoidable due to the complexity of satellite images. However, the distribution of real-world human-annotated label noises on satellite images an...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10757363/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850106317635584000 |
|---|---|
| author | Longkang Peng Tao Wei Xuehong Chen Xiaobei Chen Rui Sun Luoma Wan Jin Chen Xiaolin Zhu |
| author_facet | Longkang Peng Tao Wei Xuehong Chen Xiaobei Chen Rui Sun Luoma Wan Jin Chen Xiaolin Zhu |
| author_sort | Longkang Peng |
| collection | DOAJ |
| description | Human-labeled training datasets are essential for convolutional neural networks (ConvNets) in satellite image scene classification. Annotation errors are unavoidable due to the complexity of satellite images. However, the distribution of real-world human-annotated label noises on satellite images and their impact on ConvNets have not been investigated. To fill this research gap, this article, for the first time, collected real-world labels from 32 participants and explored how their annotated label noise affects three representative ConvNets (VGG16, GoogleNet, and ResNet-50) for remote sensing image scene classification. We found that 1) human-annotated label noise exhibits significant class and instance dependence; 2) an additional 1% of human-annotated label noise in training data leads to a 0.5% reduction in the overall accuracy of ConvNets classification; and 3) the error pattern of ConvNet predictions was strongly correlated with that of participant's labels. To uncover the mechanism underlying the impact of human labeling errors on ConvNets, we compared it with three types of simulated label noise: uniform noise, class-dependent noise, and instance-dependent noise. Our results show that the impact of human-annotated label noise on ConvNets significantly differs from all three types of simulated label noise, while both class dependence and instance dependence contribute to the impact of human-annotated label noise on ConvNets. Additionally, the label noise estimation algorithm (confident learning) cannot fully identify label noise. These observations necessitate a reevaluation of the handling of noisy labels, and we anticipate that our real-world label noise dataset would facilitate the future development and assessment of label-noise learning algorithms. |
| format | Article |
| id | doaj-art-45cae411c52b4aeda5b1ff5cada15de8 |
| institution | OA Journals |
| issn | 1939-1404 2151-1535 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing |
| spelling | doaj-art-45cae411c52b4aeda5b1ff5cada15de82025-08-20T02:38:51ZengIEEEIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing1939-14042151-15352025-01-01181500151410.1109/JSTARS.2024.350246110757363Human-Annotated Label Noise and Their Impact on ConvNets for Remote Sensing Image Scene ClassificationLongkang Peng0https://orcid.org/0009-0003-0015-0199Tao Wei1https://orcid.org/0000-0001-8094-4906Xuehong Chen2https://orcid.org/0000-0001-7223-8649Xiaobei Chen3Rui Sun4Luoma Wan5https://orcid.org/0000-0002-8202-510XJin Chen6https://orcid.org/0000-0002-6497-4141Xiaolin Zhu7https://orcid.org/0000-0001-6967-786XSchool of Psychology, Shenzhen University, Shenzhen, ChinaSchool of Psychology, Shenzhen University, Shenzhen, ChinaState Key Laboratory of Remote Sensing Science, Faculty of Geographical Science, Beijing Normal University, Beijing, ChinaSchool of Psychology, Shenzhen University, Shenzhen, ChinaSchool of Psychology, Shenzhen University, Shenzhen, ChinaDepartment of Land Surveying and Geo-Informatics and the Otto Poon Charitable Foundation Smart Cities Research Institute, The Hong Kong Polytechnic University, Hong KongState Key Laboratory of Remote Sensing Science, Faculty of Geographical Science, Beijing Normal University, Beijing, ChinaDepartment of Land Surveying and Geo-Informatics and the Otto Poon Charitable Foundation Smart Cities Research Institute, The Hong Kong Polytechnic University, Hong KongHuman-labeled training datasets are essential for convolutional neural networks (ConvNets) in satellite image scene classification. Annotation errors are unavoidable due to the complexity of satellite images. However, the distribution of real-world human-annotated label noises on satellite images and their impact on ConvNets have not been investigated. To fill this research gap, this article, for the first time, collected real-world labels from 32 participants and explored how their annotated label noise affects three representative ConvNets (VGG16, GoogleNet, and ResNet-50) for remote sensing image scene classification. We found that 1) human-annotated label noise exhibits significant class and instance dependence; 2) an additional 1% of human-annotated label noise in training data leads to a 0.5% reduction in the overall accuracy of ConvNets classification; and 3) the error pattern of ConvNet predictions was strongly correlated with that of participant's labels. To uncover the mechanism underlying the impact of human labeling errors on ConvNets, we compared it with three types of simulated label noise: uniform noise, class-dependent noise, and instance-dependent noise. Our results show that the impact of human-annotated label noise on ConvNets significantly differs from all three types of simulated label noise, while both class dependence and instance dependence contribute to the impact of human-annotated label noise on ConvNets. Additionally, the label noise estimation algorithm (confident learning) cannot fully identify label noise. These observations necessitate a reevaluation of the handling of noisy labels, and we anticipate that our real-world label noise dataset would facilitate the future development and assessment of label-noise learning algorithms.https://ieeexplore.ieee.org/document/10757363/Convolutional neural network (ConvNet)human-annotated label noiselabel noiseremote sensingscene classification |
| spellingShingle | Longkang Peng Tao Wei Xuehong Chen Xiaobei Chen Rui Sun Luoma Wan Jin Chen Xiaolin Zhu Human-Annotated Label Noise and Their Impact on ConvNets for Remote Sensing Image Scene Classification IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing Convolutional neural network (ConvNet) human-annotated label noise label noise remote sensing scene classification |
| title | Human-Annotated Label Noise and Their Impact on ConvNets for Remote Sensing Image Scene Classification |
| title_full | Human-Annotated Label Noise and Their Impact on ConvNets for Remote Sensing Image Scene Classification |
| title_fullStr | Human-Annotated Label Noise and Their Impact on ConvNets for Remote Sensing Image Scene Classification |
| title_full_unstemmed | Human-Annotated Label Noise and Their Impact on ConvNets for Remote Sensing Image Scene Classification |
| title_short | Human-Annotated Label Noise and Their Impact on ConvNets for Remote Sensing Image Scene Classification |
| title_sort | human annotated label noise and their impact on convnets for remote sensing image scene classification |
| topic | Convolutional neural network (ConvNet) human-annotated label noise label noise remote sensing scene classification |
| url | https://ieeexplore.ieee.org/document/10757363/ |
| work_keys_str_mv | AT longkangpeng humanannotatedlabelnoiseandtheirimpactonconvnetsforremotesensingimagesceneclassification AT taowei humanannotatedlabelnoiseandtheirimpactonconvnetsforremotesensingimagesceneclassification AT xuehongchen humanannotatedlabelnoiseandtheirimpactonconvnetsforremotesensingimagesceneclassification AT xiaobeichen humanannotatedlabelnoiseandtheirimpactonconvnetsforremotesensingimagesceneclassification AT ruisun humanannotatedlabelnoiseandtheirimpactonconvnetsforremotesensingimagesceneclassification AT luomawan humanannotatedlabelnoiseandtheirimpactonconvnetsforremotesensingimagesceneclassification AT jinchen humanannotatedlabelnoiseandtheirimpactonconvnetsforremotesensingimagesceneclassification AT xiaolinzhu humanannotatedlabelnoiseandtheirimpactonconvnetsforremotesensingimagesceneclassification |