Detecting and reducing heterogeneity of error in acoustic classification
Abstract Passive acoustic monitoring can be an effective method for monitoring species, allowing the assembly of large audio datasets, removing logistical constraints in data collection and reducing anthropogenic monitoring disturbances. However, the analysis of large acoustic datasets is challengin...
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2022-11-01
|
| Series: | Methods in Ecology and Evolution |
| Subjects: | |
| Online Access: | https://doi.org/10.1111/2041-210X.13967 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849771914942218240 |
|---|---|
| author | Oliver C. Metcalf Jos Barlow Yves Bas Erika Berenguer Christian Devenish Filipe França Stuart Marsden Charlotte Smith Alexander C. Lees |
| author_facet | Oliver C. Metcalf Jos Barlow Yves Bas Erika Berenguer Christian Devenish Filipe França Stuart Marsden Charlotte Smith Alexander C. Lees |
| author_sort | Oliver C. Metcalf |
| collection | DOAJ |
| description | Abstract Passive acoustic monitoring can be an effective method for monitoring species, allowing the assembly of large audio datasets, removing logistical constraints in data collection and reducing anthropogenic monitoring disturbances. However, the analysis of large acoustic datasets is challenging and fully automated machine learning processes are rarely developed or implemented in ecological field studies. One of the greatest uncertainties hindering the development of these methods is spatial generalisability—can an algorithm trained on data from one place be used elsewhere? We demonstrate that heterogeneity of error across space is a problem that could go undetected using common classification accuracy metrics. Second, we develop a method to assess the extent of heterogeneity of error in a random forest classification model for six Amazonian bird species. Finally, we propose two complementary ways to reduce heterogeneity of error, by (i) accounting for it in the thresholding process and (ii) using a secondary classifier that uses contextual data. We found that using a thresholding approach that accounted for heterogeneity of precision error reduced the coefficient of variation of the precision score from a mean of 0.61 ± 0.17 (SD) to 0.41 ± 0.25 in comparison to the initial classification with threshold selection based on F‐score. The use of a secondary, contextual classification with thresholding selection accounting for heterogeneity of precision reduced it further still, to 0.16 ± 0.13, and was significantly lower than the initial classification in all but one species. Mean average precision scores increased, from 0.66 ± 0.4 for the initial classification, to 0.95 ± 0.19, a significant improvement for all species. We recommend assessing—and if necessary correcting for—heterogeneity of precision error when using automated classification on acoustic data to quantify species presence as a function of an environmental, spatial or temporal predictor variable. |
| format | Article |
| id | doaj-art-aade5be4e7ba4818b7ac8f3f9da05a6a |
| institution | DOAJ |
| issn | 2041-210X |
| language | English |
| publishDate | 2022-11-01 |
| publisher | Wiley |
| record_format | Article |
| series | Methods in Ecology and Evolution |
| spelling | doaj-art-aade5be4e7ba4818b7ac8f3f9da05a6a2025-08-20T03:02:28ZengWileyMethods in Ecology and Evolution2041-210X2022-11-0113112559257110.1111/2041-210X.13967Detecting and reducing heterogeneity of error in acoustic classificationOliver C. Metcalf0Jos Barlow1Yves Bas2Erika Berenguer3Christian Devenish4Filipe França5Stuart Marsden6Charlotte Smith7Alexander C. Lees8Division of Biology and Conservation Ecology, Department of Natural Sciences Manchester Metropolitan University Manchester UKLancaster Environment Centre Lancaster University Lancaster UKCESCO, MNHN, CNRS, Sorbonne Univ Paris FranceCESCO, MNHN, CNRS, Sorbonne Univ Paris FranceDivision of Biology and Conservation Ecology, Department of Natural Sciences Manchester Metropolitan University Manchester UKSchool of Biological Sciences University of Bristol Bristol UKDivision of Biology and Conservation Ecology, Department of Natural Sciences Manchester Metropolitan University Manchester UKCESCO, MNHN, CNRS, Sorbonne Univ Paris FranceDivision of Biology and Conservation Ecology, Department of Natural Sciences Manchester Metropolitan University Manchester UKAbstract Passive acoustic monitoring can be an effective method for monitoring species, allowing the assembly of large audio datasets, removing logistical constraints in data collection and reducing anthropogenic monitoring disturbances. However, the analysis of large acoustic datasets is challenging and fully automated machine learning processes are rarely developed or implemented in ecological field studies. One of the greatest uncertainties hindering the development of these methods is spatial generalisability—can an algorithm trained on data from one place be used elsewhere? We demonstrate that heterogeneity of error across space is a problem that could go undetected using common classification accuracy metrics. Second, we develop a method to assess the extent of heterogeneity of error in a random forest classification model for six Amazonian bird species. Finally, we propose two complementary ways to reduce heterogeneity of error, by (i) accounting for it in the thresholding process and (ii) using a secondary classifier that uses contextual data. We found that using a thresholding approach that accounted for heterogeneity of precision error reduced the coefficient of variation of the precision score from a mean of 0.61 ± 0.17 (SD) to 0.41 ± 0.25 in comparison to the initial classification with threshold selection based on F‐score. The use of a secondary, contextual classification with thresholding selection accounting for heterogeneity of precision reduced it further still, to 0.16 ± 0.13, and was significantly lower than the initial classification in all but one species. Mean average precision scores increased, from 0.66 ± 0.4 for the initial classification, to 0.95 ± 0.19, a significant improvement for all species. We recommend assessing—and if necessary correcting for—heterogeneity of precision error when using automated classification on acoustic data to quantify species presence as a function of an environmental, spatial or temporal predictor variable.https://doi.org/10.1111/2041-210X.13967automated signal recognitionautonomous recording unitbioacousticsecoacousticsmachine‐learning |
| spellingShingle | Oliver C. Metcalf Jos Barlow Yves Bas Erika Berenguer Christian Devenish Filipe França Stuart Marsden Charlotte Smith Alexander C. Lees Detecting and reducing heterogeneity of error in acoustic classification Methods in Ecology and Evolution automated signal recognition autonomous recording unit bioacoustics ecoacoustics machine‐learning |
| title | Detecting and reducing heterogeneity of error in acoustic classification |
| title_full | Detecting and reducing heterogeneity of error in acoustic classification |
| title_fullStr | Detecting and reducing heterogeneity of error in acoustic classification |
| title_full_unstemmed | Detecting and reducing heterogeneity of error in acoustic classification |
| title_short | Detecting and reducing heterogeneity of error in acoustic classification |
| title_sort | detecting and reducing heterogeneity of error in acoustic classification |
| topic | automated signal recognition autonomous recording unit bioacoustics ecoacoustics machine‐learning |
| url | https://doi.org/10.1111/2041-210X.13967 |
| work_keys_str_mv | AT olivercmetcalf detectingandreducingheterogeneityoferrorinacousticclassification AT josbarlow detectingandreducingheterogeneityoferrorinacousticclassification AT yvesbas detectingandreducingheterogeneityoferrorinacousticclassification AT erikaberenguer detectingandreducingheterogeneityoferrorinacousticclassification AT christiandevenish detectingandreducingheterogeneityoferrorinacousticclassification AT filipefranca detectingandreducingheterogeneityoferrorinacousticclassification AT stuartmarsden detectingandreducingheterogeneityoferrorinacousticclassification AT charlottesmith detectingandreducingheterogeneityoferrorinacousticclassification AT alexanderclees detectingandreducingheterogeneityoferrorinacousticclassification |