Feature-Based Normality Models for Anomaly Detection
Detecting previously unseen anomalies in sensor data is a challenging problem for artificial intelligence when sensor-specific and deployment-specific characteristics of the time series need to be learned from a short calibration period. From the application point of view, this challenge becomes inc...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-08-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/25/15/4757 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849405838675935232 |
|---|---|
| author | Hui Yie Teh Kevin I-Kai Wang Andreas W. Kempa-Liehr |
| author_facet | Hui Yie Teh Kevin I-Kai Wang Andreas W. Kempa-Liehr |
| author_sort | Hui Yie Teh |
| collection | DOAJ |
| description | Detecting previously unseen anomalies in sensor data is a challenging problem for artificial intelligence when sensor-specific and deployment-specific characteristics of the time series need to be learned from a short calibration period. From the application point of view, this challenge becomes increasingly important because many applications are gravitating towards utilising low-cost sensors for Internet of Things deployments. While these sensors offer cost-effectiveness and customisation, their data quality does not match that of their high-end counterparts. To improve sensor data quality while addressing the challenges of anomaly detection in Internet of Things applications, we present an anomaly detection framework that learns a normality model of sensor data. The framework models the typical behaviour of individual sensors, which is crucial for the reliable detection of sensor data anomalies, especially when dealing with sensors observing significantly different signal characteristics. Our framework learns sensor-specific normality models from a small set of anomaly-free training data while employing an unsupervised feature engineering approach to select statistically significant features. The selected features are subsequently used to train a Local Outlier Factor anomaly detection model, which adaptively determines the boundary separating normal data from anomalies. The proposed anomaly detection framework is evaluated on three real-world public environmental monitoring datasets with heterogeneous sensor readings. The sensor-specific normality models are learned from extremely short calibration periods (as short as the first 3 days or 10% of the total recorded data) and outperform four other state-of-the-art anomaly detection approaches with respect to F1-score (between 5.4% and 9.3% better) and Matthews correlation coefficient (between 4.0% and 7.6% better). |
| format | Article |
| id | doaj-art-ae095969651b46a1a80b6ab0e0b34b4e |
| institution | Kabale University |
| issn | 1424-8220 |
| language | English |
| publishDate | 2025-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj-art-ae095969651b46a1a80b6ab0e0b34b4e2025-08-20T03:36:34ZengMDPI AGSensors1424-82202025-08-012515475710.3390/s25154757Feature-Based Normality Models for Anomaly DetectionHui Yie Teh0Kevin I-Kai Wang1Andreas W. Kempa-Liehr2Department of Electrical, Computer and Software Engineering, The University of Auckland, Auckland 1142, New ZealandDepartment of Electrical, Computer and Software Engineering, The University of Auckland, Auckland 1142, New ZealandDepartment of Engineering Science and Biomedical Engineering, The University of Auckland, Auckland 1142, New ZealandDetecting previously unseen anomalies in sensor data is a challenging problem for artificial intelligence when sensor-specific and deployment-specific characteristics of the time series need to be learned from a short calibration period. From the application point of view, this challenge becomes increasingly important because many applications are gravitating towards utilising low-cost sensors for Internet of Things deployments. While these sensors offer cost-effectiveness and customisation, their data quality does not match that of their high-end counterparts. To improve sensor data quality while addressing the challenges of anomaly detection in Internet of Things applications, we present an anomaly detection framework that learns a normality model of sensor data. The framework models the typical behaviour of individual sensors, which is crucial for the reliable detection of sensor data anomalies, especially when dealing with sensors observing significantly different signal characteristics. Our framework learns sensor-specific normality models from a small set of anomaly-free training data while employing an unsupervised feature engineering approach to select statistically significant features. The selected features are subsequently used to train a Local Outlier Factor anomaly detection model, which adaptively determines the boundary separating normal data from anomalies. The proposed anomaly detection framework is evaluated on three real-world public environmental monitoring datasets with heterogeneous sensor readings. The sensor-specific normality models are learned from extremely short calibration periods (as short as the first 3 days or 10% of the total recorded data) and outperform four other state-of-the-art anomaly detection approaches with respect to F1-score (between 5.4% and 9.3% better) and Matthews correlation coefficient (between 4.0% and 7.6% better).https://www.mdpi.com/1424-8220/25/15/4757anomaly detectionfeature engineeringnormality modelsensor data qualitytime series analytics |
| spellingShingle | Hui Yie Teh Kevin I-Kai Wang Andreas W. Kempa-Liehr Feature-Based Normality Models for Anomaly Detection Sensors anomaly detection feature engineering normality model sensor data quality time series analytics |
| title | Feature-Based Normality Models for Anomaly Detection |
| title_full | Feature-Based Normality Models for Anomaly Detection |
| title_fullStr | Feature-Based Normality Models for Anomaly Detection |
| title_full_unstemmed | Feature-Based Normality Models for Anomaly Detection |
| title_short | Feature-Based Normality Models for Anomaly Detection |
| title_sort | feature based normality models for anomaly detection |
| topic | anomaly detection feature engineering normality model sensor data quality time series analytics |
| url | https://www.mdpi.com/1424-8220/25/15/4757 |
| work_keys_str_mv | AT huiyieteh featurebasednormalitymodelsforanomalydetection AT kevinikaiwang featurebasednormalitymodelsforanomalydetection AT andreaswkempaliehr featurebasednormalitymodelsforanomalydetection |