Human Operator Mental Fatigue Assessment Based on Video: ML-Driven Approach and Its Application to HFAVD Dataset
The detection of the human mental fatigue state holds immense significance due to its direct impact on work efficiency, specifically in system operation control. Numerous approaches have been proposed to address the challenge of fatigue detection, aiming to identify signs of fatigue and alert the in...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-11-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/14/22/10510 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850217254872940544 |
|---|---|
| author | Walaa Othman Batol Hamoud Nikolay Shilov Alexey Kashevnik |
| author_facet | Walaa Othman Batol Hamoud Nikolay Shilov Alexey Kashevnik |
| author_sort | Walaa Othman |
| collection | DOAJ |
| description | The detection of the human mental fatigue state holds immense significance due to its direct impact on work efficiency, specifically in system operation control. Numerous approaches have been proposed to address the challenge of fatigue detection, aiming to identify signs of fatigue and alert the individual. This paper introduces an approach to human mental fatigue assessment based on the application of machine learning techniques to the video of a working operator. For validation purposes, the approach was applied to a dataset, “Human Fatigue Assessment Based on Video Data” (HFAVD) integrating video data with features computed by using our computer vision deep learning models. The incorporated features encompass head movements represented by Euler angles (roll, pitch, and yaw), vital signs (blood pressure, heart rate, oxygen saturation, and respiratory rate), and eye and mouth states (blinking and yawning). The integration of these features eliminates the need for the manual calculation or detection of these parameters, and it obviates the requirement for sensors and external devices, which are commonly employed in existing datasets. The main objective of our work is to advance research in fatigue detection, particularly in work and academic settings. For this reason, we conducted a series of experiments by utilizing machine learning techniques to analyze the dataset and assess the fatigue state based on the features predicted by our models. The results reveal that the random forest technique consistently achieved the highest accuracy and F1-score across all experiments, predominantly exceeding 90%. These findings suggest that random forest is a highly promising technique for this task and prove the strong connection and association among the predicted features used to annotate the videos and the state of fatigue. |
| format | Article |
| id | doaj-art-b003633a576d411a9142b2740a9dec99 |
| institution | OA Journals |
| issn | 2076-3417 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-b003633a576d411a9142b2740a9dec992025-08-20T02:08:07ZengMDPI AGApplied Sciences2076-34172024-11-0114221051010.3390/app142210510Human Operator Mental Fatigue Assessment Based on Video: ML-Driven Approach and Its Application to HFAVD DatasetWalaa Othman0Batol Hamoud1Nikolay Shilov2Alexey Kashevnik3Saint Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaSaint Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaSaint Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaSaint Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 199178 St. Petersburg, RussiaThe detection of the human mental fatigue state holds immense significance due to its direct impact on work efficiency, specifically in system operation control. Numerous approaches have been proposed to address the challenge of fatigue detection, aiming to identify signs of fatigue and alert the individual. This paper introduces an approach to human mental fatigue assessment based on the application of machine learning techniques to the video of a working operator. For validation purposes, the approach was applied to a dataset, “Human Fatigue Assessment Based on Video Data” (HFAVD) integrating video data with features computed by using our computer vision deep learning models. The incorporated features encompass head movements represented by Euler angles (roll, pitch, and yaw), vital signs (blood pressure, heart rate, oxygen saturation, and respiratory rate), and eye and mouth states (blinking and yawning). The integration of these features eliminates the need for the manual calculation or detection of these parameters, and it obviates the requirement for sensors and external devices, which are commonly employed in existing datasets. The main objective of our work is to advance research in fatigue detection, particularly in work and academic settings. For this reason, we conducted a series of experiments by utilizing machine learning techniques to analyze the dataset and assess the fatigue state based on the features predicted by our models. The results reveal that the random forest technique consistently achieved the highest accuracy and F1-score across all experiments, predominantly exceeding 90%. These findings suggest that random forest is a highly promising technique for this task and prove the strong connection and association among the predicted features used to annotate the videos and the state of fatigue.https://www.mdpi.com/2076-3417/14/22/10510operator fatigue detectioncomputer visionphysiological indicatormachine learning |
| spellingShingle | Walaa Othman Batol Hamoud Nikolay Shilov Alexey Kashevnik Human Operator Mental Fatigue Assessment Based on Video: ML-Driven Approach and Its Application to HFAVD Dataset Applied Sciences operator fatigue detection computer vision physiological indicator machine learning |
| title | Human Operator Mental Fatigue Assessment Based on Video: ML-Driven Approach and Its Application to HFAVD Dataset |
| title_full | Human Operator Mental Fatigue Assessment Based on Video: ML-Driven Approach and Its Application to HFAVD Dataset |
| title_fullStr | Human Operator Mental Fatigue Assessment Based on Video: ML-Driven Approach and Its Application to HFAVD Dataset |
| title_full_unstemmed | Human Operator Mental Fatigue Assessment Based on Video: ML-Driven Approach and Its Application to HFAVD Dataset |
| title_short | Human Operator Mental Fatigue Assessment Based on Video: ML-Driven Approach and Its Application to HFAVD Dataset |
| title_sort | human operator mental fatigue assessment based on video ml driven approach and its application to hfavd dataset |
| topic | operator fatigue detection computer vision physiological indicator machine learning |
| url | https://www.mdpi.com/2076-3417/14/22/10510 |
| work_keys_str_mv | AT walaaothman humanoperatormentalfatigueassessmentbasedonvideomldrivenapproachanditsapplicationtohfavddataset AT batolhamoud humanoperatormentalfatigueassessmentbasedonvideomldrivenapproachanditsapplicationtohfavddataset AT nikolayshilov humanoperatormentalfatigueassessmentbasedonvideomldrivenapproachanditsapplicationtohfavddataset AT alexeykashevnik humanoperatormentalfatigueassessmentbasedonvideomldrivenapproachanditsapplicationtohfavddataset |