Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits
Improving the performance of human–computer interaction systems is an essential indicator of aircraft intelligence. To address the limitations of single-modal interaction methods, a multimodal interaction model based on gaze and EEG target selection is proposed using deep learning technology. This m...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-03-01
|
| Series: | Future Internet |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1999-5903/17/3/127 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850090611406798848 |
|---|---|
| author | Li Wang Heming Zhang Changyuan Wang |
| author_facet | Li Wang Heming Zhang Changyuan Wang |
| author_sort | Li Wang |
| collection | DOAJ |
| description | Improving the performance of human–computer interaction systems is an essential indicator of aircraft intelligence. To address the limitations of single-modal interaction methods, a multimodal interaction model based on gaze and EEG target selection is proposed using deep learning technology. This model consists of two parts: target classification and intention recognition. The target classification model based on long short-term memory networks is established and trained by combining the eye movement information of the operator. The intention recognition model based on transformers is constructed and trained by combining the operator’s EEG information. In the application scenario of the aircraft radar page system, the highest accuracy of the target classification model is 98%. The intention recognition rate obtained by training the 32-channel EEG information in the intention recognition model is 98.5%, which is higher than other compared models. In addition, we validated the model on a simulated flight platform, and the experimental results show that the proposed multimodal interaction framework outperforms the single gaze interaction in terms of performance. |
| format | Article |
| id | doaj-art-b13313c55bfe4f049da9b45e3a2f8421 |
| institution | DOAJ |
| issn | 1999-5903 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Future Internet |
| spelling | doaj-art-b13313c55bfe4f049da9b45e3a2f84212025-08-20T02:42:32ZengMDPI AGFuture Internet1999-59032025-03-0117312710.3390/fi17030127Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft CockpitsLi Wang0Heming Zhang1Changyuan Wang2School of Electronic & Electrical Engineering, Baoji University of Arts and Sciences, Baoji 721016, ChinaSchool of Optoelectronic Engineering, Xi’an Technological University, Xi’an 710000, ChinaSchool of Computer Science, Xi’an Technological University, Xi’an 710021, ChinaImproving the performance of human–computer interaction systems is an essential indicator of aircraft intelligence. To address the limitations of single-modal interaction methods, a multimodal interaction model based on gaze and EEG target selection is proposed using deep learning technology. This model consists of two parts: target classification and intention recognition. The target classification model based on long short-term memory networks is established and trained by combining the eye movement information of the operator. The intention recognition model based on transformers is constructed and trained by combining the operator’s EEG information. In the application scenario of the aircraft radar page system, the highest accuracy of the target classification model is 98%. The intention recognition rate obtained by training the 32-channel EEG information in the intention recognition model is 98.5%, which is higher than other compared models. In addition, we validated the model on a simulated flight platform, and the experimental results show that the proposed multimodal interaction framework outperforms the single gaze interaction in terms of performance.https://www.mdpi.com/1999-5903/17/3/127deep learningmulti-modallong short-term memory networktransformerhuman–computer interaction |
| spellingShingle | Li Wang Heming Zhang Changyuan Wang Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits Future Internet deep learning multi-modal long short-term memory network transformer human–computer interaction |
| title | Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits |
| title_full | Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits |
| title_fullStr | Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits |
| title_full_unstemmed | Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits |
| title_short | Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits |
| title_sort | deep neural network based modeling of multimodal human computer interaction in aircraft cockpits |
| topic | deep learning multi-modal long short-term memory network transformer human–computer interaction |
| url | https://www.mdpi.com/1999-5903/17/3/127 |
| work_keys_str_mv | AT liwang deepneuralnetworkbasedmodelingofmultimodalhumancomputerinteractioninaircraftcockpits AT hemingzhang deepneuralnetworkbasedmodelingofmultimodalhumancomputerinteractioninaircraftcockpits AT changyuanwang deepneuralnetworkbasedmodelingofmultimodalhumancomputerinteractioninaircraftcockpits |