FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses
With the growing use of head-mounted displays (HMDs) such as smart glasses, text input remains a challenge, especially in mobile environments. Conventional methods like physical keyboards, voice recognition, and virtual keyboards each have limitations—physical keyboards lack portability, voice input...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-07-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/15/8122 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849407817396518912 |
|---|---|
| author | Ryo Yuasa Katashi Nagao |
| author_facet | Ryo Yuasa Katashi Nagao |
| author_sort | Ryo Yuasa |
| collection | DOAJ |
| description | With the growing use of head-mounted displays (HMDs) such as smart glasses, text input remains a challenge, especially in mobile environments. Conventional methods like physical keyboards, voice recognition, and virtual keyboards each have limitations—physical keyboards lack portability, voice input has privacy concerns, and virtual keyboards struggle with accuracy due to a lack of tactile feedback. FlickPose is a novel text input system designed for smart glasses and mobile HMD users, integrating flick-based input and hand pose recognition. It features two key selection methods: the touch-panel method, where users tap a floating UI panel to select characters, and the raycast method, where users point a virtual ray from their wrist and confirm input via a pinch motion. FlickPose uses five left-hand poses to select characters. A machine learning model trained for hand pose recognition outperforms Random Forest and LightGBM models in accuracy and consistency. FlickPose was tested against the standard virtual keyboard of Meta Quest 3 in three tasks (hiragana, alphanumeric, and kanji input). Results showed that raycast had the lowest error rate, reducing unintended key presses; touch-panel had more deletions, likely due to misjudgments in key selection; and frequent HMD users preferred raycast, as it maintained input accuracy while allowing users to monitor their text. A key feature of FlickPose is adaptive tracking, which ensures the keyboard follows user movement. While further refinements in hand pose recognition are needed, the system provides an efficient, mobile-friendly alternative for HMD text input. Future research will explore real-world application compatibility and improve usability in dynamic environments. |
| format | Article |
| id | doaj-art-8b6e6e86ad0a48b086d4f5a6e12bb974 |
| institution | Kabale University |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-8b6e6e86ad0a48b086d4f5a6e12bb9742025-08-20T03:35:57ZengMDPI AGApplied Sciences2076-34172025-07-011515812210.3390/app15158122FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart GlassesRyo Yuasa0Katashi Nagao1Graduate School of Informatics, Nagoya University, Nagoya 464-8603, JapanGraduate School of Informatics, Nagoya University, Nagoya 464-8603, JapanWith the growing use of head-mounted displays (HMDs) such as smart glasses, text input remains a challenge, especially in mobile environments. Conventional methods like physical keyboards, voice recognition, and virtual keyboards each have limitations—physical keyboards lack portability, voice input has privacy concerns, and virtual keyboards struggle with accuracy due to a lack of tactile feedback. FlickPose is a novel text input system designed for smart glasses and mobile HMD users, integrating flick-based input and hand pose recognition. It features two key selection methods: the touch-panel method, where users tap a floating UI panel to select characters, and the raycast method, where users point a virtual ray from their wrist and confirm input via a pinch motion. FlickPose uses five left-hand poses to select characters. A machine learning model trained for hand pose recognition outperforms Random Forest and LightGBM models in accuracy and consistency. FlickPose was tested against the standard virtual keyboard of Meta Quest 3 in three tasks (hiragana, alphanumeric, and kanji input). Results showed that raycast had the lowest error rate, reducing unintended key presses; touch-panel had more deletions, likely due to misjudgments in key selection; and frequent HMD users preferred raycast, as it maintained input accuracy while allowing users to monitor their text. A key feature of FlickPose is adaptive tracking, which ensures the keyboard follows user movement. While further refinements in hand pose recognition are needed, the system provides an efficient, mobile-friendly alternative for HMD text input. Future research will explore real-world application compatibility and improve usability in dynamic environments.https://www.mdpi.com/2076-3417/15/15/8122mixed realitytext inputhand trackinghand pose recognitiondeep learning |
| spellingShingle | Ryo Yuasa Katashi Nagao FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses Applied Sciences mixed reality text input hand tracking hand pose recognition deep learning |
| title | FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses |
| title_full | FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses |
| title_fullStr | FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses |
| title_full_unstemmed | FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses |
| title_short | FlickPose: A Hand Tracking-Based Text Input System for Mobile Users Wearing Smart Glasses |
| title_sort | flickpose a hand tracking based text input system for mobile users wearing smart glasses |
| topic | mixed reality text input hand tracking hand pose recognition deep learning |
| url | https://www.mdpi.com/2076-3417/15/15/8122 |
| work_keys_str_mv | AT ryoyuasa flickposeahandtrackingbasedtextinputsystemformobileuserswearingsmartglasses AT katashinagao flickposeahandtrackingbasedtextinputsystemformobileuserswearingsmartglasses |