HandFI: Multilevel Interacting Hand Reconstruction Based on Multilevel Feature Fusion in RGB Images
Interacting hand reconstruction presents significant opportunities in various applications. However, it currently faces challenges such as the difficulty in distinguishing the features of both hands, misalignment of hand meshes with input images, and modeling the complex spatial relationships betwee...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/25/1/88 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841548915101401088 |
|---|---|
| author | Huimin Pan Yuting Cai Jiayi Yang Shaojia Niu Quanli Gao Xihan Wang |
| author_facet | Huimin Pan Yuting Cai Jiayi Yang Shaojia Niu Quanli Gao Xihan Wang |
| author_sort | Huimin Pan |
| collection | DOAJ |
| description | Interacting hand reconstruction presents significant opportunities in various applications. However, it currently faces challenges such as the difficulty in distinguishing the features of both hands, misalignment of hand meshes with input images, and modeling the complex spatial relationships between interacting hands. In this paper, we propose a multilevel feature fusion interactive network for hand reconstruction (HandFI). Within this network, the hand feature separation module utilizes attentional mechanisms and positional coding to distinguish between left-hand and right-hand features while maintaining the spatial relationship of the features. The hand fusion and attention module promotes the alignment of hand vertices with the image by integrating multi-scale hand features while introducing cross-attention to help determine the complex spatial relationships between interacting hands, thereby enhancing the accuracy of two-hand reconstruction. We evaluated our method with existing approaches using the InterHand 2.6M, RGB2Hands, and EgoHands datasets. Extensive experimental results demonstrated that our method outperformed other representative methods, with performance metrics of 9.38 mm for the MPJPE and 9.61 mm for the MPVPE. Additionally, the results obtained in real-world scenes further validated the generalization capability of our method. |
| format | Article |
| id | doaj-art-46b6a091abff4167a7bfb35bdb1f9b94 |
| institution | Kabale University |
| issn | 1424-8220 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj-art-46b6a091abff4167a7bfb35bdb1f9b942025-01-10T13:20:51ZengMDPI AGSensors1424-82202024-12-012518810.3390/s25010088HandFI: Multilevel Interacting Hand Reconstruction Based on Multilevel Feature Fusion in RGB ImagesHuimin Pan0Yuting Cai1Jiayi Yang2Shaojia Niu3Quanli Gao4Xihan Wang5School of Computer Science, Xi’an Polytechnic University, Xi’an 710600, ChinaSchool of Computer Science, Xi’an Polytechnic University, Xi’an 710600, ChinaSchool of Computer Science, Xi’an Polytechnic University, Xi’an 710600, ChinaSchool of Computer Science, Xi’an Polytechnic University, Xi’an 710600, ChinaSchool of Computer Science, Xi’an Polytechnic University, Xi’an 710600, ChinaSchool of Computer Science, Xi’an Polytechnic University, Xi’an 710600, ChinaInteracting hand reconstruction presents significant opportunities in various applications. However, it currently faces challenges such as the difficulty in distinguishing the features of both hands, misalignment of hand meshes with input images, and modeling the complex spatial relationships between interacting hands. In this paper, we propose a multilevel feature fusion interactive network for hand reconstruction (HandFI). Within this network, the hand feature separation module utilizes attentional mechanisms and positional coding to distinguish between left-hand and right-hand features while maintaining the spatial relationship of the features. The hand fusion and attention module promotes the alignment of hand vertices with the image by integrating multi-scale hand features while introducing cross-attention to help determine the complex spatial relationships between interacting hands, thereby enhancing the accuracy of two-hand reconstruction. We evaluated our method with existing approaches using the InterHand 2.6M, RGB2Hands, and EgoHands datasets. Extensive experimental results demonstrated that our method outperformed other representative methods, with performance metrics of 9.38 mm for the MPJPE and 9.61 mm for the MPVPE. Additionally, the results obtained in real-world scenes further validated the generalization capability of our method.https://www.mdpi.com/1424-8220/25/1/88interacting hand reconstructionfeature fusionMANO |
| spellingShingle | Huimin Pan Yuting Cai Jiayi Yang Shaojia Niu Quanli Gao Xihan Wang HandFI: Multilevel Interacting Hand Reconstruction Based on Multilevel Feature Fusion in RGB Images Sensors interacting hand reconstruction feature fusion MANO |
| title | HandFI: Multilevel Interacting Hand Reconstruction Based on Multilevel Feature Fusion in RGB Images |
| title_full | HandFI: Multilevel Interacting Hand Reconstruction Based on Multilevel Feature Fusion in RGB Images |
| title_fullStr | HandFI: Multilevel Interacting Hand Reconstruction Based on Multilevel Feature Fusion in RGB Images |
| title_full_unstemmed | HandFI: Multilevel Interacting Hand Reconstruction Based on Multilevel Feature Fusion in RGB Images |
| title_short | HandFI: Multilevel Interacting Hand Reconstruction Based on Multilevel Feature Fusion in RGB Images |
| title_sort | handfi multilevel interacting hand reconstruction based on multilevel feature fusion in rgb images |
| topic | interacting hand reconstruction feature fusion MANO |
| url | https://www.mdpi.com/1424-8220/25/1/88 |
| work_keys_str_mv | AT huiminpan handfimultilevelinteractinghandreconstructionbasedonmultilevelfeaturefusioninrgbimages AT yutingcai handfimultilevelinteractinghandreconstructionbasedonmultilevelfeaturefusioninrgbimages AT jiayiyang handfimultilevelinteractinghandreconstructionbasedonmultilevelfeaturefusioninrgbimages AT shaojianiu handfimultilevelinteractinghandreconstructionbasedonmultilevelfeaturefusioninrgbimages AT quanligao handfimultilevelinteractinghandreconstructionbasedonmultilevelfeaturefusioninrgbimages AT xihanwang handfimultilevelinteractinghandreconstructionbasedonmultilevelfeaturefusioninrgbimages |