Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data
This paper presents a novel Arabic Sign Language (ArSL) recognition system, using selected 2D hands and body key points from successive video frames. The system recognizes the recorded video signs, for both signer dependent and signer independent modes, using the concatenation of a 3D CNN skeleton n...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9389720/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846128594073419776 |
|---|---|
| author | Mohamed A. Bencherif Mohammed Algabri Mohamed A. Mekhtiche Mohammed Faisal Mansour Alsulaiman Hassan Mathkour Muneer Al-Hammadi Hamid Ghaleb |
| author_facet | Mohamed A. Bencherif Mohammed Algabri Mohamed A. Mekhtiche Mohammed Faisal Mansour Alsulaiman Hassan Mathkour Muneer Al-Hammadi Hamid Ghaleb |
| author_sort | Mohamed A. Bencherif |
| collection | DOAJ |
| description | This paper presents a novel Arabic Sign Language (ArSL) recognition system, using selected 2D hands and body key points from successive video frames. The system recognizes the recorded video signs, for both signer dependent and signer independent modes, using the concatenation of a 3D CNN skeleton network and a 2D point convolution network. To accomplish this, we built a new ArSL video-based sign database. We will present the detailed methodology of recording the new dataset, which comprises 80 static and dynamic signs that were repeated five times by 40 signers. The signs include Arabic alphabet, numbers, and some daily use signs. To facilitate building an online sign recognition system, we introduce the inverse efficiency score to find a sufficient optimal number of successive frames for the recognition decision, in order to cope with a near real-time automatic ArSL system, where tradeoff between accuracy and speed is crucial to avoid delayed sign classification. For the dependent mode, best results were obtained for dynamic signs with an accuracy of 98.39%, and 88.89% for the static signs, and for the independent mode, we obtained for the dynamic signs an accuracy of 96.69%, and 86.34% for the static signs. When both the static and dynamic signs were mixed and the system trained with all the signs, accuracies of 89.62% and 88.09% were obtained in the signer dependent and signer independent modes respectively. |
| format | Article |
| id | doaj-art-26eb12bf4cf14a6f8c57d83a6fadf4d9 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-26eb12bf4cf14a6f8c57d83a6fadf4d92024-12-11T00:02:10ZengIEEEIEEE Access2169-35362021-01-019596125962710.1109/ACCESS.2021.30697149389720Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton DataMohamed A. Bencherif0https://orcid.org/0000-0001-8147-8679Mohammed Algabri1https://orcid.org/0000-0001-7962-8121Mohamed A. Mekhtiche2https://orcid.org/0000-0001-9478-9206Mohammed Faisal3https://orcid.org/0000-0001-7720-0076Mansour Alsulaiman4https://orcid.org/0000-0003-2866-184XHassan Mathkour5Muneer Al-Hammadi6https://orcid.org/0000-0001-8174-5888Hamid Ghaleb7https://orcid.org/0000-0001-5768-0404Department of Computer Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi ArabiaDepartment of Computer Science, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi ArabiaDepartment of Computer Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi ArabiaCollege of Applied Computer Science, King Saud University (KSU), Riyadh, Saudi ArabiaDepartment of Computer Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi ArabiaDepartment of Computer Science, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi ArabiaDepartment of Computer Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi ArabiaDepartment of Software Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi ArabiaThis paper presents a novel Arabic Sign Language (ArSL) recognition system, using selected 2D hands and body key points from successive video frames. The system recognizes the recorded video signs, for both signer dependent and signer independent modes, using the concatenation of a 3D CNN skeleton network and a 2D point convolution network. To accomplish this, we built a new ArSL video-based sign database. We will present the detailed methodology of recording the new dataset, which comprises 80 static and dynamic signs that were repeated five times by 40 signers. The signs include Arabic alphabet, numbers, and some daily use signs. To facilitate building an online sign recognition system, we introduce the inverse efficiency score to find a sufficient optimal number of successive frames for the recognition decision, in order to cope with a near real-time automatic ArSL system, where tradeoff between accuracy and speed is crucial to avoid delayed sign classification. For the dependent mode, best results were obtained for dynamic signs with an accuracy of 98.39%, and 88.89% for the static signs, and for the independent mode, we obtained for the dynamic signs an accuracy of 96.69%, and 86.34% for the static signs. When both the static and dynamic signs were mixed and the system trained with all the signs, accuracies of 89.62% and 88.09% were obtained in the signer dependent and signer independent modes respectively.https://ieeexplore.ieee.org/document/9389720/Arabic sign languageOpenPoseskeletonkey pointsparallel convolutions |
| spellingShingle | Mohamed A. Bencherif Mohammed Algabri Mohamed A. Mekhtiche Mohammed Faisal Mansour Alsulaiman Hassan Mathkour Muneer Al-Hammadi Hamid Ghaleb Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data IEEE Access Arabic sign language OpenPose skeleton key points parallel convolutions |
| title | Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data |
| title_full | Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data |
| title_fullStr | Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data |
| title_full_unstemmed | Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data |
| title_short | Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data |
| title_sort | arabic sign language recognition system using 2d hands and body skeleton data |
| topic | Arabic sign language OpenPose skeleton key points parallel convolutions |
| url | https://ieeexplore.ieee.org/document/9389720/ |
| work_keys_str_mv | AT mohamedabencherif arabicsignlanguagerecognitionsystemusing2dhandsandbodyskeletondata AT mohammedalgabri arabicsignlanguagerecognitionsystemusing2dhandsandbodyskeletondata AT mohamedamekhtiche arabicsignlanguagerecognitionsystemusing2dhandsandbodyskeletondata AT mohammedfaisal arabicsignlanguagerecognitionsystemusing2dhandsandbodyskeletondata AT mansouralsulaiman arabicsignlanguagerecognitionsystemusing2dhandsandbodyskeletondata AT hassanmathkour arabicsignlanguagerecognitionsystemusing2dhandsandbodyskeletondata AT muneeralhammadi arabicsignlanguagerecognitionsystemusing2dhandsandbodyskeletondata AT hamidghaleb arabicsignlanguagerecognitionsystemusing2dhandsandbodyskeletondata |