Implementing Gesture Recognition in a Sign Language Learning Application

Background:: People suffering from hearing and speaking disabilities have a few ways of communicating with other people. One of these is to communicate through the use of sign language. Objective:: Developing a system for sign language recognition becomes essential for deaf as well as a mute person. The recognition system acts as a translator between a disabled and an able person. This eliminates the hindrances in exchange of ideas. Most of the existing systems are very poorly designed with limited support for the needs of their day to day facilities. Methods:: The proposed system embedded with gesture recognition capability has been introduced here which extracts signs from a video sequence and displays them on screen. On the other hand, a speech to text as well as text to speech system is also introduced to further facilitate the grieved people. To get the best out of human computer relationship, the proposed solution consists of various cutting-edge technologies and Machine Learning based sign recognition models which have been trained by using Tensor Flow and Keras library. Result:: The proposed architecture works better than several gesture recognition techniques like background elimination and conversion to HSV because of sharply defined image provided to the model for classification. The results of testing indicate reliable recognition systems with high accuracy that includes most of the essential and necessary features for any deaf and dumb person in his/her day to day tasks. Conclusion:: It’s the need of current technological advances to develop reliable solutions which can be deployed to assist deaf and dumb people to adjust to normal life. Instead of focusing on a standalone technology, a plethora of them have been introduced in this proposed work. Proposed Sign Recognition System is based on feature extraction and classification. The trained model helps in identification of different gestures.

Download Full-text

Human-centered design for a sign language learning application

Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments ◽

10.1145/3389189.3398007 ◽

2020 ◽

Author(s):

Jerry Schnepp ◽

Rosalee Wolfe ◽

Gilbert Brionez ◽

Souad Baowidan ◽

Ronan Johnson ◽

...

Keyword(s):

Language Learning ◽

Sign Language ◽

Human Centered Design

Download Full-text

Alphabetical Gesture Recognition of American Sign Language using E-Voice Smart Glove

2020 IEEE 23rd International Multitopic Conference (INMIC) ◽

10.1109/inmic50486.2020.9318185 ◽

2020 ◽

Author(s):

Muhammad Saad Amin ◽

Muhammad Talha Amin ◽

Muhammad Yasir Latif ◽

Ali Asghar Jathol ◽

Nisar Ahmed ◽

...

Keyword(s):

American Sign Language ◽

Sign Language ◽

Gesture Recognition ◽

American Sign

Download Full-text

3D Avatar Approach for Continuous Sign Movement Using Speech/Text

Applied Sciences ◽

10.3390/app11083439 ◽

2021 ◽

Vol 11 (8) ◽

pp. 3439

Author(s):

Debashis Das Chakladar ◽

Pradeep Kumar ◽

Shubham Mandal ◽

Partha Pratim Roy ◽

Masakazu Iwamura ◽

...

Keyword(s):

Language Learning ◽

Sign Language ◽

Language Processing ◽

Hearing Impaired ◽

Learning System ◽

English Sentence ◽

Impaired People ◽

Indian Sign Language ◽

Impaired Person ◽

3D Avatar

Sign language is a visual language for communication used by hearing-impaired people with the help of hand and finger movements. Indian Sign Language (ISL) is a well-developed and standard way of communication for hearing-impaired people living in India. However, other people who use spoken language always face difficulty while communicating with a hearing-impaired person due to lack of sign language knowledge. In this study, we have developed a 3D avatar-based sign language learning system that converts the input speech/text into corresponding sign movements for ISL. The system consists of three modules. Initially, the input speech is converted into an English sentence. Then, that English sentence is converted into the corresponding ISL sentence using the Natural Language Processing (NLP) technique. Finally, the motion of the 3D avatar is defined based on the ISL sentence. The translation module achieves a 10.50 SER (Sign Error Rate) score.

Download Full-text