Literature Survey: Sign Language Recognition Using Gesture Recognition and Natural Language Processing

Background:: People suffering from hearing and speaking disabilities have a few ways of communicating with other people. One of these is to communicate through the use of sign language. Objective:: Developing a system for sign language recognition becomes essential for deaf as well as a mute person. The recognition system acts as a translator between a disabled and an able person. This eliminates the hindrances in exchange of ideas. Most of the existing systems are very poorly designed with limited support for the needs of their day to day facilities. Methods:: The proposed system embedded with gesture recognition capability has been introduced here which extracts signs from a video sequence and displays them on screen. On the other hand, a speech to text as well as text to speech system is also introduced to further facilitate the grieved people. To get the best out of human computer relationship, the proposed solution consists of various cutting-edge technologies and Machine Learning based sign recognition models which have been trained by using Tensor Flow and Keras library. Result:: The proposed architecture works better than several gesture recognition techniques like background elimination and conversion to HSV because of sharply defined image provided to the model for classification. The results of testing indicate reliable recognition systems with high accuracy that includes most of the essential and necessary features for any deaf and dumb person in his/her day to day tasks. Conclusion:: It’s the need of current technological advances to develop reliable solutions which can be deployed to assist deaf and dumb people to adjust to normal life. Instead of focusing on a standalone technology, a plethora of them have been introduced in this proposed work. Proposed Sign Recognition System is based on feature extraction and classification. The trained model helps in identification of different gestures.

Download Full-text

Wi-SL: Contactless Fine-Grained Gesture Recognition Uses Channel State Information

Sensors ◽

10.3390/s20144025 ◽

2020 ◽

Vol 20 (14) ◽

pp. 4025

Author(s):

Zhanjun Hao ◽

Yu Duan ◽

Xiaochao Dang ◽

Yang Liu ◽

Daiyang Zhang

Keyword(s):

Sign Language ◽

Gesture Recognition ◽

Channel State Information ◽

Computational Cost ◽

Support Vector ◽

Language Recognition ◽

Sign Language Recognition ◽

Channel State ◽

State Information ◽

Fine Grained

In recent years, with the development of wireless sensing technology and the widespread popularity of WiFi devices, human perception based on WiFi has become possible, and gesture recognition has become an active topic in the field of human-computer interaction. As a kind of gesture, sign language is widely used in life. The establishment of an effective sign language recognition system can help people with aphasia and hearing impairment to better interact with the computer and facilitate their daily life. For this reason, this paper proposes a contactless fine-grained gesture recognition method using Channel State Information (CSI), namely Wi-SL. This method uses a commercial WiFi device to establish the correlation mapping between the amplitude and phase difference information of the subcarrier level in the wireless signal and the sign language action, without requiring the user to wear any device. We combine an efficient denoising method to filter environmental interference with an effective selection of optimal subcarriers to reduce the computational cost of the system. We also use K-means combined with a Bagging algorithm to optimize the Support Vector Machine (SVM) classification (KSB) model to enhance the classification of sign language action data. We implemented the algorithms and evaluated them for three different scenarios. The experimental results show that the average accuracy of Wi-SL gesture recognition can reach 95.8%, which realizes device-free, non-invasive, high-precision sign language gesture recognition.

Download Full-text

Spoken language processing techniques for sign language recognition and translation

Technology and Disability ◽

10.3233/tad-2008-20207 ◽

2008 ◽

Vol 20 (2) ◽

pp. 121-133 ◽

Cited By ~ 15

Author(s):

Philippe Dreuw ◽

Daniel Stein ◽

Thomas Deselaers ◽

David Rybach ◽

Morteza Zahedi ◽

...

Keyword(s):

Sign Language ◽

Language Processing ◽

Spoken Language ◽

Language Recognition ◽

Sign Language Recognition ◽

Spoken Language Processing ◽

Processing Techniques

Download Full-text

Analysis of Japanese-Japanese Sign Language Dictionary Using Natural Language Processing

Japanese Journal of Sign Language Studies ◽

10.7877/jasl.17.85 ◽

2008 ◽

Vol 17 (0) ◽

pp. 85-92

Author(s):

Tomohiro Kuroda ◽

Kazuya Okamoto ◽

Tadamasa Takemura ◽

Naoki Oboshi ◽

Yoshihiro Kuroda ◽

...

Keyword(s):

Natural Language Processing ◽

Natural Language ◽

Sign Language ◽

Language Processing ◽

Japanese Sign Language

Download Full-text

An evaluation of real-time requirements for automatic sign language recognition using ANNs and HMMs - The LIBRAS use case

Journal of Interactive Systems ◽

10.5753/jis.2013.624 ◽

2013 ◽

Vol 4 (1) ◽

pp. 1

Author(s):

Ednaldo Brigante Pizzolato ◽

Mauro dos Santos Anjo ◽

Sebastian Feuerstack

Keyword(s):

Real Time ◽

Sign Language ◽

Gesture Recognition ◽

Large Set ◽

Language Recognition ◽

Sign Language Recognition ◽

Sign Languages ◽

Real Time System ◽

Dynamic Gestures ◽

Time Requirements

Sign languages are the natural way Deafs use to communicate with other people. They have their own formal semantic definitions and syntactic rules and are composed by a large set of gestures involving hands and head. Automatic recognition of sign languages (ARSL) tries to recognize the signs and translate them into a written language. ARSL is a challenging task as it involves background segmentation, hands and head posture modeling, recognition and tracking, temporal analysis and syntactic and semantic interpretation. Moreover, when real-time requirements are considered, this task becomes even more challenging. In this paper, we present a study of real time requirements of automatic sign language recognition of small sets of static and dynamic gestures of the Brazilian Sign Language (LIBRAS). For the task of static gesture recognition, we implemented a system that is able to work on small sub-sets of the alphabet - like A,E,I,O,U and B,C,F,L,V - reaching very high recognition rates. For the task of dynamic gesture recognition, we tested our system over a small set of LIBRAS words and collected the execution times. The aim was to gather knowledge regarding execution time of all the recognition processes (like segmentation, analysis and recognition itself) to evaluate the feasibility of building a real-time system to recognize small sets of both static and dynamic gestures. Our findings indicate that the bottleneck of our current architecture is the recognition phase.

Download Full-text

Indian Sign Language Recognition through Hybrid ConvNet-LSTM Networks

EMITTER International Journal of Engineering Technology ◽

10.24003/emitter.v9i1.613 ◽

2021 ◽

Vol 9 (1) ◽

pp. 182-203

Author(s):

Muthu Mariappan H ◽

Dr Gomathi V

Keyword(s):

Neural Network ◽

Computer Vision ◽

Real Time ◽

Sign Language ◽

Gesture Recognition ◽

Language Translation ◽

Video Gaming ◽

Language Recognition ◽

Sign Language Recognition ◽

Indian Sign Language

Dynamic hand gesture recognition is a challenging task of Human-Computer Interaction (HCI) and Computer Vision. The potential application areas of gesture recognition include sign language translation, video gaming, video surveillance, robotics, and gesture-controlled home appliances. In the proposed research, gesture recognition is applied to recognize sign language words from real-time videos. Classifying the actions from video sequences requires both spatial and temporal features. The proposed system handles the former by the Convolutional Neural Network (CNN), which is the core of several computer vision solutions and the latter by the Recurrent Neural Network (RNN), which is more efficient in handling the sequences of movements. Thus, the real-time Indian sign language (ISL) recognition system is developed using the hybrid CNN-RNN architecture. The system is trained with the proposed CasTalk-ISL dataset. The ultimate purpose of the presented research is to deploy a real-time sign language translator to break the hurdles present in the communication between hearing-impaired people and normal people. The developed system achieves 95.99% top-1 accuracy and 99.46% top-3 accuracy on the test dataset. The obtained results outperform the existing approaches using various deep models on different datasets.

Download Full-text

A comparison of convolutional neural networks for Kazakh sign language recognition

Eastern-European Journal of Enterprise Technologies ◽

10.15587/1729-4061.2021.241535 ◽

2021 ◽

Vol 5 (2 (113)) ◽

pp. 44-54

Author(s):

Chingiz Kenshimov ◽

Samat Mukhanov ◽

Timur Merembayev ◽

Didar Yedilkhan

Keyword(s):

Neural Networks ◽

Sign Language ◽

Convolutional Neural Networks ◽

Gesture Recognition ◽

State Of The Art ◽

Hand Gesture ◽

Language Recognition ◽

Sign Language Recognition ◽

Important Means ◽

Complex Relationships

For people with disabilities, sign language is the most important means of communication. Therefore, more and more authors of various papers and scientists around the world are proposing solutions to use intelligent hand gesture recognition systems. Such a system is aimed not only for those who wish to understand a sign language, but also speak using gesture recognition software. In this paper, a new benchmark dataset for Kazakh fingerspelling, able to train deep neural networks, is introduced. The dataset contains more than 10122 gesture samples for 42 alphabets. The alphabet has its own peculiarities as some characters are shown in motion, which may influence sign recognition. Research and analysis of convolutional neural networks, comparison, testing, results and analysis of LeNet, AlexNet, ResNet and EffectiveNet – EfficientNetB7 methods are described in the paper. EffectiveNet architecture is state-of-the-art (SOTA) and is supposed to be a new one compared to other architectures under consideration. On this dataset, we showed that the LeNet and EffectiveNet networks outperform other competing algorithms. Moreover, EffectiveNet can achieve state-of-the-art performance on nother hand gesture datasets. The architecture and operation principle of these algorithms reflect the effectiveness of their application in sign language recognition. The evaluation of the CNN model score is conducted by using the accuracy and penalty matrix. During training epochs, LeNet and EffectiveNet showed better results: accuracy and loss function had similar and close trends. The results of EffectiveNet were explained by the tools of the SHapley Additive exPlanations (SHAP) framework. SHAP explored the model to detect complex relationships between features in the images. Focusing on the SHAP tool may help to further improve the accuracy of the model

Download Full-text