scholarly journals Analytical review of models and methods for automatic recognition of gestures and sign languages

2021 ◽  
pp. 10-20
Author(s):  
Dmitry Ryumin ◽  
Ildar Kagirov ◽  
Alexander Axyonov ◽  
Alexey Karpov
Keyword(s):  
Sign Language ◽  
Language Translation ◽  
Automatic Recognition ◽  
Software Systems ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Sign Languages ◽  
Wide Range ◽  
Dynamic Gestures ◽  
Analytical Review

Introduction: Currently, the recognition of gestures and sign languages is one of the most intensively developing areas in computer vision and applied linguistics. The results of current investigations are applied in a wide range of areas, from sign language translation to gesture-based interfaces. In that regard, various systems and methods for the analysis of gestural data are being developed. Purpose: A detailed review of methods and a comparative analysis of current approaches in automatic recognition of gestures and sign languages. Results: The main gesture recognition problems are the following: detection of articulators (mainly hands), pose estimation and segmentation of gestures in the flow of speech. The authors conclude that the use of two-stream convolutional and recurrent neural network architectures is generally promising for efficient extraction and processing of spatial and temporal features, thus solving the problem of dynamic gestures and coarticulations. This solution, however, heavily depends on the quality and availability of data sets. Practical relevance: This review can be considered a contribution to the study of rapidly developing sign language recognition, irrespective to particular natural sign languages. The results of the work can be used in the development of software systems for automatic gesture and sign language recognition.

scholarly journals An evaluation of real-time requirements for automatic sign language recognition using ANNs and HMMs - The LIBRAS use case

2013 ◽  
Vol 4 (1) ◽  
pp. 1
Author(s):  
Ednaldo Brigante Pizzolato ◽  
Mauro dos Santos Anjo ◽  
Sebastian Feuerstack
Keyword(s):  
Real Time ◽  
Sign Language ◽  
Gesture Recognition ◽  
Large Set ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Sign Languages ◽  
Real Time System ◽  
Dynamic Gestures ◽  
Time Requirements

Sign languages are the natural way Deafs use to communicate with other people. They have their own formal semantic definitions and syntactic rules and are composed by a large set of gestures involving hands and head. Automatic recognition of sign languages (ARSL) tries to recognize the signs and translate them into a written language. ARSL is a challenging task as it involves background segmentation, hands and head posture modeling, recognition and tracking, temporal analysis and syntactic and semantic interpretation. Moreover, when real-time requirements are considered, this task becomes even more challenging. In this paper, we present a study of real time requirements of automatic sign language recognition of small sets of static and dynamic gestures of the Brazilian Sign Language (LIBRAS). For the task of static gesture recognition, we implemented a system that is able to work on small sub-sets of the alphabet - like A,E,I,O,U and B,C,F,L,V - reaching very high recognition rates. For the task of dynamic gesture recognition, we tested our system over a small set of LIBRAS words and collected the execution times. The aim was to gather knowledge regarding execution time of all the recognition processes (like segmentation, analysis and recognition itself) to evaluate the feasibility of building a real-time system to recognize small sets of both static and dynamic gestures. Our findings indicate that the bottleneck of our current architecture is the recognition phase.

scholarly journals Experimenting the Automatic Recognition of Non-Conventionalized Units in Sign Language

Algorithms ◽  
2020 ◽  
Vol 13 (12) ◽  
pp. 310
Author(s):  
Valentin Belissen ◽  
Annelies Braffort ◽  
Michèle Gouiffès
Keyword(s):  
Neural Network ◽  
Parallel Processing ◽  
Convolutional Neural Network ◽  
Sign Language ◽  
Performance Metrics ◽  
Automatic Recognition ◽  
Upper Body ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Sign Languages

Sign Languages (SLs) are visual–gestural languages that have developed naturally in deaf communities. They are based on the use of lexical signs, that is, conventionalized units, as well as highly iconic structures, i.e., when the form of an utterance and the meaning it carries are not independent. Although most research in automatic Sign Language Recognition (SLR) has focused on lexical signs, we wish to broaden this perspective and consider the recognition of non-conventionalized iconic and syntactic elements. We propose the use of corpora made by linguists like the finely and consistently annotated dialogue corpus Dicta-Sign-LSF-v2. We then redefined the problem of automatic SLR as the recognition of linguistic descriptors, with carefully thought out performance metrics. Moreover, we developed a compact and generalizable representation of signers in videos by parallel processing of the hands, face and upper body, then an adapted learning architecture based on a Recurrent Convolutional Neural Network (RCNN). Through a study focused on the recognition of four linguistic descriptors, we show the soundness of the proposed approach and pave the way for a wider understanding of Continuous Sign Language Recognition (CSLR).

scholarly journals Smart Communication Assistant for Deaf and Dumb People

2021 ◽  
Vol 9 (VII) ◽  
pp. 1358-1360
Author(s):  
Ajabe Harshada
Keyword(s):  
Sign Language ◽  
Recognition System ◽  
Research Area ◽  
Deaf People ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Sign Languages ◽  
Automatic Interpretation ◽  
Dynamic Gestures ◽  
Independent Life

Communication is the medium by which we can share our thoughts or convey the messages with other person. Nowadays we can give commands using voice recognition. But what if one absolutely cannot hear anything and eventually cannot speak. So the Sign Language is the main communicating tool for hearing impaired and mute people, and also to ensure an independent life for them, the automatic interpretation of sign language is an extensive research area. Sign language recognition (SLR) aims to interpret sign languages automatically by an application in order to help the deaf people to communicate with hearing society conveniently. Our aim is to design a system to help the Deaf and Dumb person to communicate with the rest of the world using sign language. With the use of image processing and artificial intelligence, many techniques and algorithms have been developed in this area. Every sign language recognition system is trained for recognizing the signs and converting them into required pattern. The proposed system aim to provide speech to speechless, in this paper we have introduced Sign Language Recognition using CNN for dynamic gestures to achieve faster results with high accuracy.

scholarly journals Indian Sign Language Recognition through Hybrid ConvNet-LSTM Networks

2021 ◽  
Vol 9 (1) ◽  
pp. 182-203
Author(s):  
Muthu Mariappan H ◽  
Dr Gomathi V
Keyword(s):  
Neural Network ◽  
Computer Vision ◽  
Real Time ◽  
Sign Language ◽  
Gesture Recognition ◽  
Language Translation ◽  
Video Gaming ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Indian Sign Language

Dynamic hand gesture recognition is a challenging task of Human-Computer Interaction (HCI) and Computer Vision. The potential application areas of gesture recognition include sign language translation, video gaming, video surveillance, robotics, and gesture-controlled home appliances. In the proposed research, gesture recognition is applied to recognize sign language words from real-time videos. Classifying the actions from video sequences requires both spatial and temporal features. The proposed system handles the former by the Convolutional Neural Network (CNN), which is the core of several computer vision solutions and the latter by the Recurrent Neural Network (RNN), which is more efficient in handling the sequences of movements. Thus, the real-time Indian sign language (ISL) recognition system is developed using the hybrid CNN-RNN architecture. The system is trained with the proposed CasTalk-ISL dataset. The ultimate purpose of the presented research is to deploy a real-time sign language translator to break the hurdles present in the communication between hearing-impaired people and normal people. The developed system achieves 95.99% top-1 accuracy and 99.46% top-3 accuracy on the test dataset. The obtained results outperform the existing approaches using various deep models on different datasets.

scholarly journals Sign Language Recognition using Hybrid Neural Networks

2019 ◽  
Vol 9 (2) ◽  
pp. 1092-1098
Keyword(s):  
Neural Networks ◽  
Genetic Algorithms ◽  
Artificial Neural Networks ◽  
Sign Language ◽  
Back Propagation ◽  
Optimization Techniques ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Sign Languages ◽  
Artificial Neural

Language has a prime role in communication between persons, in learning, in distribution of concepts and in preserving public contacts. The hearing-impaired have to challenge communication obstacles in a mostly hearing-capable culture. There are hundreds Sign Languages that are used all around the world today .The Sign Languages are established depending on the country and area of the deaf public. The aim of sign language recognition is to offer an effectual and correct tool to transcribe hand gesture into text. It can play a vital role in the communiqué between deaf and hearing society. Sign language recognition (SLR), as one of the significant research fields of human–computer interaction (HCI), has produced more and more interest in HCI society. Since, artificial neural networks are best suited for automated pattern recognition problems; they are used as a classification tool for this research. Back propagation is the most important algorithm for training neural networks. But, it easily gets trapped in local minima leading to inaccurate solutions. Therefore, some global search and optimization techniques were required to hybridize with artificial neural networks. One such technique is Genetic algorithms that imitate the principle of natural evolution. So, in this article, a hybrid intelligent system is proposed for sign language recognition in which artificial neural networks are merged with genetic algorithms. Results show that proposed hybrid model outperformed the existing back propagation based system.

scholarly journals Sign Language Recognition-A Survey of Techniques

2020 ◽  
Vol 9 (2) ◽  
pp. 854-857
Keyword(s):  
Sign Language ◽  
Recognition System ◽  
Spoken Language ◽  
Hearing Impaired ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Sign Languages ◽  
Data Glove ◽  
Depth Sensors ◽  
Impaired People

Sign language is the only method of communication for the hearing and speech impaired people around the world. Most of the speech and hearing-impaired people understand single sign language. Thus, there is an increasing demand for sign language interpreters. For regular people learning sign language is difficult, and for speech and hearing-impaired person, learning spoken language is impossible. There is a lot of research being done in the domain of automatic sign language recognition. Different methods such as, computer vision, data glove, depth sensors can be used to train a computer to interpret sign language. The interpretation is being done from sign to text, text to sign, speech to sign and sign to speech. Different countries use different sign languages, the signers of different sign languages are unable to communicate with each other. Analyzing the characteristic features of gestures provides insights about the sign language, some common features in sign languages gestures will help in designing a sign language recognition system. This type of system will help in reducing the communication gap between sign language users and spoken language users.

scholarly journals Vision-based Pakistani Sign Language Recognition Using Bag-of-Words and Support Vector Machines

2022 ◽  
Author(s):  
Muhammad Shaheer Mirza ◽  
Sheikh Muhammad Munaf ◽  
Shahid Ali ◽  
Fahad Azim ◽  
Saad Jawaid Khan
Keyword(s):  
Sign Language ◽  
Recognition System ◽  
Support Vector ◽  
Bag Of Words ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Sign Languages ◽  
Vector Machines ◽  
Rgb Images ◽  
Native Signers

Abstract In order to perform their daily activities, a person is required to communicating with others. This can be a major obstacle for the deaf population of the world, who communicate using sign languages (SL). Pakistani Sign Language (PSL) is used by more than 250,000 deaf Pakistanis. Developing a SL recognition system would greatly facilitate these people. This study aimed to collect data of static and dynamic PSL alphabets and to develop a vision-based system for their recognition using Bag-of-Words (BoW) and Support Vector Machine (SVM) techniques. A total of 5,120 images for 36 static PSL alphabet signs and 353 videos with 45,224 frames for 3 dynamic PSL alphabet signs were collected from 10 native signers of PSL. The developed system used the collected data as input, resized the data to various scales and converted the RGB images into grayscale. The resized grayscale images were segmented using Thresholding technique and features were extracted using Speeded Up Robust Feature (SURF). The obtained SURF descriptors were clustered using K-means clustering. A BoW was obtained by computing the Euclidean distance between the SURF descriptors and the clustered data. The codebooks were divided into training and testing using 5-fold cross validation. The highest overall classification accuracy for static PSL signs was 97.80% at 750×750 image dimensions and 500 Bags. For dynamic PSL signs a 96.53% accuracy was obtained at 480×270 video resolution and 200 Bags.

scholarly journals Conductive Porous MXene for Bionic, Wearable, and Precise Gesture Motion Sensors

Research ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-11
Author(s):  
Shengshun Duan ◽  
Yucheng Lin ◽  
Zhehan Wang ◽  
Junyi Tang ◽  
Yinhui Li ◽  
...  
Keyword(s):  
Sign Language ◽  
Detection Threshold ◽  
Motion Sensors ◽  
Joint Movement ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Cyclic Movements ◽  
Wide Range ◽  
Movement Monitoring ◽  
Slit Sensilla

Reliable, wide range, and highly sensitive joint movement monitoring is essential for training activities, human behavior analysis, and human-machine interfaces. Yet, most current motion sensors work on the nano/microcracks induced by the tensile deformation on the convex surface of joints during joint movements, which cannot satisfy requirements of ultrawide detectable angle range, high angle sensitivity, conformability, and consistence under cyclic movements. In nature, scorpions sense small vibrations by allowing for compression strain conversion from external mechanical vibrations through crack-shaped slit sensilla. Here, we demonstrated that ultraconformal sensors based on controlled slit structures, inspired by the geometry of a scorpion’s slit sensilla, exhibit high sensitivity (0.45%deg-1), ultralow angle detection threshold (~15°), fast response/relaxation times (115/72 ms), wide range (15° ~120°), and durability (over 1000 cycles). Also, a user-friendly, hybrid sign language system has been developed to realize Chinese and American sign language recognition and feedback through video and speech broadcasts, making these conformal motion sensors promising candidates for joint movement monitoring in wearable electronics and robotics technology.

Sign Language Recognition Using Image Processing

2017 ◽  
Vol 7 (8) ◽  
pp. 142 ◽  
Author(s):  
Kamal Preet Kour ◽  
Lini Mathew
Keyword(s):  
Image Processing ◽  
Sign Language ◽  
Feature Detection ◽  
Hand Gesture ◽  
Normal Person ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Major Drawback ◽  
Sign Languages ◽  
System Feature

One of the major drawback of our society is the barrier that is created between disabled or handicapped persons and the normal person. Communication is the only medium by which we can share our thoughts or convey the message but for a person with disability (deaf and dumb) faces difficulty in communication with normal person. For many deaf and dumb people , sign language is the basic means of communication. Sign language recognition (SLR) aims to interpret sign languages automatically by a computer in order to help the deaf communicate with hearing society conveniently.  Our aim is to design a system  to help the person who trained the hearing impaired to communicate with the rest of the world using sign language or hand gesture recognition techniques. In this system, feature detection and feature extraction of hand gesture is done with the help of SURF algorithm using image processing. All this work is done using MATLAB software. With the help of this algorithm, a person can easily trained a deaf and dumb.

scholarly journals Sign Language Recognition System Using Cnn

2020 ◽  
pp. 1-6
Author(s):  
I-Te Chen ◽  
◽  
Rung Shiang ◽  
Hung-Yuan Huang ◽  
◽  
...  
Keyword(s):  
Sign Language ◽  
Language Translation ◽  
Recognition Rate ◽  
Training Model ◽  
Recognition System ◽  
Read Aloud ◽  
System Response ◽  
Translation System ◽  
Language Recognition ◽  
Sign Language Recognition

In this study, we have built an automatic sign language translation system for deaf and dumb persons to communicate with ordinary people. According to the Statistics Department of the Taiwan Ministry of Health and Welfare, there are 119,682 hearing impaired persons, and 14,831 voice function or language dysfunctions. The Deaf and dumb persons’ account for 11.7% of the population with physical and mental disabilities. However, there are only 488 qualified people with the sign language translation skill certificate, which shows the importance of automatic sign language translation systems. This system collects 11 signals including five fingers’ curvature, 3-axis gyroscope and 3-axis accelerometer from left and right hand separately. In addition, a total of 22 signals are collected by the two sensors, Flex sensor and GY-521 six-axis with single-board computer Arduino MEGA 2560; and then uploaded to server via ESP-01S Wi-Fi module. While server receives the 22 signals, it converts to a RGB picture using PHP program. As a result, we can compare the picture with the model trained by TensorFlow and the compared result is stored in the database. Meanwhile, the comparison stored in database which can be accessed by APP programs would be displayed on the screen of the mobile device and be read aloud. The TensorFlow training model collects 25 sign language gestures, each based on 100 training gesture pictures, and a sign language recognition training model is Convolutional Neural Network (CNN). In this study, the results of the sign language recognition training model are further confirmed by 10 people other than those in training database. So far, the indeed recognition rate of sign language is about 84.4%, and the system response time is about 2.243 seconds.

Sign in / Sign up

Export Citation Format

Share Document