Effect of Sign-recognition Performance on the Usability of Sign-language Dictionary Search

2021 ◽  
Vol 14 (4) ◽  
pp. 1-33
Author(s):  
Saad Hassan ◽  
Oliver Alonzo ◽  
Abraham Glasser ◽  
Matt Huenerfauth
Keyword(s):  
Sign Language ◽  
Recognition Performance ◽  
The Other ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Search System ◽  
Wizard Of Oz ◽  
Sign Recognition ◽  
Technology Users ◽  
The Relationship

Advances in sign-language recognition technology have enabled researchers to investigate various methods that can assist users in searching for an unfamiliar sign in ASL using sign-recognition technology. Users can generate a query by submitting a video of themselves performing the sign they believe they encountered somewhere and obtain a list of possible matches. However, there is disagreement among developers of such technology on how to report the performance of their systems, and prior research has not examined the relationship between the performance of search technology and users’ subjective judgements for this task. We conducted three studies using a Wizard-of-Oz prototype of a webcam-based ASL dictionary search system to investigate the relationship between the performance of such a system and user judgements. We found that, in addition to the position of the desired word in a list of results, the placement of the desired word above or below the fold and the similarity of the other words in the results list affected users’ judgements of the system. We also found that metrics that incorporate the precision of the overall list correlated better with users’ judgements than did metrics currently reported in prior ASL dictionary research.

Technological Aids for Deaf and Mute in Modern World

2020 ◽  
Vol 14 ◽  
Author(s):  
Vasu Mehra ◽  
Dhiraj Pandey ◽  
Aayush Rastogi ◽  
Aditya Singh ◽  
Harsh Preet Singh
Keyword(s):  
Sign Language ◽  
Gesture Recognition ◽  
Recognition System ◽  
Modern World ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Background Elimination ◽  
Sign Recognition ◽  
Technological Advances ◽  
Better Than

Background:: People suffering from hearing and speaking disabilities have a few ways of communicating with other people. One of these is to communicate through the use of sign language. Objective:: Developing a system for sign language recognition becomes essential for deaf as well as a mute person. The recognition system acts as a translator between a disabled and an able person. This eliminates the hindrances in exchange of ideas. Most of the existing systems are very poorly designed with limited support for the needs of their day to day facilities. Methods:: The proposed system embedded with gesture recognition capability has been introduced here which extracts signs from a video sequence and displays them on screen. On the other hand, a speech to text as well as text to speech system is also introduced to further facilitate the grieved people. To get the best out of human computer relationship, the proposed solution consists of various cutting-edge technologies and Machine Learning based sign recognition models which have been trained by using Tensor Flow and Keras library. Result:: The proposed architecture works better than several gesture recognition techniques like background elimination and conversion to HSV because of sharply defined image provided to the model for classification. The results of testing indicate reliable recognition systems with high accuracy that includes most of the essential and necessary features for any deaf and dumb person in his/her day to day tasks. Conclusion:: It’s the need of current technological advances to develop reliable solutions which can be deployed to assist deaf and dumb people to adjust to normal life. Instead of focusing on a standalone technology, a plethora of them have been introduced in this proposed work. Proposed Sign Recognition System is based on feature extraction and classification. The trained model helps in identification of different gestures.

scholarly journals Global-Local Enhancement Network for NMF-Aware Sign Language Recognition

2021 ◽  
Vol 17 (3) ◽  
pp. 1-19
Author(s):  
Hezhen Hu ◽  
Wengang Zhou ◽  
Junfu Pu ◽  
Houqiang Li
Keyword(s):  
Sign Language ◽  
Daily Life ◽  
The Other ◽  
Challenging Problem ◽  
Language Recognition ◽  
Local Enhancement ◽  
Sign Language Recognition ◽  
Vocabulary Size ◽  
Hand Gestures ◽  
Fine Grained

Sign language recognition (SLR) is a challenging problem, involving complex manual features (i.e., hand gestures) and fine-grained non-manual features (NMFs) (i.e., facial expression, mouth shapes, etc .). Although manual features are dominant, non-manual features also play an important role in the expression of a sign word. Specifically, many sign words convey different meanings due to non-manual features, even though they share the same hand gestures. This ambiguity introduces great challenges in the recognition of sign words. To tackle the above issue, we propose a simple yet effective architecture called Global-Local Enhancement Network (GLE-Net), including two mutually promoted streams toward different crucial aspects of SLR. Of the two streams, one captures the global contextual relationship, while the other stream captures the discriminative fine-grained cues. Moreover, due to the lack of datasets explicitly focusing on this kind of feature, we introduce the first non-manual-feature-aware isolated Chinese sign language dataset (NMFs-CSL) with a total vocabulary size of 1,067 sign words in daily life. Extensive experiments on NMFs-CSL and SLR500 datasets demonstrate the effectiveness of our method.

scholarly journals A Novel Communication System For Deaf And Dumb People using gesture

2020 ◽  
Vol 32 ◽  
pp. 02003
Author(s):  
Pritesh Ambavane ◽  
Rahul Karjavkar ◽  
Hemant Pathare ◽  
Shubham Relekar ◽  
Bhavana Alte ◽  
...  
Keyword(s):  
Sign Language ◽  
The Other ◽  
Hand Gesture ◽  
Language Recognition ◽  
Text To Speech ◽  
Sign Language Recognition ◽  
Digital Conversion ◽  
Analog To Digital ◽  
Analog To Digital Conversion ◽  
The World

Human Beings know each other and contact with themselves through thoughts and ideas.The best way to present our idea is through speech. Some people don’t have the power of speech; the only way they communicate with others is through sign language. Now a days technology has reduced the gap through systems which can be used to change the sign language used by these people to speech. Sign language recognition (SLR) and gesture-based control are two major applications used for hand gesture recognition technologies. On the other side the controller converts the sign language in to the text and speech which gets converted with the help of text to speech conversion and analog to digital conversion. A Dumb person throughout the world uses sign language for the communication.The best way to present our idea is through speech. Some people don’t have the power of speech; the only way they communicate with others is through sign language. Now a days technology has reduced the gap through systems which can be used to change the sign language used by these people to speech. Sign language recognition (SLR) and gesture-based control are two major applications used for hand gesture recognition technologies. On the other side the controller converts the sign language in to the text and speech which gets converted with the help of text to speech conversion and analog to digital conversion. A Dumb person throughout the world uses sign language for the communication.

SIGN LANGUAGE RECOGNITION BASED ON HMM/ANN/DP

2000 ◽  
Vol 14 (05) ◽  
pp. 587-602 ◽  
Author(s):  
WEN GAO ◽  
JIYONG MA ◽  
JIANGQIN WU ◽  
CHUNLI WANG
Keyword(s):  
Sign Language ◽  
Data Stream ◽  
Recognition Performance ◽  
Combined Approach ◽  
Language Recognition ◽  
Geometrical Analysis ◽  
Sign Language Recognition ◽  
Design And Implementation ◽  
Invariant Feature ◽  
New Ideas

In this paper, a system designed for helping the deaf to communicate with others is presented. Some useful new ideas are proposed in design and implementation. An algorithm based on geometrical analysis for the purpose of extracting invariant feature to signer position is presented. An ANN–DP combined approach is employed for segmenting subwords automatically from the data stream of sign signals. To tackle the epenthesis movement problem, a DP-based method has been used to obtain the context-dependent models. Some techniques for system implementation are also given, including fast matching, frame prediction and search algorithms. The implemented system is able to recognize continuous large vocabulary Chinese Sign Language. Experiments show that proposed techniques in this paper are efficient on either recognition speed or recognition performance.

scholarly journals Signer-Independent Sign Language Recognition with Adversarial Neural Networks

2021 ◽  
Vol 11 (2) ◽  
pp. 121-129
Author(s):  
Pedro M. Ferreira ◽  
◽  
Diogo Pernes ◽  
Ana Rebelo ◽  
Jaime S. Cardoso
Keyword(s):  
Sign Language ◽  
Research Field ◽  
Training Procedure ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Learning Stage ◽  
Sign Recognition ◽  
Proposed Model ◽  
Adversarial Training ◽  
Latent Representations

Sign Language Recognition (SLR) has become an appealing topic in modern societies because such technology can ideally be used to bridge the gap between deaf and hearing people. Although important steps have been made towards the development of real-world SLR systems, signer-independent SLR is still one of the bottleneck problems of this research field. In this regard, we propose a deep neural network along with an adversarial training objective, specifically designed to address the signer-independent problem. Specifically, the proposed model consists of an encoder, mapping from input images to latent representations, and two classifiers operating on these underlying representations: (i) the sign-classifier, for predicting the class/sign labels, and (ii) the signer-classifier, for predicting their signer identities. During the learning stage, the encoder is simultaneously trained to help the sign-classifier as much as possible while trying to fool the signer-classifier. This adversarial training procedure allows learning signer-invariant latent representations that are in fact highly discriminative for sign recognition. Experimental results demonstrate the effectiveness of the proposed model and its capability of dealing with the large inter-signer variations.

scholarly journals EM-Sign: A Non-Contact Recognition Method Based on 24 GHz Doppler Radar for Continuous Signs and Dialogues

Electronics ◽  
2020 ◽  
Vol 9 (10) ◽  
pp. 1577 ◽  
Author(s):  
Linting Ye ◽  
Shengchang Lan ◽  
Kang Zhang ◽  
Guiyuan Zhang
Keyword(s):  
American Sign Language ◽  
Sign Language ◽  
Doppler Radar ◽  
Alternative Form ◽  
Language Recognition ◽  
Training Process ◽  
Sign Language Recognition ◽  
Radar Sensors ◽  
Sign Recognition ◽  
Mode Decomposition

We studied continuous sign language recognition using Doppler radar sensors. Four signs in Chinese sign language and American sign language were captured and extracted by complex empirical mode decomposition (CEMD) to obtain spectrograms. Image sharpening was used to enhance the micro-Doppler signatures of the signs. To classify the different signs, we utilized an improved Yolov3-tiny network by replacing the framework with ResNet and fine-tuned the network in advance. This method can remove the epentheses from the training process. Experimental results revealed that the proposed method can surpass the state-of-the-art sign language recognition methods in continuous sign recognition with a precision of 0.924, a recall of 0.993, an F1-measure of 0.957 and a mean average precision (mAP) of 0.99. In addition, dialogue recognition in three daily conversation scenarios was performed and evaluated. The average word error rate (WER) was 0.235, 10% lower than in of other works. Our work provides an alternative form of sign language recognition and a new approach to simplify the training process and achieve a better continuous sign language recognition effect.

scholarly journals Bangla Sign Language (BdSL) Alphabets and Numerals Classification Using a Deep Learning Model

Sensors ◽  
2022 ◽  
Vol 22 (2) ◽  
pp. 574
Author(s):  
Kanchon Kanti Podder ◽  
Muhammad E. H. Chowdhury ◽  
Anas M. Tahir ◽  
Zaid Bin Mahbub ◽  
Amith Khandakar ◽  
...  
Keyword(s):  
Deep Learning ◽  
Sign Language ◽  
Image Data ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Hand Detection ◽  
Hand Orientation ◽  
Sign Recognition ◽  
Language Interpretation ◽  
Segmentation Approach

A real-time Bangla Sign Language interpreter can enable more than 200 k hearing and speech-impaired people to the mainstream workforce in Bangladesh. Bangla Sign Language (BdSL) recognition and detection is a challenging topic in computer vision and deep learning research because sign language recognition accuracy may vary on the skin tone, hand orientation, and background. This research has used deep machine learning models for accurate and reliable BdSL Alphabets and Numerals using two well-suited and robust datasets. The dataset prepared in this study comprises of the largest image database for BdSL Alphabets and Numerals in order to reduce inter-class similarity while dealing with diverse image data, which comprises various backgrounds and skin tones. The papers compared classification with and without background images to determine the best working model for BdSL Alphabets and Numerals interpretation. The CNN model trained with the images that had a background was found to be more effective than without background. The hand detection portion in the segmentation approach must be more accurate in the hand detection process to boost the overall accuracy in the sign recognition. It was found that ResNet18 performed best with 99.99% accuracy, precision, F1 score, sensitivity, and 100% specificity, which outperforms the works in the literature for BdSL Alphabets and Numerals recognition. This dataset is made publicly available for researchers to support and encourage further research on Bangla Sign Language Interpretation so that the hearing and speech-impaired individuals can benefit from this research.

scholarly journals Sign language recognition using Kinect sensor based on color stream and skeleton points

2021 ◽  
Vol 47 (2) ◽  
pp. 769-778
Author(s):  
Isack Bulugu
Keyword(s):  
Sign Language ◽  
Recognition Accuracy ◽  
Learning Algorithm ◽  
Recognition System ◽  
Kinect Sensor ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Sign Recognition ◽  
Average Accuracy ◽  
High Recognition Accuracy

This paper presents a sign language recognition system based on color stream and skeleton points. Several approaches have been established to address sign language recognition problems. However, most of the previous approaches still have poor recognition accuracy. The proposed approach uses Kinect sensor based on color stream and skeleton points from the depth stream to improved recognition accuracy. Techniques within this approach use hand trajectories and hand shapes in combating sign recognition challenges. Therefore, for a particular sign a representative feature vector is extracted, which consists of hand trajectories and hand shapes. A sparse dictionary learning algorithm, Label Consistent K-SVD (LC-KSVD) is applied to obtain a discriminative dictionary. Based on that, the system was further developed to a new classification approach for better results. The proposed system was fairly evaluated based on 21 sign words including one-handed signs and two-handed signs. It was observed that the proposed system gets high recognition accuracy of 98.25%, and obtained an average accuracy of 95.34% for signer independent recognition. Keywords: Sign language, Color stream, Skeleton points, Kinect sensor, Discriminative dictionary.

Sign in / Sign up

Export Citation Format

Share Document