Intelligent Malaysian Sign Language Translation System Using Convolutional-Based Attention Module with Residual Network

The deaf-mutes population always feels helpless when they are not understood by others and vice versa. This is a big humanitarian problem and needs localised solution. To solve this problem, this study implements a convolutional neural network (CNN), convolutional-based attention module (CBAM) to recognise Malaysian Sign Language (MSL) from images. Two different experiments were conducted for MSL signs, using CBAM-2DResNet (2-Dimensional Residual Network) implementing “Within Blocks” and “Before Classifier” methods. Various metrics such as the accuracy, loss, precision, recall, F1-score, confusion matrix, and training time are recorded to evaluate the models’ efficiency. The experimental results showed that CBAM-ResNet models achieved a good performance in MSL signs recognition tasks, with accuracy rates of over 90% through a little of variations. The CBAM-ResNet “Before Classifier” models are more efficient than “Within Blocks” CBAM-ResNet models. Thus, the best trained model of CBAM-2DResNet is chosen to develop a real-time sign recognition system for translating from sign language to text and from text to sign language in an easy way of communication between deaf-mutes and other people. All experiment results indicated that the “Before Classifier” of CBAMResNet models is more efficient in recognising MSL and it is worth for future research.

Download Full-text

Sign Language Recognition System Using Cnn

10.47363/jmhc/2020(2)138 ◽

2020 ◽

pp. 1-6

Author(s):

I-Te Chen ◽

◽

Rung Shiang ◽

Hung-Yuan Huang ◽

◽

...

Keyword(s):

Sign Language ◽

Language Translation ◽

Recognition Rate ◽

Training Model ◽

Recognition System ◽

Read Aloud ◽

System Response ◽

Translation System ◽

Language Recognition ◽

Sign Language Recognition

In this study, we have built an automatic sign language translation system for deaf and dumb persons to communicate with ordinary people. According to the Statistics Department of the Taiwan Ministry of Health and Welfare, there are 119,682 hearing impaired persons, and 14,831 voice function or language dysfunctions. The Deaf and dumb persons’ account for 11.7% of the population with physical and mental disabilities. However, there are only 488 qualified people with the sign language translation skill certificate, which shows the importance of automatic sign language translation systems. This system collects 11 signals including five fingers’ curvature, 3-axis gyroscope and 3-axis accelerometer from left and right hand separately. In addition, a total of 22 signals are collected by the two sensors, Flex sensor and GY-521 six-axis with single-board computer Arduino MEGA 2560; and then uploaded to server via ESP-01S Wi-Fi module. While server receives the 22 signals, it converts to a RGB picture using PHP program. As a result, we can compare the picture with the model trained by TensorFlow and the compared result is stored in the database. Meanwhile, the comparison stored in database which can be accessed by APP programs would be displayed on the screen of the mobile device and be read aloud. The TensorFlow training model collects 25 sign language gestures, each based on 100 training gesture pictures, and a sign language recognition training model is Convolutional Neural Network (CNN). In this study, the results of the sign language recognition training model are further confirmed by 10 people other than those in training database. So far, the indeed recognition rate of sign language is about 84.4%, and the system response time is about 2.243 seconds.

Download Full-text

Technological Aids for Deaf and Mute in Modern World

Recent Patents on Engineering ◽

10.2174/1872212114999201116214802 ◽

2020 ◽

Vol 14 ◽

Author(s):

Vasu Mehra ◽

Dhiraj Pandey ◽

Aayush Rastogi ◽

Aditya Singh ◽

Harsh Preet Singh

Keyword(s):

Sign Language ◽

Gesture Recognition ◽

Recognition System ◽

Modern World ◽

Language Recognition ◽

Sign Language Recognition ◽

Background Elimination ◽

Sign Recognition ◽

Technological Advances ◽

Better Than

Background:: People suffering from hearing and speaking disabilities have a few ways of communicating with other people. One of these is to communicate through the use of sign language. Objective:: Developing a system for sign language recognition becomes essential for deaf as well as a mute person. The recognition system acts as a translator between a disabled and an able person. This eliminates the hindrances in exchange of ideas. Most of the existing systems are very poorly designed with limited support for the needs of their day to day facilities. Methods:: The proposed system embedded with gesture recognition capability has been introduced here which extracts signs from a video sequence and displays them on screen. On the other hand, a speech to text as well as text to speech system is also introduced to further facilitate the grieved people. To get the best out of human computer relationship, the proposed solution consists of various cutting-edge technologies and Machine Learning based sign recognition models which have been trained by using Tensor Flow and Keras library. Result:: The proposed architecture works better than several gesture recognition techniques like background elimination and conversion to HSV because of sharply defined image provided to the model for classification. The results of testing indicate reliable recognition systems with high accuracy that includes most of the essential and necessary features for any deaf and dumb person in his/her day to day tasks. Conclusion:: It’s the need of current technological advances to develop reliable solutions which can be deployed to assist deaf and dumb people to adjust to normal life. Instead of focusing on a standalone technology, a plethora of them have been introduced in this proposed work. Proposed Sign Recognition System is based on feature extraction and classification. The trained model helps in identification of different gestures.

Download Full-text

Spoken Language to Sign Language Translation System Based on HamNoSys

Proceedings of the 2019 International Symposium on Signal Processing Systems - SSPS 2019 ◽

10.1145/3364908.3365300 ◽

2019 ◽

Author(s):

Zhehan Kang

Keyword(s):

Sign Language ◽

Language Translation ◽

Spoken Language ◽

Translation System

Download Full-text

Neural Sign Language Translation Based on Human Keypoint Estimation

Applied Sciences ◽

10.3390/app9132683 ◽

2019 ◽

Vol 9 (13) ◽

pp. 2683 ◽

Cited By ~ 12

Author(s):

Sang-Ki Ko ◽

Chang Jo Kim ◽

Hyedong Jung ◽

Choongsang Cho

Keyword(s):

Neural Network ◽

Sign Language ◽

Language Translation ◽

Network Models ◽

Training Data ◽

Translation System ◽

Body Parts ◽

Neural Network Models ◽

Translation Model ◽

Starting Point

We propose a sign language translation system based on human keypoint estimation. It is well-known that many problems in the field of computer vision require a massive dataset to train deep neural network models. The situation is even worse when it comes to the sign language translation problem as it is far more difficult to collect high-quality training data. In this paper, we introduce the KETI (Korea Electronics Technology Institute) sign language dataset, which consists of 14,672 videos of high resolution and quality. Considering the fact that each country has a different and unique sign language, the KETI sign language dataset can be the starting point for further research on the Korean sign language translation. Using the KETI sign language dataset, we develop a neural network model for translating sign videos into natural language sentences by utilizing the human keypoints extracted from the face, hands, and body parts. The obtained human keypoint vector is normalized by the mean and standard deviation of the keypoints and used as input to our translation model based on the sequence-to-sequence architecture. As a result, we show that our approach is robust even when the size of the training data is not sufficient. Our translation model achieved 93.28% (55.28%, respectively) translation accuracy on the validation set (test set, respectively) for 105 sentences that can be used in emergency situations. We compared several types of our neural sign translation models based on different attention mechanisms in terms of classical metrics for measuring the translation performance.

Download Full-text