Real-Time Hand Gesture Recognition: A Long Short-Term Memory Approach with Electromyography

Abstract Recent developments of sensors that allow tracking of human movements and gestures enable rapid progress of applications in domains like medical rehabilitation or robotic control. Especially the inertial measurement unit (IMU) is an excellent device for real-time scenarios as it rapidly delivers data input. Therefore, a computational model must be able to learn gesture sequences in a fast yet robust way. We recently introduced an echo state network (ESN) framework for continuous gesture recognition (Tietz et al., 2019) including novel approaches for gesture spotting, i.e., the automatic detection of the start and end phase of a gesture. Although our results showed good classification performance, we identified significant factors which also negatively impact the performance like subgestures and gesture variability. To address these issues, we include experiments with Long Short-Term Memory (LSTM) networks, which is a state-of-the-art model for sequence processing, to compare the obtained results with our framework and to evaluate their robustness regarding pitfalls in the recognition process. In this study, we analyze the two conceptually different approaches processing continuous, variable-length gesture sequences, which shows interesting results comparing the distinct gesture accomplishments. In addition, our results demonstrate that our ESN framework achieves comparably good performance as the LSTM network but has significantly lower training times. We conclude from the present work that ESNs are viable models for continuous gesture recognition delivering reasonable performance for applications requiring real-time performance as in robotic or rehabilitation tasks. From our discussion of this comparative study, we suggest prospective improvements on both the experimental and network architecture level.

Download Full-text

Hand Gesture Recognition in Video Sequences Using Deep Convolutional and Recurrent Neural Networks

Applied Computer Systems ◽

10.2478/acss-2020-0007 ◽

2020 ◽

Vol 25 (1) ◽

pp. 57-61

Author(s):

Falah Obaid ◽

Amin Babadi ◽

Ahmad Yoosofan

Keyword(s):

Neural Network ◽

Neural Networks ◽

Deep Learning ◽

Gesture Recognition ◽

Recurrent Neural Network ◽

Short Term Memory ◽

Hand Gesture ◽

Short Term ◽

Term Memory ◽

Long Short Term Memory

AbstractDeep learning is a new branch of machine learning, which is widely used by researchers in a lot of artificial intelligence applications, including signal processing and computer vision. The present research investigates the use of deep learning to solve the hand gesture recognition (HGR) problem and proposes two models using deep learning architecture. The first model comprises a convolutional neural network (CNN) and a recurrent neural network with a long short-term memory (RNN-LSTM). The accuracy of model achieves up to 82 % when fed by colour channel, and 89 % when fed by depth channel. The second model comprises two parallel convolutional neural networks, which are merged by a merge layer, and a recurrent neural network with a long short-term memory fed by RGB-D. The accuracy of the latest model achieves up to 93 %.

Download Full-text

Real-Time Anomaly Detection in Distribution Grids Using Long Short Term Memory Network

10.1109/epec52095.2021.9621640 ◽

2021 ◽

Author(s):

Ming Zhou ◽

Petr Musilek

Keyword(s):

Anomaly Detection ◽

Real Time ◽

Short Term Memory ◽

Short Term ◽

Term Memory ◽

Memory Network ◽

Long Short Term Memory ◽

Distribution Grids

Download Full-text

Classification and forecasting of real-time server traffic flows employing long short-term memory for hybrid E/O data center networks

Journal of Optical Communications and Networking ◽

10.1364/jocn.411017 ◽

2021 ◽

Vol 13 (5) ◽

pp. 85

Author(s):

Mihail Balanici ◽

Stephan Pachnicke

Keyword(s):

Real Time ◽

Data Center ◽

Short Term Memory ◽

Traffic Flows ◽

Data Center Networks ◽

Short Term ◽

Term Memory ◽

Long Short Term Memory

Download Full-text

Decoding Movements from Cortical Ensemble Activity Using a Long Short-Term Memory Recurrent Network

Neural Computation ◽

10.1162/neco_a_01189 ◽

2019 ◽

Vol 31 (6) ◽

pp. 1085-1113 ◽

Cited By ~ 5

Author(s):

Po-He Tseng ◽

Núria Armengol Urpi ◽

Mikhail Lebedev ◽

Miguel Nicolelis

Keyword(s):

Real Time ◽

Short Term Memory ◽

Unscented Kalman Filter ◽

Bipedal Walking ◽

Motor Tasks ◽

Short Term ◽

Term Memory ◽

Cortical Areas ◽

Long Short Term Memory ◽

Ensemble Activity

Although many real-time neural decoding algorithms have been proposed for brain-machine interface (BMI) applications over the years, an optimal, consensual approach remains elusive. Recent advances in deep learning algorithms provide new opportunities for improving the design of BMI decoders, including the use of recurrent artificial neural networks to decode neuronal ensemble activity in real time. Here, we developed a long-short term memory (LSTM) decoder for extracting movement kinematics from the activity of large ( N = 134–402) populations of neurons, sampled simultaneously from multiple cortical areas, in rhesus monkeys performing motor tasks. Recorded regions included primary motor, dorsal premotor, supplementary motor, and primary somatosensory cortical areas. The LSTM's capacity to retain information for extended periods of time enabled accurate decoding for tasks that required both movements and periods of immobility. Our LSTM algorithm significantly outperformed the state-of-the-art unscented Kalman filter when applied to three tasks: center-out arm reaching, bimanual reaching, and bipedal walking on a treadmill. Notably, LSTM units exhibited a variety of well-known physiological features of cortical neuronal activity, such as directional tuning and neuronal dynamics across task epochs. LSTM modeled several key physiological attributes of cortical circuits involved in motor tasks. These findings suggest that LSTM-based approaches could yield a better algorithm strategy for neuroprostheses that employ BMIs to restore movement in severely disabled patients.

Download Full-text