3D Hand Pose Estimation Based on Five-Layer Ensemble CNN

Estimating accurate 3D hand pose from a single RGB image is a highly challenging problem in pose estimation due to self-geometric ambiguities, self-occlusions, and the absence of depth information. To this end, a novel Five-Layer Ensemble CNN (5LENet) is proposed based on hierarchical thinking, which is designed to decompose the hand pose estimation task into five single-finger pose estimation sub-tasks. Then, the sub-task estimation results are fused to estimate full 3D hand pose. The hierarchical method is of great benefit to extract deeper and better finger feature information, which can effectively improve the estimation accuracy of 3D hand pose. In addition, we also build a hand model with the center of the palm (represented as Palm) connected to the middle finger according to the topological structure of hand, which can further boost the performance of 3D hand pose estimation. Additionally, extensive quantitative and qualitative results on two public datasets demonstrate the effectiveness of 5LENet, yielding new state-of-the-art 3D estimation accuracy, which is superior to most advanced estimation methods.

Download Full-text

CrossFuNet: RGB and Depth Cross-Fusion Network for Hand Pose Estimation

Sensors ◽

10.3390/s21186095 ◽

2021 ◽

Vol 21 (18) ◽

pp. 6095

Author(s):

Xiaojing Sun ◽

Bin Wang ◽

Longxiang Huang ◽

Qian Zhang ◽

Sulei Zhu ◽

...

Keyword(s):

Pose Estimation ◽

Depth Map ◽

Depth Information ◽

Feature Maps ◽

Hand Pose Estimation ◽

Depth Sensors ◽

Key Points ◽

Rgb Images ◽

Public Datasets ◽

Hand Pose

Despite recent successes in hand pose estimation from RGB images or depth maps, inherent challenges remain. RGB-based methods suffer from heavy self-occlusions and depth ambiguity. Depth sensors rely heavily on distance and can only be used indoors, thus there are many limitations to the practical application of depth-based methods. The aforementioned challenges have inspired us to combine the two modalities to offset the shortcomings of the other. In this paper, we propose a novel RGB and depth information fusion network to improve the accuracy of 3D hand pose estimation, which is called CrossFuNet. Specifically, the RGB image and the paired depth map are input into two different subnetworks, respectively. The feature maps are fused in the fusion module in which we propose a completely new approach to combine the information from the two modalities. Then, the common method is used to regress the 3D key-points by heatmaps. We validate our model on two public datasets and the results reveal that our model outperforms the state-of-the-art methods.

Download Full-text

Cascaded Hierarchical CNN for RGB-Based 3D Hand Pose Estimation

Mathematical Problems in Engineering ◽

10.1155/2020/8432840 ◽

2020 ◽

Vol 2020 ◽

pp. 1-13

Author(s):

Shiming Dai ◽

Wei Liu ◽

Wenji Yang ◽

Lili Fan ◽

Jihao Zhang

Keyword(s):

Pose Estimation ◽

Depth Image ◽

Estimation Methods ◽

Hierarchical Network ◽

Human Machine Interaction ◽

Depth Cameras ◽

Hand Pose Estimation ◽

Public Datasets ◽

Rgb Image ◽

Hand Pose

3D hand pose estimation can provide basic information about gestures, which has an important significance in the fields of Human-Machine Interaction (HMI) and Virtual Reality (VR). In recent years, 3D hand pose estimation from a single depth image has made great research achievements due to the development of depth cameras. However, 3D hand pose estimation from a single RGB image is still a highly challenging problem. In this work, we propose a novel four-stage cascaded hierarchical CNN (4CHNet), which leverages hierarchical network to decompose hand pose estimation into finger pose estimation and palm pose estimation, extracts separately finger features and palm features, and finally fuses them to estimate 3D hand pose. Compared with direct estimation methods, the hand feature information extracted by the hierarchical network is more representative. Furthermore, concatenating various stages of the network for end-to-end training can make each stage mutually beneficial and progress. The experimental results on two public datasets demonstrate that our 4CHNet can significantly improve the accuracy of 3D hand pose estimation from a single RGB image.

Download Full-text

A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods

Sensors ◽

10.3390/s20041074 ◽

2020 ◽

Vol 20 (4) ◽

pp. 1074 ◽

Cited By ~ 3

Author(s):

Weiya Chen ◽

Chenchen Yu ◽

Chenyu Tu ◽

Zehua Lyu ◽

Jing Tang ◽

...

Keyword(s):

Computer Vision ◽

Pose Estimation ◽

Wearable Sensors ◽

Complex Structure ◽

Estimation Methods ◽

Human Computer Interactions ◽

Hand Pose Estimation ◽

Timely Review ◽

Kinematic Models ◽

Hand Pose

Real-time sensing and modeling of the human body, especially the hands, is an important research endeavor for various applicative purposes such as in natural human computer interactions. Hand pose estimation is a big academic and technical challenge due to the complex structure and dexterous movement of human hands. Boosted by advancements from both hardware and artificial intelligence, various prototypes of data gloves and computer-vision-based methods have been proposed for accurate and rapid hand pose estimation in recent years. However, existing reviews either focused on data gloves or on vision methods or were even based on a particular type of camera, such as the depth camera. The purpose of this survey is to conduct a comprehensive and timely review of recent research advances in sensor-based hand pose estimation, including wearable and vision-based solutions. Hand kinematic models are firstly discussed. An in-depth review is conducted on data gloves and vision-based sensor systems with corresponding modeling methods. Particularly, this review also discusses deep-learning-based methods, which are very promising in hand pose estimation. Moreover, the advantages and drawbacks of the current hand gesture estimation methods, the applicative scope, and related challenges are also discussed.

Download Full-text

Real-Time Energy Efficient Hand Pose Estimation: A Case Study

Sensors ◽

10.3390/s20102828 ◽

2020 ◽

Vol 20 (10) ◽

pp. 2828

Author(s):

Mhd Rashed Al Koutayni ◽

Vladimir Rybalkin ◽

Jameel Malik ◽

Ahmed Elhayek ◽

Christian Weis ◽

...

Keyword(s):

Neural Network ◽

Real Time ◽

Pose Estimation ◽

Energy Efficient ◽

Graphics Processing Units ◽

Estimation Algorithm ◽

High Energy ◽

Estimation Methods ◽

Hand Pose Estimation ◽

Hand Pose

The estimation of human hand pose has become the basis for many vital applications where the user depends mainly on the hand pose as a system input. Virtual reality (VR) headset, shadow dexterous hand and in-air signature verification are a few examples of applications that require to track the hand movements in real-time. The state-of-the-art 3D hand pose estimation methods are based on the Convolutional Neural Network (CNN). These methods are implemented on Graphics Processing Units (GPUs) mainly due to their extensive computational requirements. However, GPUs are not suitable for the practical application scenarios, where the low power consumption is crucial. Furthermore, the difficulty of embedding a bulky GPU into a small device prevents the portability of such applications on mobile devices. The goal of this work is to provide an energy efficient solution for an existing depth camera based hand pose estimation algorithm. First, we compress the deep neural network model by applying the dynamic quantization techniques on different layers to achieve maximum compression without compromising accuracy. Afterwards, we design a custom hardware architecture. For our device we selected the FPGA as a target platform because FPGAs provide high energy efficiency and can be integrated in portable devices. Our solution implemented on Xilinx UltraScale+ MPSoC FPGA is 4.2× faster and 577.3× more energy efficient than the original implementation of the hand pose estimation algorithm on NVIDIA GeForce GTX 1070.

Download Full-text

Wearable Device for High-Speed Hand Pose Estimation with a Ultrasmall Camera

Journal of Robotics and Mechatronics ◽

10.20965/jrm.2015.p0167 ◽

2015 ◽

Vol 27 (2) ◽

pp. 167-173 ◽

Cited By ~ 5

Author(s):

Motomasa Tomida ◽

◽

Kiyoshi Hoshino

Keyword(s):

Pose Estimation ◽

High Speed ◽

Computational Cost ◽

Wearable Device ◽

Estimation Accuracy ◽

Hand Pose Estimation ◽

Data Matching ◽

Pip Joint ◽

Image Characteristic ◽

Hand Pose

<div class=""abs_img""> <img src=""[disp_template_path]/JRM/abst-image/00270002/06.jpg"" width=""300"" /> Hand pose estimation with ultrasmall camera</div> Operating a robot intentionally by using various complex motions of the hands and fingers requires a system that accurately detects hand and finger motions at high speed. This study uses an ultrasmall camera and compact computer for development of a wearable device of hand pose estimation, also called a hand-capture device. The accurate estimations, however, require data matching with a large database. But a compact computer usually has only limited memory and low machine power. We avoided this problem by reducing frequently used image characteristics from 1,600 dimensions to 64 dimensions of characteristic quantities. This saved on memory and lowered computational cost while achieving high accuracy and speed. To enable an operator to wear the device comfortably, the camera was placed as close to the back of the hand as possible to enable hand pose estimation from hand images without fingertips. A prototype device with a compact computer used to evaluate performance indicated that the device achieved high-speed estimation. Estimation accuracy was 2.32°±14.61° at the PIP joint of the index finger and 3.06°±10.56° at the CM joint of the thumb – as accurate as obtained using previous methods. This indicated that dimensional compression of image-characteristic quantities is important for realizing a compact hand-capture device. </span>

Download Full-text

InterNet+: A Light Network for Hand Pose Estimation

Sensors ◽

10.3390/s21206747 ◽

2021 ◽

Vol 21 (20) ◽

pp. 6747

Author(s):

Yang Liu ◽

Jie Jiang ◽

Jiahao Sun ◽

Xianghan Wang

Keyword(s):

Performance Improvement ◽

Pose Estimation ◽

Network Architecture ◽

Activation Function ◽

Depth Information ◽

Global Features ◽

Hand Pose Estimation ◽

Feature Extractor ◽

Rgb Images ◽

Hand Pose

Hand pose estimation from RGB images has always been a difficult task, owing to the incompleteness of the depth information. Moon et al. improved the accuracy of hand pose estimation by using a new network, InterNet, through their unique design. Still, the network still has potential for improvement. Based on the architecture of MobileNet v3 and MoGA, we redesigned a feature extractor that introduced the latest achievements in the field of computer vision, such as the ACON activation function and the new attention mechanism module, etc. Using these modules effectively with our network, architecture can better extract global features from an RGB image of the hand, leading to a greater performance improvement compared to InterNet and other similar networks.

Download Full-text

User-Adaptable Hand Pose Estimation Technique for Human-Robot Interaction

Journal of Robotics and Mechatronics ◽

10.20965/jrm.2009.p0739 ◽

2009 ◽

Vol 21 (6) ◽

pp. 739-748 ◽

Cited By ~ 1

Author(s):

Albert Causo ◽

◽

Etsuko Ueda ◽

Kentaro Takemura ◽

Yoshio Matsumoto ◽

...

Keyword(s):

Pose Estimation ◽

Human Robot Interaction ◽

Robot Interaction ◽

Camera System ◽

Data Glove ◽

Hand Pose Estimation ◽

Hand Model ◽

Hand Size ◽

Hand Shape ◽

Hand Pose

Hand pose estimation using a multi-camera system allows natural non-contact interfacing unlike when using bulky data gloves. To enable any user to use the system regardless of gender or physical differences such as hand size, we propose hand model individualization using only multiple cameras. From the calibration motion, our method estimates the finger link lengths as well as the hand shape by minimizing the gap between the hand model and observation. We confirmed the feasibility of our proposal by comparing 1) actual and estimated link lengths and 2) hand pose estimation results using our calibrated hand model, a prior hand model and data obtained from data glove measurements.

Download Full-text

Depth-Based Hand Pose Estimation: Methods, Data, and Challenges

International Journal of Computer Vision ◽

10.1007/s11263-018-1081-7 ◽

2018 ◽

Vol 126 (11) ◽

pp. 1180-1198 ◽

Cited By ~ 11

Author(s):

James Steven Supančič ◽

Grégory Rogez ◽

Yi Yang ◽

Jamie Shotton ◽

Deva Ramanan

Keyword(s):

Pose Estimation ◽

Estimation Methods ◽

Hand Pose Estimation ◽

Hand Pose

Download Full-text

CFAM: Estimating 3D Hand Poses from a Single RGB Image with Attention

Applied Sciences ◽

10.3390/app10020618 ◽

2020 ◽

Vol 10 (2) ◽

pp. 618

Author(s):

Xianghan Wang ◽

Jie Jiang ◽

Yanming Guo ◽

Lai Kang ◽

Yingmei Wei ◽

...

Keyword(s):

Computer Vision ◽

Pose Estimation ◽

Spatial Information ◽

Image Features ◽

Estimation Methods ◽

Feature Maps ◽

Hand Pose Estimation ◽

Rgb Images ◽

Rgb Image ◽

Hand Pose

Precise 3D hand pose estimation can be used to improve the performance of human–computer interaction (HCI). Specifically, computer-vision-based hand pose estimation can make this process more natural. Most traditional computer-vision-based hand pose estimation methods use depth images as the input, which requires complicated and expensive acquisition equipment. Estimation through a single RGB image is more convenient and less expensive. Previous methods based on RGB images utilize only 2D keypoint score maps to recover 3D hand poses but ignore the hand texture features and the underlying spatial information in the RGB image, which leads to a relatively low accuracy. To address this issue, we propose a channel fusion attention mechanism that combines 2D keypoint features and RGB image features at the channel level. In particular, the proposed method replans weights by using cascading RGB images and 2D keypoint features, which enables rational planning and the utilization of various features. Moreover, our method improves the fusion performance of different types of feature maps. Multiple contrast experiments on public datasets demonstrate that the accuracy of our proposed method is comparable to the state-of-the-art accuracy.

Download Full-text

Multihead Self Attention Hand Pose Estimation

E3S Web of Conferences ◽

10.1051/e3sconf/202021803023 ◽

2020 ◽

Vol 218 ◽

pp. 03023

Author(s):

Zhiqin Zhang ◽

Bo Zhang ◽

Fen Li ◽

Dehua Kong

Keyword(s):

Neural Networks ◽

Pose Estimation ◽

State Of The Art ◽

Salient Feature ◽

Estimation Methods ◽

Feature Maps ◽

Test Dataset ◽

Hand Pose Estimation ◽

Network Backbone ◽

Hand Pose

In This paper, we propose a hand pose estimation neural networks architecture named MSAHP which can improve PCK (percentage correct keypoints) greatly by fusing self-attention module in CNN (Convolutional Neural Networks). The proposed network is based on a ResNet (Residual Neural Network) backbone and concatenate discriminative features through multiple different scale feature maps, then multiple head self-attention module was used to focus on the salient feature map area. In recent years, self-attention mechanism was applicated widely in NLP and speech recognition, which can improve greatly key metrics. But in compute vision especially for hand pose estimation, we did not find the application. Experiments on hand pose estimation dataset demonstrate the improved PCK of our MSAHP than the existing state-of-the-art hand pose estimation methods. Specifically, the proposed method can achieve 93.68% PCK score on our mixed test dataset.

Download Full-text