scholarly journals Multihead Self Attention Hand Pose Estimation

2020 ◽  
Vol 218 ◽  
pp. 03023
Author(s):  
Zhiqin Zhang ◽  
Bo Zhang ◽  
Fen Li ◽  
Dehua Kong
Keyword(s):  
Neural Networks ◽  
Pose Estimation ◽  
State Of The Art ◽  
Salient Feature ◽  
Estimation Methods ◽  
Feature Maps ◽  
Test Dataset ◽  
Hand Pose Estimation ◽  
Network Backbone ◽  
Hand Pose

In This paper, we propose a hand pose estimation neural networks architecture named MSAHP which can improve PCK (percentage correct keypoints) greatly by fusing self-attention module in CNN (Convolutional Neural Networks). The proposed network is based on a ResNet (Residual Neural Network) backbone and concatenate discriminative features through multiple different scale feature maps, then multiple head self-attention module was used to focus on the salient feature map area. In recent years, self-attention mechanism was applicated widely in NLP and speech recognition, which can improve greatly key metrics. But in compute vision especially for hand pose estimation, we did not find the application. Experiments on hand pose estimation dataset demonstrate the improved PCK of our MSAHP than the existing state-of-the-art hand pose estimation methods. Specifically, the proposed method can achieve 93.68% PCK score on our mixed test dataset.

scholarly journals CFAM: Estimating 3D Hand Poses from a Single RGB Image with Attention

2020 ◽  
Vol 10 (2) ◽  
pp. 618
Author(s):  
Xianghan Wang ◽  
Jie Jiang ◽  
Yanming Guo ◽  
Lai Kang ◽  
Yingmei Wei ◽  
...  
Keyword(s):  
Computer Vision ◽  
Pose Estimation ◽  
Spatial Information ◽  
Image Features ◽  
Estimation Methods ◽  
Feature Maps ◽  
Hand Pose Estimation ◽  
Rgb Images ◽  
Rgb Image ◽  
Hand Pose

Precise 3D hand pose estimation can be used to improve the performance of human–computer interaction (HCI). Specifically, computer-vision-based hand pose estimation can make this process more natural. Most traditional computer-vision-based hand pose estimation methods use depth images as the input, which requires complicated and expensive acquisition equipment. Estimation through a single RGB image is more convenient and less expensive. Previous methods based on RGB images utilize only 2D keypoint score maps to recover 3D hand poses but ignore the hand texture features and the underlying spatial information in the RGB image, which leads to a relatively low accuracy. To address this issue, we propose a channel fusion attention mechanism that combines 2D keypoint features and RGB image features at the channel level. In particular, the proposed method replans weights by using cascading RGB images and 2D keypoint features, which enables rational planning and the utilization of various features. Moreover, our method improves the fusion performance of different types of feature maps. Multiple contrast experiments on public datasets demonstrate that the accuracy of our proposed method is comparable to the state-of-the-art accuracy.

scholarly journals A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods

Sensors ◽  
2020 ◽  
Vol 20 (4) ◽  
pp. 1074 ◽  
Author(s):  
Weiya Chen ◽  
Chenchen Yu ◽  
Chenyu Tu ◽  
Zehua Lyu ◽  
Jing Tang ◽  
...  
Keyword(s):  
Computer Vision ◽  
Pose Estimation ◽  
Wearable Sensors ◽  
Complex Structure ◽  
Estimation Methods ◽  
Human Computer Interactions ◽  
Hand Pose Estimation ◽  
Timely Review ◽  
Kinematic Models ◽  
Hand Pose

Real-time sensing and modeling of the human body, especially the hands, is an important research endeavor for various applicative purposes such as in natural human computer interactions. Hand pose estimation is a big academic and technical challenge due to the complex structure and dexterous movement of human hands. Boosted by advancements from both hardware and artificial intelligence, various prototypes of data gloves and computer-vision-based methods have been proposed for accurate and rapid hand pose estimation in recent years. However, existing reviews either focused on data gloves or on vision methods or were even based on a particular type of camera, such as the depth camera. The purpose of this survey is to conduct a comprehensive and timely review of recent research advances in sensor-based hand pose estimation, including wearable and vision-based solutions. Hand kinematic models are firstly discussed. An in-depth review is conducted on data gloves and vision-based sensor systems with corresponding modeling methods. Particularly, this review also discusses deep-learning-based methods, which are very promising in hand pose estimation. Moreover, the advantages and drawbacks of the current hand gesture estimation methods, the applicative scope, and related challenges are also discussed.

scholarly journals Real-Time Energy Efficient Hand Pose Estimation: A Case Study

Sensors ◽  
2020 ◽  
Vol 20 (10) ◽  
pp. 2828
Author(s):  
Mhd Rashed Al Koutayni ◽  
Vladimir Rybalkin ◽  
Jameel Malik ◽  
Ahmed Elhayek ◽  
Christian Weis ◽  
...  
Keyword(s):  
Neural Network ◽  
Real Time ◽  
Pose Estimation ◽  
Energy Efficient ◽  
Graphics Processing Units ◽  
Estimation Algorithm ◽  
High Energy ◽  
Estimation Methods ◽  
Hand Pose Estimation ◽  
Hand Pose

The estimation of human hand pose has become the basis for many vital applications where the user depends mainly on the hand pose as a system input. Virtual reality (VR) headset, shadow dexterous hand and in-air signature verification are a few examples of applications that require to track the hand movements in real-time. The state-of-the-art 3D hand pose estimation methods are based on the Convolutional Neural Network (CNN). These methods are implemented on Graphics Processing Units (GPUs) mainly due to their extensive computational requirements. However, GPUs are not suitable for the practical application scenarios, where the low power consumption is crucial. Furthermore, the difficulty of embedding a bulky GPU into a small device prevents the portability of such applications on mobile devices. The goal of this work is to provide an energy efficient solution for an existing depth camera based hand pose estimation algorithm. First, we compress the deep neural network model by applying the dynamic quantization techniques on different layers to achieve maximum compression without compromising accuracy. Afterwards, we design a custom hardware architecture. For our device we selected the FPGA as a target platform because FPGAs provide high energy efficiency and can be integrated in portable devices. Our solution implemented on Xilinx UltraScale+ MPSoC FPGA is 4.2× faster and 577.3× more energy efficient than the original implementation of the hand pose estimation algorithm on NVIDIA GeForce GTX 1070.

scholarly journals CrossFuNet: RGB and Depth Cross-Fusion Network for Hand Pose Estimation

Sensors ◽  
2021 ◽  
Vol 21 (18) ◽  
pp. 6095
Author(s):  
Xiaojing Sun ◽  
Bin Wang ◽  
Longxiang Huang ◽  
Qian Zhang ◽  
Sulei Zhu ◽  
...  
Keyword(s):  
Pose Estimation ◽  
Depth Map ◽  
Depth Information ◽  
Feature Maps ◽  
Hand Pose Estimation ◽  
Depth Sensors ◽  
Key Points ◽  
Rgb Images ◽  
Public Datasets ◽  
Hand Pose

Despite recent successes in hand pose estimation from RGB images or depth maps, inherent challenges remain. RGB-based methods suffer from heavy self-occlusions and depth ambiguity. Depth sensors rely heavily on distance and can only be used indoors, thus there are many limitations to the practical application of depth-based methods. The aforementioned challenges have inspired us to combine the two modalities to offset the shortcomings of the other. In this paper, we propose a novel RGB and depth information fusion network to improve the accuracy of 3D hand pose estimation, which is called CrossFuNet. Specifically, the RGB image and the paired depth map are input into two different subnetworks, respectively. The feature maps are fused in the fusion module in which we propose a completely new approach to combine the information from the two modalities. Then, the common method is used to regress the 3D key-points by heatmaps. We validate our model on two public datasets and the results reveal that our model outperforms the state-of-the-art methods.

scholarly journals 3D Hand Pose Estimation with Neural Networks

2013 ◽  
pp. 504-512
Author(s):  
Jose Antonio Serra ◽  
José Garcia-Rodriguez ◽  
Sergio Orts-Escolano ◽  
Juan Manuel Garcia-Chamizo ◽  
Anastassia Angelopoulou ◽  
...  
Keyword(s):  
Neural Networks ◽  
Pose Estimation ◽  
Hand Pose Estimation ◽  
Hand Pose

scholarly journals Depth-Based Hand Pose Estimation: Methods, Data, and Challenges

2018 ◽  
Vol 126 (11) ◽  
pp. 1180-1198 ◽  
Author(s):  
James Steven Supančič ◽  
Grégory Rogez ◽  
Yi Yang ◽  
Jamie Shotton ◽  
Deva Ramanan
Keyword(s):  
Pose Estimation ◽  
Estimation Methods ◽  
Hand Pose Estimation ◽  
Hand Pose

scholarly journals Learning hand latent features for unsupervised 3D hand pose estimation

2019 ◽  
Vol 2 (1) ◽  
pp. 1
Author(s):  
Jamal Firmat Banzi1,2 ◽  
Isack Bulugu3 ◽  
Zhongfu Ye1
Keyword(s):  
Pose Estimation ◽  
Predictive Coding ◽  
Depth Map ◽  
Real Data ◽  
Training Data ◽  
Estimation Methods ◽  
Hand Pose Estimation ◽  
Latent Features ◽  
Estimation System ◽  
Hand Pose

Recent hand pose estimation methods require large numbers of annotated training data to extract the dynamic information from a hand representation. Nevertheless, precise and dense annotation on the real data is difficult to come by and the amount of information passed to the training algorithm is significantly higher. This paper presents an approach to developing a hand pose estimation system which can accurately regress a 3D pose in an unsupervised manner. The whole process is performed in three stages. Firstly, the hand is modelled by a novel latent tree dependency model (LTDM) which transforms internal joints location to an explicit representation. Secondly, we perform predictive coding of image sequences of hand poses in order to capture latent features underlying a given image without supervision. A mapping is then performed between an image depth and a generated representation. Thirdly, the hand joints are regressed using convolutional neural networks to finally estimate the latent pose given some depth map. Finally, an unsupervised error term which is a part of the recurrent architecture ensures smooth estimations of the final pose. To demonstrate the performance of the proposed system, a complete experiment is conducted on three challenging public datasets, ICVL, MSRA, and NYU. The empirical results show the significant performance of our method which is comparable or better than state-of-the-art approaches.

scholarly journals Towards markerless surgical tool and hand pose estimation

2021 ◽  
Author(s):  
Jonas Hein ◽  
Matthias Seibold ◽  
Federica Bogo ◽  
Mazda Farshad ◽  
Marc Pollefeys ◽  
...  
Keyword(s):  
Pose Estimation ◽  
State Of The Art ◽  
Synthetic Data ◽  
Ground Truth ◽  
Real Data ◽  
Computer Assisted ◽  
Data Generation ◽  
Hand Pose Estimation ◽  
Surgical Tool ◽  
Hand Pose

Abstract Purpose:  Tracking of tools and surgical activity is becoming more and more important in the context of computer assisted surgery. In this work, we present a data generation framework, dataset and baseline methods to facilitate further research in the direction of markerless hand and instrument pose estimation in realistic surgical scenarios. Methods:  We developed a rendering pipeline to create inexpensive and realistic synthetic data for model pretraining. Subsequently, we propose a pipeline to capture and label real data with hand and object pose ground truth in an experimental setup to gather high-quality real data. We furthermore present three state-of-the-art RGB-based pose estimation baselines. Results:  We evaluate three baseline models on the proposed datasets. The best performing baseline achieves an average tool 3D vertex error of 16.7 mm on synthetic data as well as 13.8 mm on real data which is comparable to the state-of-the art in RGB-based hand/object pose estimation. Conclusion:  To the best of our knowledge, we propose the first synthetic and real data generation pipelines to generate hand and object pose labels for open surgery. We present three baseline models for RGB based object and object/hand pose estimation based on RGB frames. Our realistic synthetic data generation pipeline may contribute to overcome the data bottleneck in the surgical domain and can easily be transferred to other medical applications.

Sign in / Sign up

Export Citation Format

Share Document