Spatio-temporal 3D pose estimation and tracking of human body parts using the Shape Flow algorithm

We present a deep learning-based multitask framework for joint 3D human pose estimation and action recognition from RGB sensors using simple cameras. The approach proceeds along two stages. In the first, a real-time 2D pose detector is run to determine the precise pixel location of important keypoints of the human body. A two-stream deep neural network is then designed and trained to map detected 2D keypoints into 3D poses. In the second stage, the Efficient Neural Architecture Search (ENAS) algorithm is deployed to find an optimal network architecture that is used for modeling the spatio-temporal evolution of the estimated 3D poses via an image-based intermediate representation and performing action recognition. Experiments on Human3.6M, MSR Action3D and SBU Kinect Interaction datasets verify the effectiveness of the proposed method on the targeted tasks. Moreover, we show that the method requires a low computational budget for training and inference. In particular, the experimental results show that by using a monocular RGB sensor, we can develop a 3D pose estimation and human action recognition approach that reaches the performance of RGB-depth sensors. This opens up many opportunities for leveraging RGB cameras (which are much cheaper than depth cameras and extensively deployed in private and public places) to build intelligent recognition systems.

Download Full-text

Pointless Pose: Part Affinity Field-Based 3D Pose Estimation without Detecting Keypoints

Electronics ◽

10.3390/electronics10080929 ◽

2021 ◽

Vol 10 (8) ◽

pp. 929

Author(s):

Jue Wang ◽

Zhigang Luo

Keyword(s):

Pose Estimation ◽

Human Body ◽

3D Pose Estimation ◽

New Approach ◽

Keypoint Detection ◽

Body Regions ◽

In The Wild ◽

Orientation Vector ◽

Human Pose ◽

Local Image

Human pose estimation finds its application in an extremely wide domain and is therefore never pointless. We propose in this paper a new approach that, unlike any prior one that we are aware of, bypasses the 2D keypoint detection step based on which the 3D pose is estimated, and is thus pointless. Our motivation is rather straightforward: 2D keypoint detection is vulnerable to occlusions and out-of-image absences, in which case the 2D errors propagate to 3D recovery and deteriorate the results. To this end, we resort to explicitly estimating the human body regions of interest (ROI) and their 3D orientations. Even if a portion of the human body, like the lower arm, is partially absent, the predicted orientation vector pointing from the upper arm will take advantage of the local image evidence and recover the 3D pose. This is achieved, specifically, by deforming a skeleton-shaped puppet template to fit the estimated orientation vectors. Despite its simple nature, the proposed approach yields truly robust and state-of-the-art results on several benchmarks and in-the-wild data.

Download Full-text

A Novel Multiple Person Pose Estimation Optimization Model Utilizing Genetic Algorithm

Journal of Physics Conference Series ◽

10.1088/1742-6596/2129/1/012027 ◽

2021 ◽

Vol 2129 (1) ◽

pp. 012027

Author(s):

Qing Zhang ◽

Lei Ding ◽

Kai Qing Zhou ◽

Jian Feng Li

Keyword(s):

Genetic Algorithm ◽

Pose Estimation ◽

Human Body ◽

Optimization Model ◽

Body Part ◽

Body Parts ◽

Position Information ◽

Assembly Method ◽

Optimal Value ◽

Human Pose

Abstract For traditional human pose estimation models rely on a large amount of human body feature information, this paper proposes an optimization model using genetic algorithm to solve the problem of multiple person body part assembly. Different from other human body parts assembly method. The method proposed in this paper depends on the joints position information, namely the sum of the connection distances between the joints as the objective function, and finds the optimal value to obtain the best human pose assembly information. The simulation results show that compared with the traditional OpenPose model, the model proposed in this paper can obtain the same human skeleton using less position information.

Download Full-text

Augmented Reality with Human Body Interaction Based on Monocular 3D Pose Estimation

Advanced Concepts for Intelligent Vision Systems - Lecture Notes in Computer Science ◽

10.1007/978-3-642-17688-3_31 ◽

2010 ◽

pp. 321-331 ◽

Cited By ~ 4

Author(s):

Huei-Yung Lin ◽

Ting-Wen Chen

Keyword(s):

Augmented Reality ◽

Pose Estimation ◽

Human Body ◽

3D Pose Estimation ◽

Body Interaction

Download Full-text

A Novel Self-Intersection Penalty Term for Statistical Body Shape Models and Its Applications in 3D Pose Estimation

Applied Sciences ◽

10.3390/app9030400 ◽

2019 ◽

Vol 9 (3) ◽

pp. 400 ◽

Cited By ~ 1

Author(s):

Zaiqiang Wu ◽

Wei Jiang ◽

Hao Luo ◽

Lin Cheng

Keyword(s):

Pose Estimation ◽

Body Shape ◽

The Body ◽

The Self ◽

Body Parts ◽

3D Pose Estimation ◽

Penalty Term ◽

Partial Derivatives ◽

Shape Models ◽

Low Dimensional

Statistical body shape models are widely used in 3D pose estimation due to their low-dimensional parameters representation. However, it is difficult to avoid self-intersection between body parts accurately. Motivated by this fact, we proposed a novel self-intersection penalty term for statistical body shape models applied in 3D pose estimation. To avoid the trouble of computing self-intersection for complex surfaces like the body meshes, the gradient of our proposed self-intersection penalty term is manually derived from the perspective of geometry. First, the self-intersection penalty term is defined as the volume of the self-intersection region. To calculate the partial derivatives with respect to the coordinates of the vertices, we employed detection rays to divide vertices of statistical body shape models into different groups depending on whether the vertex is in the region of self-intersection. Second, the partial derivatives could be easily derived by the normal vectors of neighboring triangles of the vertices. Finally, this penalty term could be applied in gradient-based optimization algorithms to remove the self-intersection of triangular meshes without using any approximation. Qualitative and quantitative evaluations were conducted to demonstrate the effectiveness and generality of our proposed method compared with previous approaches. The experimental results show that our proposed penalty term can avoid self-intersection to exclude unreasonable predictions and improves the accuracy of 3D pose estimation indirectly. Further more, the proposed method could be employed universally in triangular mesh based 3D reconstruction.

Download Full-text

Progressive Bi-C3D Pose Grammar for Human Pose Estimation

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i07.7004 ◽

2020 ◽

Vol 34 (07) ◽

pp. 13033-13040 ◽

Cited By ~ 2

Author(s):

Lu Zhou ◽

Yingying Chen ◽

Jinqiao Wang ◽

Hanqing Lu

Keyword(s):

Pose Estimation ◽

Human Body ◽

Message Passing ◽

Contextual Information ◽

Human Pose Estimation ◽

Body Parts ◽

Multi Scale ◽

Human Pose ◽

Human Joints ◽

Body Joints

In this paper, we propose a progressive pose grammar network learned with Bi-C3D (Bidirectional Convolutional 3D) for human pose estimation. Exploiting the dependencies among the human body parts proves effective in solving the problems such as complex articulation, occlusion and so on. Therefore, we propose two articulated grammars learned with Bi-C3D to build the relationships of the human joints and exploit the contextual information of human body structure. Firstly, a local multi-scale Bi-C3D kinematics grammar is proposed to promote the message passing process among the locally related joints. The multi-scale kinematics grammar excavates different levels human context learned by the network. Moreover, a global sequential grammar is put forward to capture the long-range dependencies among the human body joints. The whole procedure can be regarded as a local-global progressive refinement process. Without bells and whistles, our method achieves competitive performance on both MPII and LSP benchmarks compared with previous methods, which confirms the feasibility and effectiveness of C3D in information interactions.

Download Full-text