Representation Learning of Temporal Dynamics for Skeleton-Based Action Recognition

We introduce a generative Bayesian switching dynamical model for action recognition in 3D skeletal data. Our model encodes highly correlated skeletal data into a few sets of low-dimensional switching temporal processes and from there decodes to the motion data and their associated action labels. We parameterize these temporal processes with regard to a switching deep autoregressive prior to accommodate both multimodal and higher-order nonlinear inter-dependencies. This results in a dynamical deep generative latent model that parses meaningful intrinsic states in skeletal dynamics and enables action recognition. These sequences of states provide visual and quantitative interpretations about motion primitives that gave rise to each action class, which have not been explored previously. In contrast to previous works, which often overlook temporal dynamics, our method explicitly model temporal transitions and is generative. Our experiments on two large-scale 3D skeletal datasets substantiate the superior performance of our model in comparison with the state-of-the-art methods. Specifically, our method achieved 6.3% higher action classification accuracy (by incorporating a dynamical generative framework), and 3.5% better predictive error (by employing a nonlinear second-order dynamical transition model) when compared with the best-performing competitors.

Download Full-text

Heterogeneous Features Fusion with Collaborative Representation Learning for 3D Action Recognition

2017 IEEE International Symposium on Multimedia (ISM) ◽

10.1109/ism.2017.30 ◽

2017 ◽

Cited By ~ 1

Author(s):

Chengwu Liang ◽

Enqing Chen ◽

Lin Qi ◽

Ling Guan

Keyword(s):

Action Recognition ◽

Representation Learning ◽

Collaborative Representation ◽

Features Fusion ◽

Heterogeneous Features

Download Full-text

Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33018545 ◽

2019 ◽

Vol 33 ◽

pp. 8545-8552 ◽

Cited By ~ 20

Author(s):

Dahun Kim ◽

Donghyeon Cho ◽

In So Kweon

Keyword(s):

Action Recognition ◽

Large Scale ◽

State Of The Art ◽

Representation Learning ◽

Space Time ◽

Temporal Relation ◽

Still Images ◽

Video Frames ◽

Spatio Temporal ◽

The Cost

Self-supervised tasks such as colorization, inpainting and zigsaw puzzle have been utilized for visual representation learning for still images, when the number of labeled images is limited or absent at all. Recently, this worthwhile stream of study extends to video domain where the cost of human labeling is even more expensive. However, the most of existing methods are still based on 2D CNN architectures that can not directly capture spatio-temporal information for video applications. In this paper, we introduce a new self-supervised task called as Space-Time Cubic Puzzles to train 3D CNNs using large scale video dataset. This task requires a network to arrange permuted 3D spatio-temporal crops. By completing Space-Time Cubic Puzzles, the network learns both spatial appearance and temporal relation of video frames, which is our final goal. In experiments, we demonstrate that our learned 3D representation is well transferred to action recognition tasks, and outperforms state-of-the-art 2D CNN-based competitors on UCF101 and HMDB51 datasets.

Download Full-text

Multi-view representation learning for multi-view action recognition

Journal of Visual Communication and Image Representation ◽

10.1016/j.jvcir.2017.01.019 ◽

2017 ◽

Vol 48 ◽

pp. 453-460 ◽

Cited By ~ 12

Author(s):

Tong Hao ◽

Dan Wu ◽

Qian Wang ◽

Jin-Sheng Sun

Keyword(s):

Action Recognition ◽

Representation Learning

Download Full-text

An Efficient Human Instance-Guided Framework for Video Action Recognition

Sensors ◽

10.3390/s21248309 ◽

2021 ◽

Vol 21 (24) ◽

pp. 8309

Author(s):

Inwoong Lee ◽

Doyoung Kim ◽

Dongyoon Wee ◽

Sanghoon Lee

Keyword(s):

Action Recognition ◽

Temporal Dynamics ◽

State Of The Art ◽

Confusion Matrix ◽

Human Action Recognition ◽

Human Action ◽

Stream Networks ◽

Comparable Performance ◽

Motion Features ◽

Temporal Action

In recent years, human action recognition has been studied by many computer vision researchers. Recent studies have attempted to use two-stream networks using appearance and motion features, but most of these approaches focused on clip-level video action recognition. In contrast to traditional methods which generally used entire images, we propose a new human instance-level video action recognition framework. In this framework, we represent the instance-level features using human boxes and keypoints, and our action region features are used as the inputs of the temporal action head network, which makes our framework more discriminative. We also propose novel temporal action head networks consisting of various modules, which reflect various temporal dynamics well. In the experiment, the proposed models achieve comparable performance with the state-of-the-art approaches on two challenging datasets. Furthermore, we evaluate the proposed features and networks to verify the effectiveness of them. Finally, we analyze the confusion matrix and visualize the recognized actions at human instance level when there are several people.

Download Full-text

Dual-Stream Structured Graph Convolution Network for Skeleton-Based Action Recognition

ACM Transactions on Multimedia Computing Communications and Applications ◽

10.1145/3450410 ◽

2021 ◽

Vol 17 (4) ◽

pp. 1-22

Author(s):

Chunyan Xu ◽

Rong Liu ◽

Tong Zhang ◽

Zhen Cui ◽

Jian Yang ◽

...

Keyword(s):

Action Recognition ◽

Temporal Dynamics ◽

Recognition Task ◽

Human Movement ◽

Body Part ◽

Human Action ◽

Human Motion ◽

Body Parts ◽

Dynamic Interactions ◽

Spatio Temporal

In this work, we propose a dual-stream structured graph convolution network ( DS-SGCN ) to solve the skeleton-based action recognition problem. The spatio-temporal coordinates and appearance contexts of the skeletal joints are jointly integrated into the graph convolution learning process on both the video and skeleton modalities. To effectively represent the skeletal graph of discrete joints, we create a structured graph convolution module specifically designed to encode partitioned body parts along with their dynamic interactions in the spatio-temporal sequence. In more detail, we build a set of structured intra-part graphs, each of which can be adopted to represent a distinctive body part (e.g., left arm, right leg, head). The inter-part graph is then constructed to model the dynamic interactions across different body parts; here each node corresponds to an intra-part graph built above, while an edge between two nodes is used to express these internal relationships of human movement. We implement the graph convolution learning on both intra- and inter-part graphs in order to obtain the inherent characteristics and dynamic interactions, respectively, of human action. After integrating the intra- and inter-levels of spatial context/coordinate cues, a convolution filtering process is conducted on time slices to capture these temporal dynamics of human motion. Finally, we fuse two streams of graph convolution responses in order to predict the category information of human action in an end-to-end fashion. Comprehensive experiments on five single/multi-modal benchmark datasets (including NTU RGB+D 60, NTU RGB+D 120, MSR-Daily 3D, N-UCLA, and HDM05) demonstrate that the proposed DS-SGCN framework achieves encouraging performance on the skeleton-based action recognition task.

Download Full-text