Human Action Recognition Combining Sequential Dynamic Images and Two-Stream Convolutional Network

2021 ◽  
Vol 58 (2) ◽  
pp. 0210007
Author(s):  
张文强 Zhang Wenqiang ◽  
王增强 Wang Zengqiang ◽  
张良 Zhang Liang
Keyword(s):  
Action Recognition ◽  
Human Action Recognition ◽  
Human Action ◽  
Convolutional Network

scholarly journals Learning Graph Convolutional Network for Skeleton-Based Human Action Recognition by Neural Searching

2020 ◽  
Vol 34 (03) ◽  
pp. 2669-2676 ◽  
Author(s):  
Wei Peng ◽  
Xiaopeng Hong ◽  
Haoyu Chen ◽  
Guoying Zhao
Keyword(s):  
Action Recognition ◽  
Large Scale ◽  
Order Approximation ◽  
Human Action Recognition ◽  
Search Space ◽  
Human Action ◽  
Higher Order ◽  
Dynamic Graph ◽  
Convolutional Network ◽  
Representational Capacity

Human action recognition from skeleton data, fuelled by the Graph Convolutional Network (GCN) with its powerful capability of modeling non-Euclidean data, has attracted lots of attention. However, many existing GCNs provide a pre-defined graph structure and share it through the entire network, which can loss implicit joint correlations especially for the higher-level features. Besides, the mainstream spectral GCN is approximated by one-order hop such that higher-order connections are not well involved. All of these require huge efforts to design a better GCN architecture. To address these problems, we turn to Neural Architecture Search (NAS) and propose the first automatically designed GCN for this task. Specifically, we explore the spatial-temporal correlations between nodes and build a search space with multiple dynamic graph modules. Besides, we introduce multiple-hop modules and expect to break the limitation of representational capacity caused by one-order approximation. Moreover, a corresponding sampling- and memory-efficient evolution strategy is proposed to search in this space. The resulted architecture proves the effectiveness of the higher-order approximation and the layer-wise dynamic graph modules. To evaluate the performance of the searched model, we conduct extensive experiments on two very large scale skeleton-based action recognition datasets. The results show that our model gets the state-of-the-art results in term of given metrics.

scholarly journals Enhanced Spatial and Extended Temporal Graph Convolutional Network for Skeleton-Based Action Recognition

Sensors ◽  
2020 ◽  
Vol 20 (18) ◽  
pp. 5260 ◽  
Author(s):  
Fanjia Li ◽  
Juanjuan Li ◽  
Aichun Zhu ◽  
Yonggang Xu ◽  
Hongsheng Yin ◽  
...  
Keyword(s):  
Action Recognition ◽  
Large Scale ◽  
Optimal Solution ◽  
Human Action Recognition ◽  
Human Action ◽  
Convolutional Network ◽  
Spatial Graph ◽  
Serial Connection ◽  
In Series ◽  
Temporal Graph

In the skeleton-based human action recognition domain, the spatial-temporal graph convolution networks (ST-GCNs) have made great progress recently. However, they use only one fixed temporal convolution kernel, which is not enough to extract the temporal cues comprehensively. Moreover, simply connecting the spatial graph convolution layer (GCL) and the temporal GCL in series is not the optimal solution. To this end, we propose a novel enhanced spatial and extended temporal graph convolutional network (EE-GCN) in this paper. Three convolution kernels with different sizes are chosen to extract the discriminative temporal features from shorter to longer terms. The corresponding GCLs are then concatenated by a powerful yet efficient one-shot aggregation (OSA) + effective squeeze-excitation (eSE) structure. The OSA module aggregates the features from each layer once to the output, and the eSE module explores the interdependency between the channels of the output. Besides, we propose a new connection paradigm to enhance the spatial features, which expand the serial connection to a combination of serial and parallel connections by adding a spatial GCL in parallel with the temporal GCLs. The proposed method is evaluated on three large scale datasets, and the experimental results show that the performance of our method exceeds previous state-of-the-art methods.

scholarly journals A Deep Learning-Based Approach to Enable Action Recognition for Construction Equipment

2020 ◽  
Vol 2020 ◽  
pp. 1-14
Author(s):  
Jinyue Zhang ◽  
Lijun Zi ◽  
Yuexian Hou ◽  
Mingen Wang ◽  
Wenting Jiang ◽  
...  
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Action Recognition ◽  
Short Term Memory ◽  
Human Action Recognition ◽  
Human Action ◽  
Image Features ◽  
Construction Equipment ◽  
Video Frame ◽  
Convolutional Network

In order to support smart construction, digital twin has been a well-recognized concept for virtually representing the physical facility. It is equally important to recognize human actions and the movement of construction equipment in virtual construction scenes. Compared to the extensive research on human action recognition (HAR) that can be applied to identify construction workers, research in the field of construction equipment action recognition (CEAR) is very limited, mainly due to the lack of available datasets with videos showing the actions of construction equipment. The contributions of this research are as follows: (1) the development of a comprehensive video dataset of 2,064 clips with five action types for excavators and dump trucks; (2) a new deep learning-based CEAR approach (known as a simplified temporal convolutional network or STCN) that combines a convolutional neural network (CNN) with long short-term memory (LSTM, an artificial recurrent neural network), where CNN is used to extract image features and LSTM is used to extract temporal features from video frame sequences; and (3) the comparison between this proposed new approach and a similar CEAR method and two of the best-performing HAR approaches, namely, three-dimensional (3D) convolutional networks (ConvNets) and two-stream ConvNets, to evaluate the performance of STCN and investigate the possibility of directly transferring HAR approaches to the field of CEAR.

scholarly journals Spatio-Temporal Pyramid Graph Convolutions for Human Action Recognition and Postural Assessment

2021 ◽  
Author(s):  
Jawad Khan
Keyword(s):  
Risk Assessment ◽  
Computer Vision ◽  
Action Recognition ◽  
Human Action Recognition ◽  
Human Action ◽  
Convolutional Network ◽  
Potential Applications ◽  
Postural Assessment ◽  
Spatio Temporal ◽  
Ergonomic Risk Assessment

Recognition of human actions and associated interactions with objects and the environment is animportant problem in computer vision due to its potential applications in a variety of domains. Themost versatile methods can generalize to various environments and deal with cluttered backgrounds,occlusions, and viewpoint variations. Among them, methods based on graph convolutionalnetworks that extract features from the skeleton have demonstrated promising performance. In thispaper, we propose a novel Spatio-Temporal Pyramid Graph Convolutional Network (ST-PGN) foronline action recognition for ergonomic risk assessment that enables the use of features from alllevels of the skeleton feature hierarchy.

scholarly journals A Lightweight Hierarchical Model with Frame-Level Joints Adaptive Graph Convolution for Skeleton-Based Action Recognition

2021 ◽  
Vol 2021 ◽  
pp. 1-13
Author(s):  
Yujian Jiang ◽  
Xue Yang ◽  
Jingyu Liu ◽  
Junming Zhang
Keyword(s):  
Action Recognition ◽  
Feature Fusion ◽  
Low Cost ◽  
Real Life ◽  
Model Performance ◽  
Human Action Recognition ◽  
Human Action ◽  
Embedded Devices ◽  
Convolutional Network ◽  
Action Sequence

In skeleton-based human action recognition methods, human behaviours can be analysed through temporal and spatial changes in the human skeleton. Skeletons are not limited by clothing changes, lighting conditions, or complex backgrounds. This recognition method is robust and has aroused great interest; however, many existing studies used deep-layer networks with large numbers of required parameters to improve the model performance and thus lost the advantage of less computation of skeleton data. It is difficult to deploy previously established models to real-life applications based on low-cost embedded devices. To obtain a model with fewer parameters and a higher accuracy, this study designed a lightweight frame-level joints adaptive graph convolutional network (FLAGCN) model to solve skeleton-based action recognition tasks. Compared with the classical 2s-AGCN model, the new model obtained a higher precision with 1/8 of the parameters and 1/9 of the floating-point operations (FLOPs). Our proposed network characterises three main improvements. First, a previous feature-fusion method replaces the multistream network and reduces the number of required parameters. Second, at the spatial level, two kinds of graph convolution methods capture different aspects of human action information. A frame-level graph convolution constructs a human topological structure for each data frame, whereas an adjacency graph convolution captures the characteristics of the adjacent joints. Third, the model proposed in this study hierarchically extracts different levels of action sequence features, making the model clear and easy to understand; further, it reduces the depth of the model and the number of parameters. A large number of experiments on the NTU RGB + D 60 and 120 data sets show that this method has the advantages of few required parameters, low computational costs, and fast speeds. It also has a simple structure and training process that make it easy to deploy in real-time recognition systems based on low-cost embedded devices.

scholarly journals Skeleton-Based Action Recognition Based on Distance Vector and Multihigh View Adaptive Networks

2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Min Zhang ◽  
Haijie Yang ◽  
Pengfei Li ◽  
Ming Jiang
Keyword(s):  
Action Recognition ◽  
Human Action Recognition ◽  
Human Action ◽  
Adaptive Networks ◽  
Convolutional Network ◽  
Current Frame ◽  
Convolutional Networks ◽  
Distance Vector ◽  
Temporal Graph ◽  
Ablation Study

Skeleton-based human action recognition has attracted much attention in the field of computer vision. Most of the previous studies are based on fixed skeleton graphs so that only the local physical dependencies among joints can be captured, resulting in the omission of implicit joint correlations. In addition, under different views, the content of the same action is very different. In some views, keypoints will be blocked, which will cause recognition errors. In this paper, an action recognition method based on distance vector and multihigh view adaptive network (DV-MHNet) is proposed to address this challenging task. Among the mentioned techniques, the multihigh (MH) view adaptive networks are constructed to automatically determine the best observation view at different heights, obtain complete keypoints information of the current frame image, and enhance the robustness and generalization of the model to recognize actions at different heights. Then, the distance vector (DV) mechanism is introduced on this basis to establish the relative distance and relative orientation between different keypoints in the same frame and the same keypoints in different frame to obtain the global potential relationship of each keypoint, and finally by constructing the spatial temporal graph convolutional network to take into account the information in space and time, the characteristics of the action are learned. This paper has done the ablation study with traditional spatial temporal graph convolutional networks and with or without multihigh view adaptive networks, which reasonably proves the effectiveness of the model. The model is evaluated on two widely used action recognition benchmarks (NTU-RGB + D and PKU-MMD). Our method achieves better performance on both datasets.

Sign in / Sign up

Export Citation Format

Share Document