Background: Intelligent monitoring of human action in production is an important step to help standardize production processes and construct a digital twin shop-floor rapidly. Human action has a significant impact on the production safety and efficiency of a shop-floor, however, because of the high individual initiative of humans, it is difficult to realize real-time action detection in a digital twin shop-floor. Methods: We proposed a real-time detection approach for shop-floor production action. This approach used the sequence data of continuous human skeleton joints sequences as the input. We then reconstructed the Joint Classification-Regression Recurrent Neural Networks (JCR-RNN) based on Temporal Convolution Network (TCN) and Graph Convolution Network (GCN). We called this approach the Temporal Action Detection Net (TAD-Net), which realized real-time shop-floor production action detection. Results: The results of the verification experiment showed that our approach has achieved a high temporal positioning score, recognition speed, and accuracy when applied to the existing Online Action Detection (OAD) dataset and the Nanjing University of Science and Technology 3 Dimensions (NJUST3D) dataset. TAD-Net can meet the actual needs of the digital twin shop-floor. Conclusions: Our method has higher recognition accuracy, temporal positioning accuracy, and faster running speed than other mainstream network models, it can better meet actual application requirements, and has important research value and practical significance for standardizing shop-floor production processes, reducing production security risks, and contributing to the understanding of real-time production action.
Video action detection by learning graph-based spatio-temporal interactions
