Multi-Level Two-Stream Fusion-Based Spatio-Temporal Attention Model for Violence Detection and Localization

Detection of violent human behavior is necessary for public safety and monitoring. However, it demands constant human observation and attention in human-based surveillance systems, which is a challenging task. Autonomous detection of violent human behavior is therefore essential for continuous uninterrupted video surveillance. In this paper, we propose a novel method for violence detection and localization in videos using the fusion of spatio-temporal features and attention model. The model consists of Fusion Convolutional Neural Network (Fusion-CNN), spatio-temporal attention modules and Bi-directional Convolutional LSTMs (BiConvLSTM). The Fusion-CNN learns both spatial and temporal features by combining multi-level inter-layer features from both RGB and Optical flow input frames. The spatial attention module is used to generate an importance mask to focus on the most important areas of the image frame. The temporal attention part, which is based on BiConvLSTM, identifies the most significant video frames which are related to violent activity. The proposed model can also localize and discriminate prominent regions in both spatial and temporal domains, given the weakly supervised training with only video-level classification labels. Experimental results evaluated on different publicly available benchmarking datasets show the superior performance of the proposed model in comparison with the existing methods. Our model achieves the improved accuracies (ACC) of 89.1%, 99.1% and 98.15% for RWF-2000, HockeyFight and Crowd-Violence datasets, respectively. For CCTV-FIGHTS dataset, we choose the mean average precision (mAp) performance metric and our model obtained 80.7% mAp.

Download Full-text

Deep Spatio-Temporal Attention Model for Grain Storage Temperature Forecasting

2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS) ◽

10.1109/icpads51040.2020.00082 ◽

2020 ◽

Author(s):

Shanshan Duan ◽

Weidong Yang ◽

Xuyu Wang ◽

Shiwen Mao ◽

Yuan Zhang

Keyword(s):

Storage Temperature ◽

Grain Storage ◽

Temporal Attention ◽

Attention Model ◽

Temperature Forecasting ◽

Spatio Temporal

Download Full-text

Multi-agent Attentional Activity Recognition

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2019/186 ◽

2019 ◽

Cited By ~ 3

Author(s):

Kaixuan Chen ◽

Lina Yao ◽

Dalin Zhang ◽

Bin Guo ◽

Zhiwen Yu

Keyword(s):

Activity Recognition ◽

State Of The Art ◽

Body Part ◽

Body Parts ◽

Temporal Attention ◽

Attention Model ◽

Proposed Model ◽

Collective Motions ◽

Multi Agent ◽

Real World Datasets

Multi-modality is an important feature of sensor based activity recognition. In this work, we consider two inherent characteristics of human activities, the spatially-temporally varying salience of features and the relations between activities and corresponding body part motions. Based on these, we propose a multi-agent spatial-temporal attention model. The spatial-temporal attention mechanism helps intelligently select informative modalities and their active periods. And the multiple agents in the proposed model represent activities with collective motions across body parts by independently selecting modalities associated with single motions. With a joint recognition goal, the agents share gained information and coordinate their selection policies to learn the optimal recognition model. The experimental results on four real-world datasets demonstrate that the proposed model outperforms the state-of-the-art methods.

Download Full-text

Pain-attentive network: a deep spatio-temporal attention model for pain estimation

Multimedia Tools and Applications ◽

10.1007/s11042-020-09397-1 ◽

2020 ◽

Vol 79 (37-38) ◽

pp. 28329-28354

Author(s):

Dong Huang ◽

Zhaoqiang Xia ◽

Joshua Mwesigye ◽

Xiaoyi Feng

Keyword(s):

Temporal Attention ◽

Attention Model ◽

Spatio Temporal

Download Full-text

An Adaptive Construction Method of Hierarchical Spatio-Temporal Index for Vector Data under Peer-to-Peer Networks

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi8110512 ◽

2019 ◽

Vol 8 (11) ◽

pp. 512 ◽

Cited By ~ 2

Author(s):

Li ◽

Wu ◽

Zhao

Keyword(s):

Data Storage ◽

High Efficiency ◽

Construction Method ◽

Vector Data ◽

Temporal Index ◽

Temporal Features ◽

Nosql Database ◽

Multi Level ◽

Spatio Temporal ◽

Index Level

Spatio-temporal indexing is a key technique in spatio-temporal data storage and management. Indexing methods based on spatial filling curves are popular in research on the spatio-temporal indexing of vector data in the Not Relational (NoSQL) database. However, the existing methods mostly focus on spatial indexing, which makes it difficult to balance the efficiencies of time and space queries. In addition, for non-point elements (line and polygon elements), it remains difficult to determine the optimal index level. To address these issues, this paper proposes an adaptive construction method of hierarchical spatio-temporal index for vector data. Firstly, a joint spatio-temporal information coding based on the combination of the partition and sort key strategies is presented. Secondly, the multilevel expression structure of spatio-temporal elements consisting of point and non-point elements in the joint coding is given. Finally, an adaptive multi-level index tree is proposed to realize the spatio-temporal index (Multi-level Sphere 3, MLS3) based on the spatio-temporal characteristics of geographical entities. Comparison with the XZ3 index algorithm proposed by GeoMesa proved that the MLS3 indexing method not only reasonably expresses the spatio-temporal features of non-point elements and determines their optimal index level, but also avoids storage hotspots while achieving spatio-temporal retrieval with high efficiency.

Download Full-text

Towards human behavior recognition based on spatio temporal features and support vector machines

10.1117/12.2269060 ◽

2017 ◽

Cited By ~ 1

Author(s):

Sawsen Ghabri ◽

Wael Ouarda ◽

Adel M. Alimi

Keyword(s):

Support Vector Machines ◽

Human Behavior ◽

Support Vector ◽

Behavior Recognition ◽

Temporal Features ◽

Vector Machines ◽

Spatio Temporal ◽

Human Behavior Recognition

Download Full-text

Spatio-Temporal Attention Model for Foreground Detection in Cross-Scene Surveillance Videos

Sensors ◽

10.3390/s19235142 ◽

2019 ◽

Vol 19 (23) ◽

pp. 5142 ◽

Cited By ~ 2

Author(s):

Dong Liang ◽

Jiaxing Pan ◽

Han Sun ◽

Huiyu Zhou

Keyword(s):

Optical Flow ◽

Feature Learning ◽

Foreground Detection ◽

Temporal Attention ◽

Frame Size ◽

Attention Model ◽

Flow Features ◽

Spatio Temporal ◽

High Level ◽

Optical Flow Features

Foreground detection is an important theme in video surveillance. Conventional background modeling approaches build sophisticated temporal statistical model to detect foreground based on low-level features, while modern semantic/instance segmentation approaches generate high-level foreground annotation, but ignore the temporal relevance among consecutive frames. In this paper, we propose a Spatio-Temporal Attention Model (STAM) for cross-scene foreground detection. To fill the semantic gap between low and high level features, appearance and optical flow features are synthesized by attention modules via the feature learning procedure. Experimental results on CDnet 2014 benchmarks validate it and outperformed many state-of-the-art methods in seven evaluation metrics. With the attention modules and optical flow, its F-measure increased 9 % and 6 % respectively. The model without any tuning showed its cross-scene generalization on Wallflower and PETS datasets. The processing speed was 10.8 fps with the frame size 256 by 256.

Download Full-text

A Novel Spatio-Temporal Violence Classification Framework Based on Material Derivative and LSTM Neural Network

Traitement du signal ◽

10.18280/ts.370501 ◽

2020 ◽

Vol 37 (5) ◽

pp. 687-701

Author(s):

Wafa Lejmi ◽

Anouar Ben Khalifa ◽

Mohamed Ali Mahjoub

Keyword(s):

Confusion Matrix ◽

Efficient Solutions ◽

Rate Of Change ◽

Surveillance Systems ◽

Material Derivative ◽

Classification Framework ◽

Temporal Features ◽

Proposed Model ◽

Normal Human ◽

Spatio Temporal

In the current era, the implementation of automated security video surveillance systems is particularly needy in terms of human violence recognition. Nevertheless, the latter encounters various interlinked difficulties which require efficient solutions as well as feasible methods that provide a relevant distinction between normal human actions and abnormal ones. In this paper, we present an overview of these issues and a literature review of the related works and current research on-going efforts on this field and suggests a novel prediction model for violence recognition, based on a preliminary spatio-temporal features extraction using the material derivative which describes the rate of change of a particle while in motion with respect to time. The classification algorithm is then carried out using a deep learning LSTM technique to classify generated features into eight specified violent and non-violent categories and a prediction value for each class of action is calculated. The whole model is trained on a public dataset and its classification capacity is evaluated on a confusion matrix which assembles all the predictions made by the system with their actual labels. The obtained results are promising and show that the proposed model can be potentially useful for detecting human violence.

Download Full-text