Abnormal Activity Recognition from Surveillance Videos Using Convolutional Neural Network

Background and motivation: Every year, millions of Muslims worldwide come to Mecca to perform the Hajj. In order to maintain the security of the pilgrims, the Saudi government has installed about 5000 closed circuit television (CCTV) cameras to monitor crowd activity efficiently. Problem: As a result, these cameras generate an enormous amount of visual data through manual or offline monitoring, requiring numerous human resources for efficient tracking. Therefore, there is an urgent need to develop an intelligent and automatic system in order to efficiently monitor crowds and identify abnormal activity. Method: The existing method is incapable of extracting discriminative features from surveillance videos as pre-trained weights of different architectures were used. This paper develops a lightweight approach for accurately identifying violent activity in surveillance environments. As the first step of the proposed framework, a lightweight CNN model is trained on our own pilgrim’s dataset to detect pilgrims from the surveillance cameras. These preprocessed salient frames are passed to a lightweight CNN model for spatial features extraction in the second step. In the third step, a Long Short Term Memory network (LSTM) is developed to extract temporal features. Finally, in the last step, in the case of violent activity or accidents, the proposed system will generate an alarm in real time to inform law enforcement agencies to take appropriate action, thus helping to avoid accidents and stampedes. Results: We have conducted multiple experiments on two publicly available violent activity datasets, such as Surveillance Fight and Hockey Fight datasets; our proposed model achieved accuracies of 81.05 and 98.00, respectively.

Download Full-text

Deepfake video detection: YOLO-Face convolution recurrent approach

PeerJ Computer Science ◽

10.7717/peerj-cs.730 ◽

2021 ◽

Vol 7 ◽

pp. e730

Author(s):

Aya Ismail ◽

Marwa Elpeltagy ◽

Mervat Zaki ◽

Kamal A. ElDahshan

Keyword(s):

Large Scale ◽

Operating Characteristic ◽

Short Term Memory ◽

Negative Impact ◽

Characteristic Curve ◽

Spatial Features ◽

Temporal Features ◽

Large Scale Dataset ◽

Long Short Term Memory ◽

Face Detector

Recently, the deepfake techniques for swapping faces have been spreading, allowing easy creation of hyper-realistic fake videos. Detecting the authenticity of a video has become increasingly critical because of the potential negative impact on the world. Here, a new project is introduced; You Only Look Once Convolution Recurrent Neural Networks (YOLO-CRNNs), to detect deepfake videos. The YOLO-Face detector detects face regions from each frame in the video, whereas a fine-tuned EfficientNet-B5 is used to extract the spatial features of these faces. These features are fed as a batch of input sequences into a Bidirectional Long Short-Term Memory (Bi-LSTM), to extract the temporal features. The new scheme is then evaluated on a new large-scale dataset; CelebDF-FaceForencics++ (c23), based on a combination of two popular datasets; FaceForencies++ (c23) and Celeb-DF. It achieves an Area Under the Receiver Operating Characteristic Curve (AUROC) 89.35% score, 89.38% accuracy, 83.15% recall, 85.55% precision, and 84.33% F1-measure for pasting data approach. The experimental analysis approves the superiority of the proposed method compared to the state-of-the-art methods.

Download Full-text

Dynamic Displacement Forecasting of Dashuitian Landslide in China Using Variational Mode Decomposition and Stack Long Short-Term Memory Network

Applied Sciences ◽

10.3390/app9152951 ◽

2019 ◽

Vol 9 (15) ◽

pp. 2951 ◽

Cited By ~ 4

Author(s):

Yin Xing ◽

Jianping Yue ◽

Chuang Chen ◽

Kanglin Cong ◽

Shaolin Zhu ◽

...

Keyword(s):

Short Term Memory ◽

Forecast Accuracy ◽

Short Term ◽

Dynamic Displacement ◽

Term Memory ◽

Mode Decomposition ◽

Proposed Model ◽

Memory Network ◽

Long Short Term Memory ◽

Lstm Network

In recent decades, landslide displacement forecasting has received increasing attention due to its ability to reduce landslide hazards. To improve the forecast accuracy of landslide displacement, a dynamic forecasting model based on variational mode decomposition (VMD) and a stack long short-term memory network (SLSTM) is proposed. VMD is used to decompose landslide displacement into different displacement subsequences, and the SLSTM network is used to forecast each displacement subsequence. Then, the forecast values of landslide displacement are obtained by reconstructing the forecast values of all displacement subsequences. On the other hand, the SLSTM networks are updated by adding the forecast values into the training set, realizing the dynamic displacement forecasting. The proposed model was verified on the Dashuitian landslide in China. The results show that compared with the two advanced forecasting models, long short-term memory (LSTM) network, and empirical mode decomposition (EMD)–LSTM network, the proposed model has higher forecast accuracy.

Download Full-text

Long Short-Term Memory Neural Network Applied to Train Dynamic Model and Speed Prediction

Algorithms ◽

10.3390/a12080173 ◽

2019 ◽

Vol 12 (8) ◽

pp. 173 ◽

Cited By ~ 4

Author(s):

Zhen Li ◽

Tao Tang ◽

Chunhai Gao

Keyword(s):

Dynamic Model ◽

Short Term Memory ◽

Fundamental Problem ◽

Learning Ability ◽

Railway Transportation ◽

Short Term ◽

Term Memory ◽

Proposed Model ◽

Memory Network ◽

Long Short Term Memory

The automatic train operation system is a significant component of the intelligent railway transportation. As a fundamental problem, the construction of the train dynamic model has been extensively researched using parametric approaches. The parametric based models may have poor performances due to unrealistic assumptions and changeable environments. In this paper, a long short-term memory network is carefully developed to build the train dynamic model in a nonparametric way. By optimizing the hyperparameters of the proposed model, more accurate outputs can be obtained with the same inputs of the parametric approaches. The proposed model was compared with two parametric methods using actual data. Experimental results suggest that the model performance is better than those of traditional models due to the strong learning ability. By exploring a detailed feature engineering process, the proposed long short-term memory network based algorithm was extended to predict train speed for multiple steps ahead.

Download Full-text

Hierarchical Spatial-Spectral Feature Extraction with Long Short Term Memory (LSTM) for Mineral Identification Using Hyperspectral Imagery

Sensors ◽

10.3390/s20236854 ◽

2020 ◽

Vol 20 (23) ◽

pp. 6854

Author(s):

Huijie Zhao ◽

Kewang Deng ◽

Na Li ◽

Ziwei Wang ◽

Wei Wei

Keyword(s):

Feature Extraction ◽

Short Term Memory ◽

Spectral Feature ◽

Spectral Features ◽

Short Term ◽

Term Memory ◽

Spatial Features ◽

Proposed Model ◽

Long Short Term Memory ◽

Mineral Identification

Deep learning models are widely employed in hyperspectral image processing to integrate both spatial features and spectral features, but the correlations between them are rarely taken into consideration. However, in hyperspectral mineral identification, not only the spectral and spatial features of minerals need to be considered, but also the correlations between them are crucial to further promote identification accuracy. In this paper, we propose hierarchical spatial-spectral feature extraction with long short term memory (HSS-LSTM) to explore correlations between spatial features and spectral features and obtain hierarchical intrinsic features for mineral identification. In the proposed model, the fusion spatial-spectral feature is primarily extracted by stacking local spatial features obtained by a convolution neural network (CNN)-based model and spectral information together. To better exploit spatial features and spectral features, an LSTM-based model is proposed to capture correlations and obtain hierarchical features for accurate mineral identification. Specifically, the proposed model shares a uniform objective function, so that all the parameters in the network can be optimized in the meantime. Experimental results on the hyperspectral data collected by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) in the Nevada mining area show that HSS-LSTM achieves an overall accuracy of 94.70% and outperforms other commonly used identification methods.

Download Full-text

Multi-Step Ahead Short-Term Load Forecasting Using Hybrid Feature Selection and Improved Long Short-Term Memory Network

Energies ◽

10.3390/en13164121 ◽

2020 ◽

Vol 13 (16) ◽

pp. 4121

Author(s):

Shaoqian Pei ◽

Hui Qin ◽

Liqiang Yao ◽

Yongqi Liu ◽

Chao Wang ◽

...

Keyword(s):

Feature Selection ◽

Short Term Memory ◽

Load Forecasting ◽

Dew Point ◽

Short Term ◽

Term Memory ◽

Proposed Model ◽

Short Term Load Forecasting ◽

Memory Network ◽

Long Short Term Memory

Short-term load forecasting (STLF) plays an important role in the economic dispatch of power systems. Obtaining accurate short-term load can greatly improve the safety and economy of a power grid operation. In recent years, a large number of short-term load forecasting methods have been proposed. However, how to select the optimal feature set and accurately predict multi-step ahead short-term load still faces huge challenges. In this paper, a hybrid feature selection method is proposed, an Improved Long Short-Term Memory network (ILSTM) is applied to predict multi-step ahead load. This method firstly takes the influence of temperature, humidity, dew point, and date type on the load into consideration. Furthermore, the maximum information coefficient is used for the preliminary screening of historical load, and Max-Relevance and Min-Redundancy (mRMR) is employed for further feature selection. Finally, the selected feature set is considered as input of the model to perform multi-step ahead short-term load prediction by the Improved Long Short-Term Memory network. In order to verify the performance of the proposed model, two categories of contrast methods are applied: (1) comparing the model with hybrid feature selection and the model which does not adopt hybrid feature selection; (2) comparing different models including Long Short-Term Memory network (LSTM), Gated Recurrent Unit (GRU), and Support Vector Regression (SVR) using hybrid feature selection. The result of the experiments, which were developed during four periods in the Hubei Province, China, show that hybrid feature selection can improve the prediction accuracy of the model, and the proposed model can accurately predict the multi-step ahead load.

Download Full-text

A combined convolutional and recurrent neural network for enhanced glaucoma detection

Scientific Reports ◽

10.1038/s41598-021-81554-4 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Soheila Gheisari ◽

Sahar Shariflou ◽

Jack Phu ◽

Paul J. Kennedy ◽

Ashish Agar ◽

...

Keyword(s):

Neural Network ◽

Recurrent Neural Network ◽

Short Term Memory ◽

Fundus Images ◽

Spatial Features ◽

Glaucoma Detection ◽

Temporal Features ◽

Long Short Term Memory ◽

Sequential Images ◽

F Measure

AbstractGlaucoma, a leading cause of blindness, is a multifaceted disease with several patho-physiological features manifesting in single fundus images (e.g., optic nerve cupping) as well as fundus videos (e.g., vascular pulsatility index). Current convolutional neural networks (CNNs) developed to detect glaucoma are all based on spatial features embedded in an image. We developed a combined CNN and recurrent neural network (RNN) that not only extracts the spatial features in a fundus image but also the temporal features embedded in a fundus video (i.e., sequential images). A total of 1810 fundus images and 295 fundus videos were used to train a CNN and a combined CNN and Long Short-Term Memory RNN. The combined CNN/RNN model reached an average F-measure of 96.2% in separating glaucoma from healthy eyes. In contrast, the base CNN model reached an average F-measure of only 79.2%. This proof-of-concept study demonstrates that extracting spatial and temporal features from fundus videos using a combined CNN and RNN, can markedly enhance the accuracy of glaucoma detection.

Download Full-text