scholarly journals Abnormal Detection in Big Data Video with an Improved Autoencoder

2021 ◽  
Vol 2021 ◽  
pp. 1-6
Author(s):  
Yihan Bian ◽  
Xinchen Tang
Keyword(s):  
Big Data ◽  
Anomaly Detection ◽  
Large Scale ◽  
Test Phase ◽  
Video Data ◽  
Detection Methods ◽  
Memory Module ◽  
Abnormal Detection ◽  
Memory Content ◽  
Increasing Demand

With the rapid growth of video surveillance data, there is an increasing demand for big data automatic anomaly detection of large-scale video data. The detection methods using reconstruction errors based on deep autoencoders have been widely discussed. However, sometimes the autoencoder could reconstruct the anomaly well and lead to missing detections. In order to solve this problem, this paper uses a memory module to enhance the autoencoder, which is called the memory-augmented autoencoder (Memory AE) method. Given the input, Memory AE first obtains the code from the encoder and then uses it as a query to retrieve the most relevant memory items for reconstruction. In the training phase, the memory content is updated and encouraged to represent prototype elements of normal data. In the test phase, the learned memory elements are fixed, and reconstruction is obtained from several selected memory records of normal data. So, the reconstruction will tend to be close to normal samples. Therefore, the reconstruction of abnormal errors will be strengthened for abnormal detection. The experimental results on two public video anomaly detection datasets, i.e., Avenue dataset and ShanghaiTech dataset, prove the effectiveness of the proposed method.

Using Exploratory Data Analysis and Big Data Analytics for Detecting Anomalies in Cloud Computing

2021 ◽  
Vol 3 (1) ◽  
Author(s):  
Ibrahim Muzaferija ◽  
Zerina Mašetić ◽  
Keyword(s):  
Cloud Computing ◽  
Big Data ◽  
Data Analysis ◽  
Anomaly Detection ◽  
Data Analytics ◽  
Large Scale ◽  
Exploratory Data Analysis ◽  
Big Data Analytics ◽  
Detection Methods ◽  
Exploratory Data

While leveraging cloud computing for large-scale distributed applications allows seamless scaling, many companies struggle following up with the amount of data generated in terms of efficient processing and anomaly detection, which is a necessary part of the management of modern applications. As the record of user behavior, weblogs surely become the research item related to anomaly detection. Many anomaly detection methods based on automated log analysis have been proposed. However, not in the context of big data applications where anomalous behavior needs to be detected in understanding phases prior to modeling a system for such use. Big Data Analytics often ignores anomalous point due to high volume of data. To address this problem, we propose a complemented methodology for Big Data Analytics – the Exploratory Data Analysis, which assists in gaining insight into data relationships without the classical hypothesis modeling. In that way, we can gain better understanding of the patterns and spot anomalies. Results show that Exploratory Data Analysis facilitates anomaly detection and the CRISP-DM Business Understanding phase, making it one of the key steps in the Data Understanding phase.

SAND

2021 ◽  
Vol 14 (10) ◽  
pp. 1717-1729
Author(s):  
Paul Boniol ◽  
John Paparrizos ◽  
Themis Palpanas ◽  
Michael J. Franklin
Keyword(s):  
Anomaly Detection ◽  
Domain Knowledge ◽  
State Of The Art ◽  
Data Distribution ◽  
Detection Methods ◽  
Current State ◽  
Normal Behavior ◽  
Real World Datasets ◽  
Increasing Demand ◽  
Entire Dataset

With the increasing demand for real-time analytics and decision making, anomaly detection methods need to operate over streams of values and handle drifts in data distribution. Unfortunately, existing approaches have severe limitations: they either require prior domain knowledge or become cumbersome and expensive to use in situations with recurrent anomalies of the same type. In addition, subsequence anomaly detection methods usually require access to the entire dataset and are not able to learn and detect anomalies in streaming settings. To address these problems, we propose SAND, a novel online method suitable for domain-agnostic anomaly detection. SAND aims to detect anomalies based on their distance to a model that represents normal behavior. SAND relies on a novel steaming methodology to incrementally update such model, which adapts to distribution drifts and omits obsolete data. The experimental results on several real-world datasets demonstrate that SAND correctly identifies single and recurrent anomalies without prior knowledge of the characteristics of these anomalies. SAND outperforms by a large margin the current state-of-the-art algorithms in terms of accuracy while achieving orders of magnitude speedups.

scholarly journals Recent Advancement in Anomaly Detection in Surveillance Videos

2019 ◽  
Vol 8 (2) ◽  
pp. 964-970
Keyword(s):  
Computer Vision ◽  
Anomaly Detection ◽  
Video Data ◽  
Detection Methods ◽  
Future Research ◽  
Detection Techniques ◽  
Advantages And Disadvantages ◽  
Recent Computer ◽  
Banking Education ◽  
To Come

Detection of Anomaly is of a notable and emergent problem into many diverse fields like information theory, deep learning, computer vision, machine learning, and statistics that have been researched within the various application from diverse domains including agriculture, health care, banking, education, and transport anomaly detection. Newly, numbers of important anomaly detection techniques along with diverseness of sort have been watched. The main aim of this paper to come up with a broad summary of the present development on detection of an anomaly, exclusively for video data with mixed types and high dimensionalities, where identifying the anomalous behaviors and event or anomalous patterns is a significant task. The paper expresses the advantages and disadvantages of the detection methods the experiments tried on the publically available benchmark dataset to assess numerous popular and classical methods and models. The objective of this analysis is to furnish an understanding of recent computer vision and machine algorithms methods and also state-of-the-art deep learnings techniques to detect anomalies for researchers. At last, the paper delivered roughly directions for future research on an anomalies detection.

scholarly journals ROI-based efficient video data processing for large-scale cloud storage in intelligent CCTV environment

2018 ◽  
Vol 7 (3.3) ◽  
pp. 151 ◽  
Author(s):  
Dong Hyeok Lee ◽  
Nam Je. Park
Keyword(s):  
Big Data ◽  
Cloud Storage ◽  
Large Scale ◽  
Image Data ◽  
Processing Technique ◽  
Video Data ◽  
Image Processing Technique ◽  
Efficient Data ◽  
Transmission Structure ◽  
Cloud Storage Environment

Background/Objectives: Big data environment is being realized. Recently, intelligent public safety environment on the foundation of the image processing technique based on big data is being introduced, and accordingly, processing CCTV images is becoming more important day by day.Methods/Statistical analysis: In this paper, an efficient technique to send image information for mass cloud storage environment was proposed. With the offered method, only the ROI area is extracted and partial object images are transmitted, and it has the strengths of higher efficiency and protected privacy with the application of a masking technique.Findings: it is general to apply the masking technique partially to face information, and in this study, the privacy of the image data registered in the cloud storage was to be protected based on this masking technique, and an efficient data transmission structure grounded on ROI area extraction was proposed.Improvements/Applications: With the offered method, only the ROI area is extracted and partial object images are transmitted, and it has the strengths of higher efficiency and protected privacy with the application of a masking technique.  

scholarly journals Hyperspectral Anomaly Detection Based on Separability-Aware Sample Cascade

2019 ◽  
Vol 11 (21) ◽  
pp. 2537 ◽  
Author(s):  
Dandan Ma ◽  
Yuan Yuan ◽  
Qi Wang
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Hyperspectral Image ◽  
Superior Performance ◽  
Detection Methods ◽  
Background Information ◽  
Detection Accuracy ◽  
Layer By Layer ◽  
Edge Preserving ◽  
Image Pixels

A hyperspectral image usually covers a large scale of ground scene, which contains various materials with different spectral properties. When directly exploring the background information using all the image pixels, complex spectral interactions and inter-/intra-difference of different samples will significantly reduce the accuracy of background evaluation and further affect the detection performance. To address this problem, this paper proposes a novel hyperspectral anomaly detection method based on separability-aware sample cascade model. Through identifying separability of hyperspectral pixels, background samples are sifted out layer-by-layer according to their separable degrees from anomalies, which can ensure the accuracy and distinctiveness of background representation. First, as spatial structure is beneficial for recognizing target, a new spectral–spatial feature extraction technique is used in this work based on the PCA technique and edge-preserving filtering. Second, depending on different separability computed by sparse representation, samples are separated into different sets which can effectively and completely reflect various characteristics of background across all the cascade layers. Meanwhile, some potential abnormal targets are removed at each selection step to avoid their effects on subsequent layers. Finally, comprehensively taking different good properties of all the separability-aware layers into consideration, a simple multilayer anomaly detection strategy is adopted to obtain the final detection map. Extensive experimental results on five real-world hyperspectral images demonstrate our method’s superior performance. Compared with seven representative anomaly detection methods, our method improves the average detection accuracy with great advantages.

scholarly journals Towards Large-Scale, Heterogeneous Anomaly Detection Systems in Industrial Networks: A Survey of Current Trends

2017 ◽  
Vol 2017 ◽  
pp. 1-17 ◽  
Author(s):  
Mikel Iturbe ◽  
Iñaki Garitano ◽  
Urko Zurutuza ◽  
Roberto Uribeetxeberria
Keyword(s):  
Big Data ◽  
Anomaly Detection ◽  
Cyber Security ◽  
Large Scale ◽  
Big Data Analytics ◽  
Research Field ◽  
Open Problems ◽  
Detection Systems ◽  
Industrial Networks ◽  
Active Research

Industrial Networks (INs) are widespread environments where heterogeneous devices collaborate to control and monitor physical processes. Some of the controlled processes belong to Critical Infrastructures (CIs), and, as such, IN protection is an active research field. Among different types of security solutions, IN Anomaly Detection Systems (ADSs) have received wide attention from the scientific community. While INs have grown in size and in complexity, requiring the development of novel, Big Data solutions for data processing, IN ADSs have not evolved at the same pace. In parallel, the development of Big Data frameworks such as Hadoop or Spark has led the way for applying Big Data Analytics to the field of cyber-security, mainly focusing on the Information Technology (IT) domain. However, due to the particularities of INs, it is not feasible to directly apply IT security mechanisms in INs, as IN ADSs face unique characteristics. In this work we introduce three main contributions. First, we survey the area of Big Data ADSs that could be applicable to INs and compare the surveyed works. Second, we develop a novel taxonomy to classify existing IN-based ADSs. And, finally, we present a discussion of open problems in the field of Big Data ADSs for INs that can lead to further development.

scholarly journals Improved Mitigation of Cyber Threats in IIoT for Smart Cities: A New-Era Approach and Scheme

Sensors ◽  
2021 ◽  
Vol 21 (6) ◽  
pp. 1976
Author(s):  
Semi Park ◽  
Kyungho Lee
Keyword(s):  
Random Forest ◽  
Anomaly Detection ◽  
Large Scale ◽  
High Efficiency ◽  
Smart Cities ◽  
Role Playing ◽  
Anomalous Behavior ◽  
Detection Methods ◽  
Game Industry ◽  
Bot Detection

Cybersecurity in Industrial Internet of Things (IIoT) has become critical as smart cities are becoming increasingly linked to industrial control systems (ICSs) used in critical infrastructure. Consequently, data-driven security systems for analyzing massive amounts of data generated by smart cities have become essential. A representative method for analyzing large-scale data is the game bot detection approach used in massively multiplayer online role-playing games. We reviewed the literature on bot detection methods to extend the anomaly detection approaches used in bot detection schemes to IIoT fields. Finally, we proposed a process wherein the data envelopment analysis (DEA) model was applied to identify features for efficiently detecting anomalous behavior in smart cities. Experimental results using random forest show that our extracted features based on a game bot can achieve an average F1-score of 0.99903 using 10-fold validation. We confirmed the applicability of the analyzed game-industry methodology to other fields and trained a random forest on the high-efficiency features identified by applying a DEA, obtaining an F1-score of 0.997 using the validation set approach. In this study, an anomaly detection method for analyzing massive smart city data based on a game industry methodology was presented and applied to the ICS dataset.

scholarly journals A Review of Video Object Detection: Datasets, Metrics and Methods

2020 ◽  
Vol 10 (21) ◽  
pp. 7834
Author(s):  
Haidi Zhu ◽  
Haoran Wei ◽  
Baoqing Li ◽  
Xiaobing Yuan ◽  
Nasser Kehtarnavaz
Keyword(s):  
Object Detection ◽  
Computational Efficiency ◽  
Large Scale ◽  
Visual Recognition ◽  
Motion Blur ◽  
Video Data ◽  
Detection Methods ◽  
Video Object ◽  
Static Images ◽  
Temporal And Spatial

Although there are well established object detection methods based on static images, their application to video data on a frame by frame basis faces two shortcomings: (i) lack of computational efficiency due to redundancy across image frames or by not using a temporal and spatial correlation of features across image frames, and (ii) lack of robustness to real-world conditions such as motion blur and occlusion. Since the introduction of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2015, a growing number of methods have appeared in the literature on video object detection, many of which have utilized deep learning models. The aim of this paper is to provide a review of these papers on video object detection. An overview of the existing datasets for video object detection together with commonly used evaluation metrics is first presented. Video object detection methods are then categorized and a description of each of them is stated. Two comparison tables are provided to see their differences in terms of both accuracy and computational efficiency. Finally, some future trends in video object detection to address the challenges involved are noted.

Sign in / Sign up

Export Citation Format

Share Document