High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning

2016 ◽  
Vol 58 ◽  
pp. 121-134 ◽  
Author(s):  
Sarah M. Erfani ◽  
Sutharshan Rajasegarar ◽  
Shanika Karunasekera ◽  
Christopher Leckie
Keyword(s):  
Deep Learning ◽  
Anomaly Detection ◽  
Large Scale ◽  
High Dimensional

scholarly journals A Hybrid Deep Learning-Based Unsupervised Anomaly Detection in High Dimensional Data

2022 ◽  
Vol 70 (3) ◽  
pp. 5363-5381
Author(s):  
Amgad Muneer ◽  
Shakirah Mohd Taib ◽  
Suliman Mohamed Fati ◽  
Abdullateef O. Balogun ◽  
Izzatdin Abdul Aziz
Keyword(s):  
Deep Learning ◽  
Anomaly Detection ◽  
High Dimensional Data ◽  
High Dimensional ◽  
Unsupervised Anomaly Detection

scholarly journals An Ensemble-Based Approach to Anomaly Detection in Marine Engine Sensor Streams for Efficient Condition Monitoring and Analysis

Sensors ◽  
2020 ◽  
Vol 20 (24) ◽  
pp. 7285
Author(s):  
Donghyun Kim ◽  
Sangbong Lee ◽  
Jihwan Lee
Keyword(s):  
Diesel Engine ◽  
Anomaly Detection ◽  
Large Scale ◽  
Abnormal Behavior ◽  
High Dimensional ◽  
Statistical Control ◽  
Marine Engine ◽  
Large Scale Dataset ◽  
Unsupervised Anomaly Detection ◽  
Sensor Streams

This study proposes an unsupervised anomaly detection method using sensor streams from the marine engine to detect the anomalous system behavior, which may be a possible sign of system failure. Previous works on marine engine anomaly detection proposed a clustering-based or statistical control chart-based approach that is unstable according to the choice of hyperparameters, or cannot fit well to the high-dimensional dataset. As a remedy to this limitation, this study adopts an ensemble-based approach to anomaly detection. The idea is to train several anomaly detectors with varying hyperparameters in parallel and then combine its result in the anomaly detection phase. Because the anomaly is detected by the combination of different detectors, it is robust to the choice of hyperparameters without loss of accuracy. To demonstrate our methodology, an actual dataset obtained from a 200,000-ton cargo vessel from a Korean shipping company that uses two-stroke diesel engine is analyzed. As a result, anomalies were successfully detected from the high-dimensional and large-scale dataset. After detecting the anomaly, clustering analysis was conducted to the anomalous observation to examine anomaly patterns. By investigating each cluster’s feature distribution, several common patterns of abnormal behavior were successfully visualized. Although we analyzed the data from two-stroke diesel engine, our method can be applied to various types of marine engine.

One‐class tensor machine with randomized projection for large‐scale anomaly detection in high‐dimensional and noisy data

2021 ◽  
Author(s):  
Imran Razzak ◽  
Nour Moustafa ◽  
Shahid Mumtaz ◽  
Guangdong Xu
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Noisy Data ◽  
High Dimensional

scholarly journals A study on Anomaly Detection GAN-based methods on image data

2019 ◽  
Author(s):  
Emanuel Silva ◽  
Johannes Lochter
Keyword(s):  
Deep Learning ◽  
Anomaly Detection ◽  
Image Data ◽  
Original Method ◽  
Detection Task ◽  
Generative Adversarial Networks ◽  
High Dimensional ◽  
Complex Data ◽  
New Techniques ◽  
Adversarial Networks

The anomaly detection task is a well know problem being researched among a variety of areas, including machine learning. The task is to identify data patterns that have a non expected behaviour, that can be a malicious data sent by an attacker or a unexpected valid behaviour, in both cases the anomaly need to be identified. With the advance of deep learning based techniques showing that this class of algorithms can learn high-dimensional and complex data patterns, naturally it became an option to the anomaly detection task. Recent researches in literature are using a sub-field of deep learning algorithms named Generative Adversarial Networks for predicting anomalous samples, since the original method can learn the data distribution. These new techniques make some changes for the anomaly detection task, and this work provides a briefly review on these methods and provides a comparison with well known methods.

scholarly journals Effect Improved for High-Dimensional and Unbalanced Data Anomaly Detection Model Based on KNN-SMOTE-LSTM

Complexity ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-17
Author(s):  
Fuguang Bao ◽  
Yongqiang Wu ◽  
Zhaogang Li ◽  
Yongzhao Li ◽  
Lili Liu ◽  
...  
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Short Term Memory ◽  
Sequence Data ◽  
Binary Classification ◽  
High Dimensional ◽  
Unbalanced Data ◽  
K Nearest Neighbors ◽  
Detection Model ◽  
Model Based

High-dimensional and unbalanced data anomaly detection is common. Effective anomaly detection is essential for problem or disaster early warning and maintaining system reliability. A significant research issue related to the data analysis of the sensor is the detection of anomalies. The anomaly detection is essentially an unbalanced sequence binary classification. The data of this type contains characteristics of large scale, high complex computation, unbalanced data distribution, and sequence relationship among data. This paper uses long short-term memory networks (LSTMs) combined with historical sequence data; also, it integrates the synthetic minority oversampling technique (SMOTE) algorithm and K-nearest neighbors (kNN), and it designs and constructs an anomaly detection network model based on kNN-SMOTE-LSTM in accordance with the data characteristic of being unbalanced. This model can continuously filter out and securely generate samples to improve the performance of the model through kNN discriminant classifier and avoid the blindness and limitations of the SMOTE algorithm in generating new samples. The experiments demonstrated that the structured kNN-SMOTE-LSTM model can significantly improve the performance of the unbalanced sequence binary classification.

scholarly journals Social Multimedia Security and Suspicious Activity Detection in SDN using Hybrid Deep Learning Technique

2020 ◽  
Vol 2 (2) ◽  
pp. 108-115
Author(s):  
Dr. Joy Iong Zong Chen ◽  
Dr. Smys S.
Keyword(s):  
Deep Learning ◽  
Anomaly Detection ◽  
Large Scale ◽  
Multimedia Data ◽  
Support Vector ◽  
Data Delivery ◽  
Energy Aware ◽  
Detection Scheme ◽  
Social Multimedia

Social multimedia traffic is growing exponentially with the increased usage and continuous development of services and applications based on multimedia. Quality of Service (QoS), Quality of Information (QoI), scalability, reliability and such factors that are essential for social multimedia networks are realized by secure data transmission. For delivering actionable and timely insights in order to meet the growing demands of the user, multimedia analytics is performed by means of a trust-based paradigm. Efficient management and control of the network is facilitated by limiting certain capabilities such as energy-aware networking and runtime security in Software Defined Networks. In social multimedia context, suspicious flow detection is performed by a hybrid deep learning based anomaly detection scheme in order to enhance the SDN reliability. The entire process is divided into two modules namely – Abnormal activities detection using support vector machine based on Gradient descent and improved restricted Boltzmann machine which facilitates the anomaly detection module, and satisfying the strict requirements of QoS like low latency and high bandwidth in SDN using end-to-end data delivery module. In social multimedia, data delivery and anomaly detection services are essential in order to improve the efficiency and effectiveness of the system. For this purpose, we use benchmark datasets as well as real time evaluation to experimentally evaluate the proposed scheme. Detection of malicious events like confidential data collection, profile cloning and identity theft are performed to analyze the performance of the system using CMU-based insider threat dataset for large scale analysis.

Anomaly Detection and Categorization in Cloud Environment using Deep Learning Techniques

2019 ◽  
Vol 7 (5) ◽  
pp. 211-214
Author(s):  
Nidhi Thakkar ◽  
Miren Karamta ◽  
Seema Joshi ◽  
M. B. Potdar
Keyword(s):  
Deep Learning ◽  
Anomaly Detection ◽  
Cloud Environment ◽  
Learning Techniques

A Fast Clustering Algorithm for Large-scale and High Dimensional Data

2009 ◽  
Vol 35 (7) ◽  
pp. 859-866
Author(s):  
Ming LIU ◽  
Xiao-Long WANG ◽  
Yuan-Chao LIU
Keyword(s):  
Large Scale ◽  
Clustering Algorithm ◽  
High Dimensional Data ◽  
High Dimensional
Sign in / Sign up

Export Citation Format

Share Document