One‐class tensor machine with randomized projection for large‐scale anomaly detection in high‐dimensional and noisy data

2021 ◽  
Author(s):  
Imran Razzak ◽  
Nour Moustafa ◽  
Shahid Mumtaz ◽  
Guangdong Xu
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Noisy Data ◽  
High Dimensional

High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning

2016 ◽  
Vol 58 ◽  
pp. 121-134 ◽  
Author(s):  
Sarah M. Erfani ◽  
Sutharshan Rajasegarar ◽  
Shanika Karunasekera ◽  
Christopher Leckie
Keyword(s):  
Deep Learning ◽  
Anomaly Detection ◽  
Large Scale ◽  
High Dimensional

scholarly journals An Ensemble-Based Approach to Anomaly Detection in Marine Engine Sensor Streams for Efficient Condition Monitoring and Analysis

Sensors ◽  
2020 ◽  
Vol 20 (24) ◽  
pp. 7285
Author(s):  
Donghyun Kim ◽  
Sangbong Lee ◽  
Jihwan Lee
Keyword(s):  
Diesel Engine ◽  
Anomaly Detection ◽  
Large Scale ◽  
Abnormal Behavior ◽  
High Dimensional ◽  
Statistical Control ◽  
Marine Engine ◽  
Large Scale Dataset ◽  
Unsupervised Anomaly Detection ◽  
Sensor Streams

This study proposes an unsupervised anomaly detection method using sensor streams from the marine engine to detect the anomalous system behavior, which may be a possible sign of system failure. Previous works on marine engine anomaly detection proposed a clustering-based or statistical control chart-based approach that is unstable according to the choice of hyperparameters, or cannot fit well to the high-dimensional dataset. As a remedy to this limitation, this study adopts an ensemble-based approach to anomaly detection. The idea is to train several anomaly detectors with varying hyperparameters in parallel and then combine its result in the anomaly detection phase. Because the anomaly is detected by the combination of different detectors, it is robust to the choice of hyperparameters without loss of accuracy. To demonstrate our methodology, an actual dataset obtained from a 200,000-ton cargo vessel from a Korean shipping company that uses two-stroke diesel engine is analyzed. As a result, anomalies were successfully detected from the high-dimensional and large-scale dataset. After detecting the anomaly, clustering analysis was conducted to the anomalous observation to examine anomaly patterns. By investigating each cluster’s feature distribution, several common patterns of abnormal behavior were successfully visualized. Although we analyzed the data from two-stroke diesel engine, our method can be applied to various types of marine engine.

scholarly journals Effect Improved for High-Dimensional and Unbalanced Data Anomaly Detection Model Based on KNN-SMOTE-LSTM

Complexity ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-17
Author(s):  
Fuguang Bao ◽  
Yongqiang Wu ◽  
Zhaogang Li ◽  
Yongzhao Li ◽  
Lili Liu ◽  
...  
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Short Term Memory ◽  
Sequence Data ◽  
Binary Classification ◽  
High Dimensional ◽  
Unbalanced Data ◽  
K Nearest Neighbors ◽  
Detection Model ◽  
Model Based

High-dimensional and unbalanced data anomaly detection is common. Effective anomaly detection is essential for problem or disaster early warning and maintaining system reliability. A significant research issue related to the data analysis of the sensor is the detection of anomalies. The anomaly detection is essentially an unbalanced sequence binary classification. The data of this type contains characteristics of large scale, high complex computation, unbalanced data distribution, and sequence relationship among data. This paper uses long short-term memory networks (LSTMs) combined with historical sequence data; also, it integrates the synthetic minority oversampling technique (SMOTE) algorithm and K-nearest neighbors (kNN), and it designs and constructs an anomaly detection network model based on kNN-SMOTE-LSTM in accordance with the data characteristic of being unbalanced. This model can continuously filter out and securely generate samples to improve the performance of the model through kNN discriminant classifier and avoid the blindness and limitations of the SMOTE algorithm in generating new samples. The experiments demonstrated that the structured kNN-SMOTE-LSTM model can significantly improve the performance of the unbalanced sequence binary classification.

A Fast Clustering Algorithm for Large-scale and High Dimensional Data

2009 ◽  
Vol 35 (7) ◽  
pp. 859-866
Author(s):  
Ming LIU ◽  
Xiao-Long WANG ◽  
Yuan-Chao LIU
Keyword(s):  
Large Scale ◽  
Clustering Algorithm ◽  
High Dimensional Data ◽  
High Dimensional

Mitigating the Insider Threat with High-Dimensional Anomaly Detection

2004 ◽  
Author(s):  
S. Pramanick ◽  
S. Rajagopalan ◽  
Eric van den Berg
Keyword(s):  
Anomaly Detection ◽  
High Dimensional ◽  
Insider Threat

Large Scale Hierarchical Anomaly Detection and Temporal Localization

2020 ◽  
Author(s):  
Soumil Kanwal ◽  
Vineet Mehta ◽  
Abhinav Dhall
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Temporal Localization

scholarly journals Multi-task learning based Encoder-Decoder: A comprehensive detection and diagnosis system for multi-sensor data

2021 ◽  
Vol 13 (5) ◽  
pp. 168781402110131
Author(s):  
Junfeng Wu ◽  
Li Yao ◽  
Bin Liu ◽  
Zheyuan Ding ◽  
Lei Zhang
Keyword(s):  
Anomaly Detection ◽  
Event Detection ◽  
Large Scale ◽  
Multivariate Time Series ◽  
Sensor Data ◽  
Unified Framework ◽  
Diagnosis System ◽  
Learning Framework ◽  
Task Learning ◽  
Detection And Diagnosis

As more and more sensor data have been collected, automated detection, and diagnosis systems are urgently needed to lessen the increasing monitoring burden and reduce the risk of system faults. A plethora of researches have been done on anomaly detection, event detection, anomaly diagnosis respectively. However, none of current approaches can explore all these respects in one unified framework. In this work, a Multi-Task Learning based Encoder-Decoder (MTLED) which can simultaneously detect anomalies, diagnose anomalies, and detect events is proposed. In MTLED, feature matrix is introduced so that features are extracted for each time point and point-wise anomaly detection can be realized in an end-to-end way. Anomaly diagnosis and event detection share the same feature matrix with anomaly detection in the multi-task learning framework and also provide important information for system monitoring. To train such a comprehensive detection and diagnosis system, a large-scale multivariate time series dataset which contains anomalies of multiple types is generated with simulation tools. Extensive experiments on the synthetic dataset verify the effectiveness of MTLED and its multi-task learning framework, and the evaluation on a real-world dataset demonstrates that MTLED can be used in other application scenarios through transfer learning.

scholarly journals Large-scale empirical validation of Bayesian Network structure learning algorithms with noisy data

2021 ◽  
Vol 131 ◽  
pp. 151-188
Author(s):  
Anthony C. Constantinou ◽  
Yang Liu ◽  
Kiattikun Chobtham ◽  
Zhigao Guo ◽  
Neville K. Kitson
Keyword(s):  
Bayesian Network ◽  
Network Structure ◽  
Large Scale ◽  
Structure Learning ◽  
Learning Algorithms ◽  
Noisy Data ◽  
Empirical Validation ◽  
Bayesian Network Structure ◽  
Bayesian Network Structure Learning
Sign in / Sign up

Export Citation Format

Share Document