Generative and Autoencoder Models for Large-Scale Mutivariate Unsupervised Anomaly Detection

2021 ◽  
pp. 45-58
Author(s):  
Nabila Ounasser ◽  
Maryem Rhanoui ◽  
Mounia Mikram ◽  
Bouchra El Asri
Sensors ◽  
2020 ◽  
Vol 20 (24) ◽  
pp. 7285
Author(s):  
Donghyun Kim ◽  
Sangbong Lee ◽  
Jihwan Lee

This study proposes an unsupervised anomaly detection method using sensor streams from the marine engine to detect the anomalous system behavior, which may be a possible sign of system failure. Previous works on marine engine anomaly detection proposed a clustering-based or statistical control chart-based approach that is unstable according to the choice of hyperparameters, or cannot fit well to the high-dimensional dataset. As a remedy to this limitation, this study adopts an ensemble-based approach to anomaly detection. The idea is to train several anomaly detectors with varying hyperparameters in parallel and then combine its result in the anomaly detection phase. Because the anomaly is detected by the combination of different detectors, it is robust to the choice of hyperparameters without loss of accuracy. To demonstrate our methodology, an actual dataset obtained from a 200,000-ton cargo vessel from a Korean shipping company that uses two-stroke diesel engine is analyzed. As a result, anomalies were successfully detected from the high-dimensional and large-scale dataset. After detecting the anomaly, clustering analysis was conducted to the anomalous observation to examine anomaly patterns. By investigating each cluster’s feature distribution, several common patterns of abnormal behavior were successfully visualized. Although we analyzed the data from two-stroke diesel engine, our method can be applied to various types of marine engine.


Information ◽  
2020 ◽  
Vol 11 (2) ◽  
pp. 105 ◽  
Author(s):  
Limengwei Liu ◽  
Modi Hu ◽  
Chaoqun Kang ◽  
Xiaoyong Li

The development and integration of information technology and industrial control networks have expanded the magnitude of new data; detecting anomalies or discovering other valid information from them is of vital importance to the stable operation of industrial control systems. This paper proposes an incremental unsupervised anomaly detection method that can quickly analyze and process large-scale real-time data. Our evaluation on the Secure Water Treatment dataset shows that the method is converging to its offline counterpart for infinitely growing data streams.


2021 ◽  
Vol 2021 ◽  
pp. 1-14
Author(s):  
Wei Zhou ◽  
Yuan Gao ◽  
Jianhang Ji ◽  
Shicheng Li ◽  
Yugen Yi

With the rapid development of high tech, Internet of Things (IoT) and artificial intelligence (AI) achieve a series of achievements in the healthcare industry. Among them, automatic glaucoma diagnosis is one of them. Glaucoma is second leading cause of blindness in the world. Although many automatic glaucoma diagnosis approaches have been proposed, they still face the following two challenges. First, the data acquisition of diseased images is extremely expensive, especially for disease with low occurrence, leading to the class imbalance. Second, large-scale labeled data are hard to obtain in medical image domain. The aforementioned challenges limit the practical application of these approaches in glaucoma diagnosis. To address these disadvantages, this paper proposes an unsupervised anomaly detection framework based on sparse principal component analysis (SPCA) for glaucoma diagnosis. In the proposed approach, we just employ the one-class normal (nonglaucoma) images for training, so the class imbalance problem can be avoided. Then, to distinguish the glaucoma (abnormal) images from the normal images, a feature set consisting of segmentation-based features and image-based features is extracted, which can capture the shape and textural changes. Next, SPCA is adopted to select the effective features from the feature set. Finally, with the usage of the extracted effective features, glaucoma diagnosis can be automatically accomplished via introducing the T 2 statistic and the control limit, overcoming the issue of insufficient labeled samples. Extensive experiments are carried out on the two public databases, and the experimental results verify the effectiveness of the proposed approach.


Author(s):  
Bin Zhou ◽  
Shenghua Liu ◽  
Bryan Hooi ◽  
Xueqi Cheng ◽  
Jing Ye

Given a large-scale rhythmic time series containing mostly normal data segments (or `beats'), can we learn how to detect anomalous beats in an effective yet efficient way? For example, how can we detect anomalous beats from electrocardiogram (ECG) readings? Existing approaches either require excessively high amounts of labeled and balanced data for classification, or rely on less regularized reconstructions, resulting in lower accuracy in anomaly detection. Therefore, we propose BeatGAN, an unsupervised anomaly detection algorithm for time series data. BeatGAN outputs explainable results to pinpoint the anomalous time ticks of an input beat, by comparing them to adversarially generated beats. Its robustness is guaranteed by its regularization of reconstruction error using an adversarial generation approach, as well as data augmentation using time series warping. Experiments show that BeatGAN accurately and efficiently detects anomalous beats in ECG time series, and routes doctors' attention to anomalous time ticks, achieving accuracy of nearly 0.95 AUC, and very fast inference (2.6 ms per beat). In addition, we show that BeatGAN accurately detects unusual motions from multivariate motion-capture time series data, illustrating its generality.


Sign in / Sign up

Export Citation Format

Share Document