A Comparative Study on Outlier Removal from a Large-scale Dataset using Unsupervised Anomaly Detection

An Ensemble-Based Approach to Anomaly Detection in Marine Engine Sensor Streams for Efficient Condition Monitoring and Analysis

Sensors ◽

10.3390/s20247285 ◽

2020 ◽

Vol 20 (24) ◽

pp. 7285

Author(s):

Donghyun Kim ◽

Sangbong Lee ◽

Jihwan Lee

Keyword(s):

Diesel Engine ◽

Anomaly Detection ◽

Large Scale ◽

Abnormal Behavior ◽

High Dimensional ◽

Statistical Control ◽

Marine Engine ◽

Large Scale Dataset ◽

Unsupervised Anomaly Detection ◽

Sensor Streams

This study proposes an unsupervised anomaly detection method using sensor streams from the marine engine to detect the anomalous system behavior, which may be a possible sign of system failure. Previous works on marine engine anomaly detection proposed a clustering-based or statistical control chart-based approach that is unstable according to the choice of hyperparameters, or cannot fit well to the high-dimensional dataset. As a remedy to this limitation, this study adopts an ensemble-based approach to anomaly detection. The idea is to train several anomaly detectors with varying hyperparameters in parallel and then combine its result in the anomaly detection phase. Because the anomaly is detected by the combination of different detectors, it is robust to the choice of hyperparameters without loss of accuracy. To demonstrate our methodology, an actual dataset obtained from a 200,000-ton cargo vessel from a Korean shipping company that uses two-stroke diesel engine is analyzed. As a result, anomalies were successfully detected from the high-dimensional and large-scale dataset. After detecting the anomaly, clustering analysis was conducted to the anomalous observation to examine anomaly patterns. By investigating each cluster’s feature distribution, several common patterns of abnormal behavior were successfully visualized. Although we analyzed the data from two-stroke diesel engine, our method can be applied to various types of marine engine.

Get full-text (via PubEx)

Generative and Autoencoder Models for Large-Scale Mutivariate Unsupervised Anomaly Detection

10.1007/978-981-16-3637-0_4 ◽

2021 ◽

pp. 45-58

Author(s):

Nabila Ounasser ◽

Maryem Rhanoui ◽

Mounia Mikram ◽

Bouchra El Asri

Keyword(s):

Anomaly Detection ◽

Large Scale ◽

Unsupervised Anomaly Detection

Get full-text (via PubEx)

A Comparative Study of Unsupervised Anomaly Detection Techniques Using Honeypot Data

IEICE Transactions on Information and Systems ◽

10.1587/transinf.e93.d.2544 ◽

2010 ◽

Vol E93-D (9) ◽

pp. 2544-2554 ◽

Cited By ~ 1

Author(s):

Jungsuk SONG ◽

Hiroki TAKAKURA ◽

Yasuo OKABE ◽

Daisuke INOUE ◽

Masashi ETO ◽

...

Keyword(s):

Anomaly Detection ◽

Comparative Study ◽

Detection Techniques ◽

Unsupervised Anomaly Detection

Get full-text (via PubEx)

Unsupervised Anomaly Detection for Network Data Streams in Industrial Control Systems

Information ◽

10.3390/info11020105 ◽

2020 ◽

Vol 11 (2) ◽

pp. 105 ◽

Cited By ~ 1

Author(s):

Limengwei Liu ◽

Modi Hu ◽

Chaoqun Kang ◽

Xiaoyong Li

Keyword(s):

Anomaly Detection ◽

Control Systems ◽

Data Streams ◽

Large Scale ◽

Stable Operation ◽

Time Data ◽

Industrial Control ◽

Industrial Control Systems ◽

Unsupervised Anomaly Detection ◽

Industrial Control Networks

The development and integration of information technology and industrial control networks have expanded the magnitude of new data; detecting anomalies or discovering other valid information from them is of vital importance to the stable operation of industrial control systems. This paper proposes an incremental unsupervised anomaly detection method that can quickly analyze and process large-scale real-time data. Our evaluation on the Secure Water Treatment dataset shows that the method is converging to its offline counterpart for infinitely growing data streams.

Get full-text (via PubEx)

Unsupervised Anomaly Detection for Glaucoma Diagnosis

Wireless Communications and Mobile Computing ◽

10.1155/2021/5978495 ◽

2021 ◽

Vol 2021 ◽

pp. 1-14

Author(s):

Wei Zhou ◽

Yuan Gao ◽

Jianhang Ji ◽

Shicheng Li ◽

Yugen Yi

Keyword(s):

Anomaly Detection ◽

Large Scale ◽

Rapid Development ◽

Class Imbalance ◽

Principal Component ◽

High Tech ◽

Class Imbalance Problem ◽

Image Domain ◽

Glaucoma Diagnosis ◽

Unsupervised Anomaly Detection

With the rapid development of high tech, Internet of Things (IoT) and artificial intelligence (AI) achieve a series of achievements in the healthcare industry. Among them, automatic glaucoma diagnosis is one of them. Glaucoma is second leading cause of blindness in the world. Although many automatic glaucoma diagnosis approaches have been proposed, they still face the following two challenges. First, the data acquisition of diseased images is extremely expensive, especially for disease with low occurrence, leading to the class imbalance. Second, large-scale labeled data are hard to obtain in medical image domain. The aforementioned challenges limit the practical application of these approaches in glaucoma diagnosis. To address these disadvantages, this paper proposes an unsupervised anomaly detection framework based on sparse principal component analysis (SPCA) for glaucoma diagnosis. In the proposed approach, we just employ the one-class normal (nonglaucoma) images for training, so the class imbalance problem can be avoided. Then, to distinguish the glaucoma (abnormal) images from the normal images, a feature set consisting of segmentation-based features and image-based features is extracted, which can capture the shape and textural changes. Next, SPCA is adopted to select the effective features from the feature set. Finally, with the usage of the extracted effective features, glaucoma diagnosis can be automatically accomplished via introducing the T 2 statistic and the control limit, overcoming the issue of insufficient labeled samples. Extensive experiments are carried out on the two public databases, and the experimental results verify the effectiveness of the proposed approach.

Get full-text (via PubEx)

BeatGAN: Anomalous Rhythm Detection using Adversarially Generated Time Series

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2019/616 ◽

2019 ◽

Cited By ~ 3

Author(s):

Bin Zhou ◽

Shenghua Liu ◽

Bryan Hooi ◽

Xueqi Cheng ◽

Jing Ye

Keyword(s):

Time Series ◽

Anomaly Detection ◽

Large Scale ◽

Data Augmentation ◽

Time Series Data ◽

Detection Algorithm ◽

Reconstruction Error ◽

Series Data ◽

Lower Accuracy ◽

Unsupervised Anomaly Detection

Given a large-scale rhythmic time series containing mostly normal data segments (or `beats'), can we learn how to detect anomalous beats in an effective yet efficient way? For example, how can we detect anomalous beats from electrocardiogram (ECG) readings? Existing approaches either require excessively high amounts of labeled and balanced data for classification, or rely on less regularized reconstructions, resulting in lower accuracy in anomaly detection. Therefore, we propose BeatGAN, an unsupervised anomaly detection algorithm for time series data. BeatGAN outputs explainable results to pinpoint the anomalous time ticks of an input beat, by comparing them to adversarially generated beats. Its robustness is guaranteed by its regularization of reconstruction error using an adversarial generation approach, as well as data augmentation using time series warping. Experiments show that BeatGAN accurately and efficiently detects anomalous beats in ECG time series, and routes doctors' attention to anomalous time ticks, achieving accuracy of nearly 0.95 AUC, and very fast inference (2.6 ms per beat). In addition, we show that BeatGAN accurately detects unusual motions from multivariate motion-capture time series data, illustrating its generality.

Get full-text (via PubEx)