Fast Memory Efficient Local Outlier Detection in Data Streams

Outlier detection in data streams is crucial to successful data mining. However, this task is made increasingly difficult by the enormous growth in the quantity of data generated by the expansion of Internet of Things (IoT). Recent advances in outlier detection based on the density-based local outlier factor (LOF) algorithms do not consider variations in data that change over time. For example, there may appear a new cluster of data points over time in the data stream. Therefore, we present a novel algorithm for streaming data, referred to as time-aware density-based incremental local outlier detection (TADILOF) to overcome this issue. In addition, we have developed a means for estimating the LOF score, termed "approximate LOF," based on historical information following the removal of outdated data. The results of experiments demonstrate that TADILOF outperforms current state-of-the-art methods in terms of AUC while achieving similar performance in terms of execution time. Moreover, we present an application of the proposed scheme to the development of an air-quality monitoring system.

Download Full-text

An Incremental Local Outlier Detection Method in the Data Stream

Applied Sciences ◽

10.3390/app8081248 ◽

2018 ◽

Vol 8 (8) ◽

pp. 1248 ◽

Cited By ~ 4

Author(s):

Haiqing Yao ◽

Xiuwen Fu ◽

Yongsheng Yang ◽

Octavian Postolache

Keyword(s):

Outlier Detection ◽

Data Streams ◽

Data Stream ◽

Nearest Neighbor ◽

Nearest Neighbors ◽

Detection Accuracy ◽

K Nearest Neighbor ◽

Major Work ◽

Wide Range ◽

Local Outlier

Outlier detection has attracted a wide range of attention for its broad applications, such as fault diagnosis and intrusion detection, among which the outlier analysis in data streams with high uncertainty and infinity is more challenging. Recent major work of outlier detection has focused on principle research of the local outlier factor, and there are few studies on incremental updating strategies, which are vital to outlier detection in data streams. In this paper, a novel incremental local outlier detection approach is introduced to dynamically evaluate the local outlier in the data stream. An extended local neighborhood consisting of k nearest neighbors, reverse nearest neighbors and shared nearest neighbors is estimated for each data. The theoretical evidence of algorithm complexity for the insertion of new data and deletion of old data in the composite neighborhood shows that the amount of affected data in the incremental calculation is finite. Finally, experiments performed on both synthetic and real datasets verify its scalability and outlier detection accuracy. All results show that the proposed approach has comparable performance with state-of-the-art k nearest neighbor-based methods.

Download Full-text

A Review of Local Outlier Factor Algorithms for Outlier Detection in Big Data Streams

Big Data and Cognitive Computing ◽

10.3390/bdcc5010001 ◽

2020 ◽

Vol 5 (1) ◽

pp. 1

Author(s):

Omar Alghushairy ◽

Raed Alsini ◽

Terence Soule ◽

Xiaogang Ma

Keyword(s):

Big Data ◽

Outlier Detection ◽

Data Streams ◽

Detection Methods ◽

Normal Range ◽

Local Outlier Factor ◽

Detection Algorithms ◽

Network Intrusion ◽

Entire Dataset ◽

Local Outlier

Outlier detection is a statistical procedure that aims to find suspicious events or items that are different from the normal form of a dataset. It has drawn considerable interest in the field of data mining and machine learning. Outlier detection is important in many applications, including fraud detection in credit card transactions and network intrusion detection. There are two general types of outlier detection: global and local. Global outliers fall outside the normal range for an entire dataset, whereas local outliers may fall within the normal range for the entire dataset, but outside the normal range for the surrounding data points. This paper addresses local outlier detection. The best-known technique for local outlier detection is the Local Outlier Factor (LOF), a density-based technique. There are many LOF algorithms for a static data environment; however, these algorithms cannot be applied directly to data streams, which are an important type of big data. In general, local outlier detection algorithms for data streams are still deficient and better algorithms need to be developed that can effectively analyze the high velocity of data streams to detect local outliers. This paper presents a literature review of local outlier detection algorithms in static and stream environments, with an emphasis on LOF algorithms. It collects and categorizes existing local outlier detection algorithms and analyzes their characteristics. Furthermore, the paper discusses the advantages and limitations of those algorithms and proposes several promising directions for developing improved local outlier detection methods for data streams.

Download Full-text