Local outlier detection method towards data stream

2011 ◽  
Author(s):  
Xiao Jian-Qiong
Keyword(s):  
Outlier Detection ◽  
Data Stream ◽  
Detection Method ◽  
Local Outlier

scholarly journals Outlier Detection Method for Flash Flood Disaster Monitoring Data based on Information Entropy

2021 ◽  
Vol 2138 (1) ◽  
pp. 012013
Author(s):  
Yongzhi Chen ◽  
Ziao Xu ◽  
Chaoqun Niu
Keyword(s):  
Outlier Detection ◽  
Information Entropy ◽  
Detection Method ◽  
Flash Flood ◽  
False Positive Rate ◽  
Flood Disaster ◽  
Detection Methods ◽  
Positive Rate ◽  
Disaster Monitoring ◽  
Local Outlier

Abstract In the research of flash flood disaster monitoring and early warning, the Internet of Things is widely used in real-time information collection. There are abnormal situations such as noise, repetition and errors in a large amount of data collected by sensors, which will lead to false alarm, lower prediction accuracy and other problems. Aiming at the characteristic that outliers flow of sensors will cause obvious fluctuation of information entropy, this paper proposes a local outlier detection method based on information entropy and optimized by sliding window and LOF (Local Outlier Factor). This method can be used to improve the data quality, thus improving the accuracy of disaster prediction. The method is applied to data stream processing of water sensor, and the experimental results show that the method can accurately detect outliers. Compared with the existing detection methods that only use data distance to determine, the test positive rate is improved and the false positive rate is reduced.

scholarly journals TADILOF: Time Aware Density-Based Incremental Local Outlier Detection in Data Streams

Sensors ◽  
2020 ◽  
Vol 20 (20) ◽  
pp. 5829 ◽  
Author(s):  
Jen-Wei Huang ◽  
Meng-Xun Zhong ◽  
Bijay Prasad Jaysawal
Keyword(s):  
Outlier Detection ◽  
Data Streams ◽  
Data Stream ◽  
State Of The Art ◽  
Streaming Data ◽  
Current State ◽  
Data Points ◽  
Local Outlier ◽  
Time Aware ◽  
Over Time

Outlier detection in data streams is crucial to successful data mining. However, this task is made increasingly difficult by the enormous growth in the quantity of data generated by the expansion of Internet of Things (IoT). Recent advances in outlier detection based on the density-based local outlier factor (LOF) algorithms do not consider variations in data that change over time. For example, there may appear a new cluster of data points over time in the data stream. Therefore, we present a novel algorithm for streaming data, referred to as time-aware density-based incremental local outlier detection (TADILOF) to overcome this issue. In addition, we have developed a means for estimating the LOF score, termed "approximate LOF," based on historical information following the removal of outdated data. The results of experiments demonstrate that TADILOF outperforms current state-of-the-art methods in terms of AUC while achieving similar performance in terms of execution time. Moreover, we present an application of the proposed scheme to the development of an air-quality monitoring system.

Towards Identifying Multicriteria Outliers

2018 ◽  
Vol 10 (3) ◽  
pp. 27-38
Author(s):  
Baroudi Rouba ◽  
Safia Nait-Bahloul
Keyword(s):  
Outlier Detection ◽  
Decision Aid ◽  
Detection Method ◽  
Real Life ◽  
Country Risk ◽  
Multicriteria Decision Aid ◽  
Financial Problem ◽  
Outranking Method ◽  
Outranking Relations ◽  
Local Outlier

This article tackles the problem of outlier detection in the multicriteria decision aid (MCDA) field. The authors propose an outlier detection method based on binary outranking relations and Local Outlier Factor (LOF) algorithm. The outlier is detected by applying LOF algorithm on the distribution of the outranking relations generated by a multicriteria outranking method. The proposed approach is illustrated on an artificial example and evaluated on a real life financial problem, the country risk problem.

scholarly journals Anomaly Pattern Detection in Streaming Data Based on the Transformation to Multiple Binary-Valued Data Streams

2021 ◽  
Vol 12 (1) ◽  
pp. 19-27
Author(s):  
Taegong Kim ◽  
Cheong Hee Park
Keyword(s):  
Outlier Detection ◽  
Data Streams ◽  
Data Stream ◽  
Detection Method ◽  
Binary Classification ◽  
Streaming Data ◽  
Pattern Detection ◽  
Detection Methods ◽  
Anomaly Pattern ◽  
Isolation Forest

Abstract Anomaly pattern detection in a data stream aims to detect a time point where outliers begin to occur abnormally. Recently, a method for anomaly pattern detection has been proposed based on binary classification for outliers and statistical tests in the data stream of binary labels of normal or an outlier. It showed that an anomaly pattern can be detected accurately even when outlier detection performance is relatively low. However, since the anomaly pattern detection method is based on the binary classification for outliers, most well-known outlier detection methods, with the output of real-valued outlier scores, can not be used directly. In this paper, we propose an anomaly pattern detection method in a data stream using the transformation to multiple binary-valued data streams from real-valued outlier scores. By using three outlier detection methods, Isolation Forest(IF), Autoencoder-based outlier detection, and Local outlier factor(LOF), the proposed anomaly pattern detection method is tested using artificial and real data sets. The experimental results show that anomaly pattern detection using Isolation Forest gives the best performance.

scholarly journals An Incremental Local Outlier Detection Method in the Data Stream

2018 ◽  
Vol 8 (8) ◽  
pp. 1248 ◽  
Author(s):  
Haiqing Yao ◽  
Xiuwen Fu ◽  
Yongsheng Yang ◽  
Octavian Postolache
Keyword(s):  
Outlier Detection ◽  
Data Streams ◽  
Data Stream ◽  
Nearest Neighbor ◽  
Nearest Neighbors ◽  
Detection Accuracy ◽  
K Nearest Neighbor ◽  
Major Work ◽  
Wide Range ◽  
Local Outlier

Outlier detection has attracted a wide range of attention for its broad applications, such as fault diagnosis and intrusion detection, among which the outlier analysis in data streams with high uncertainty and infinity is more challenging. Recent major work of outlier detection has focused on principle research of the local outlier factor, and there are few studies on incremental updating strategies, which are vital to outlier detection in data streams. In this paper, a novel incremental local outlier detection approach is introduced to dynamically evaluate the local outlier in the data stream. An extended local neighborhood consisting of k nearest neighbors, reverse nearest neighbors and shared nearest neighbors is estimated for each data. The theoretical evidence of algorithm complexity for the insertion of new data and deletion of old data in the composite neighborhood shows that the amount of affected data in the incremental calculation is finite. Finally, experiments performed on both synthetic and real datasets verify its scalability and outlier detection accuracy. All results show that the proposed approach has comparable performance with state-of-the-art k nearest neighbor-based methods.

Sign in / Sign up

Export Citation Format

Share Document