A Mixture Model-Based Combination Approach for Outlier Detection

2014 ◽  
Vol 23 (04) ◽  
pp. 1460021 ◽  
Author(s):  
Mohamed Bouguessa
Keyword(s):  
Outlier Detection ◽  
Mixture Model ◽  
Real Data ◽  
Detection Algorithm ◽  
Automatic Identification ◽  
Data Sets ◽  
Score Vector ◽  
Data Object ◽  
Multivariate Beta ◽  
To Come

In this paper, we propose an approach that combines different outlier detection algorithms in order to gain an improved effectiveness. To this end, we first estimate an outlier score vector for each data object. Each element of the estimated vectors corresponds to an outlier score produced by a specific outlier detection algorithm. We then use the multivariate beta mixture model to cluster the outlier score vectors into several components so that the component that corresponds to the outliers can be identified. A notable feature of the proposed approach is the automatic identification of outliers, while most existing methods return only a ranked list of points, expecting the outliers to come first; or require empirical threshold estimation to identify outliers. Experimental results, on both synthetic and real data sets, show that our approach substantially enhances the accuracy of outlier base detectors considered in the combination and overcome their drawbacks.

A Data Stream Outlier Detection Algorithm Based on Reverse K Nearest Neighbors

2011 ◽  
Vol 225-226 ◽  
pp. 1032-1035 ◽  
Author(s):  
Zhong Ping Zhang ◽  
Yong Xin Liang
Keyword(s):  
Outlier Detection ◽  
Data Stream ◽  
Concept Drift ◽  
Real Data ◽  
Nearest Neighbors ◽  
Detection Algorithm ◽  
Data Sets ◽  
K Nearest Neighbors ◽  
Query Manager ◽  
Current Window

This paper proposes a new data stream outlier detection algorithm SODRNN based on reverse nearest neighbors. We deal with the sliding window model, where outlier queries are performed in order to detect anomalies in the current window. The update of insertion or deletion only needs one scan of the current window, which improves efficiency. The capability of queries at arbitrary time on the whole current window is achieved by Query Manager Procedure, which can capture the phenomenon of concept drift of data stream in time. Results of experiments conducted on both synthetic and real data sets show that SODRNN algorithm is both effective and efficient.

scholarly journals A robust method for inverse transport modeling of atmospheric emissions using blind outlier detection

2014 ◽  
Vol 7 (5) ◽  
pp. 2303-2311 ◽  
Author(s):  
M. Martinez-Camara ◽  
B. Béjar Haro ◽  
A. Stohl ◽  
M. Vetterli
Keyword(s):  
Outlier Detection ◽  
Environmental Concern ◽  
Measurement Data ◽  
Real Data ◽  
Detection Algorithm ◽  
Data Sets ◽  
Data Set ◽  
Heavy Tailed ◽  
Improved Performance ◽  
Inverse Transport

Abstract. Emissions of harmful substances into the atmosphere are a serious environmental concern. In order to understand and predict their effects, it is necessary to estimate the exact quantity and timing of the emissions from sensor measurements taken at different locations. There are a number of methods for solving this problem. However, these existing methods assume Gaussian additive errors, making them extremely sensitive to outlier measurements. We first show that the errors in real-world measurement data sets come from a heavy-tailed distribution, i.e., include outliers. Hence, we propose robustifying the existing inverse methods by adding a blind outlier-detection algorithm. The improved performance of our method is demonstrated on a real data set and compared to previously proposed methods. For the blind outlier detection, we first use an existing algorithm, RANSAC, and then propose a modification called TRANSAC, which provides a further performance improvement.

A Support Based Initialization Algorithm for Categorical Data Clustering

2018 ◽  
Vol 11 (2) ◽  
pp. 53-67
Author(s):  
Ajay Kumar ◽  
Shishir Kumar
Keyword(s):  
Categorical Data ◽  
Selection Process ◽  
Numerical Data ◽  
Real Data ◽  
Data Sets ◽  
Data Set ◽  
Data Object ◽  
Data Points ◽  
Wu Method ◽  
Selection Algorithms

Several initial center selection algorithms are proposed in the literature for numerical data, but the values of the categorical data are unordered so, these methods are not applicable to a categorical data set. This article investigates the initial center selection process for the categorical data and after that present a new support based initial center selection algorithm. The proposed algorithm measures the weight of unique data points of an attribute with the help of support and then integrates these weights along the rows, to get the support of every row. Further, a data object having the largest support is chosen as an initial center followed by finding other centers that are at the greatest distance from the initially selected center. The quality of the proposed algorithm is compared with the random initial center selection method, Cao's method, Wu method and the method introduced by Khan and Ahmad. Experimental analysis on real data sets shows the effectiveness of the proposed algorithm.

scholarly journals Sensitive, reliable, and robust circRNA detection from RNA-seq with CirComPara2

2021 ◽  
Author(s):  
Enrico Gaffo ◽  
Alessia Buratin ◽  
Anna Dal Molin ◽  
Stefania Bortoluzzi
Keyword(s):  
Real Data ◽  
Detection Algorithm ◽  
Detection Methods ◽  
Circular Rnas ◽  
Data Sets ◽  
Rna Seq ◽  
Bioinformatics Tool ◽  
Diverse Data ◽  
High Throughput Study ◽  
Discovery Rates

AbstractCurrent methods for identifying circular RNAs (circRNAs) suffer from low discovery rates and inconsistent performance in diverse data sets. Therefore, the applied detection algorithm can bias high-throughput study findings by missing relevant circRNAs. Here, we show that our bioinformatics tool CirComPara2 (https://github.com/egaffo/CirComPara2), by combining multiple circRNA detection methods, consistently achieves high recall rates without loss of precision in simulated and different real-data sets.

Parallel Network Analysis and Communities Detection (PANC) Pipeline for the Analysis and Visualization of COVID-19 Data

2021 ◽  
pp. 2142002
Author(s):  
Giuseppe Agapito ◽  
Marianna Milano ◽  
Mario Cannataro
Keyword(s):  
Network Analysis ◽  
Real Data ◽  
Detection Algorithm ◽  
Data Sets ◽  
Data Set ◽  
Italian Regions ◽  
Initial Dataset ◽  
Parallel Network ◽  
Community Detection Algorithm ◽  
Similarity Matrices

A new coronavirus, causing a severe acute respiratory syndrome (COVID-19), was started at Wuhan, China, in December 2019. The epidemic has rapidly spread across the world becoming a pandemic that, as of today, has affected more than 70 million people causing over 2 million deaths. To better understand the evolution of spread of the COVID-19 pandemic, we developed PANC (Parallel Network Analysis and Communities Detection), a new parallel preprocessing methodology for network-based analysis and communities detection on Italian COVID-19 data. The goal of the methodology is to analyze set of homogeneous datasets (i.e. COVID-19 data in several regions) using a statistical test to find similar/dissimilar behaviours, mapping such similarity information on a graph and then using community detection algorithm to visualize and analyze the initial dataset. The methodology includes the following steps: (i) a parallel methodology to build similarity matrices that represent similar or dissimilar regions with respect to data; (ii) an effective workload balancing function to improve performance; (iii) the mapping of similarity matrices into networks where nodes represent Italian regions, and edges represent similarity relationships; (iv) the discovering and visualization of communities of regions that show similar behaviour. The methodology is general and can be applied to world-wide data about COVID-19, as well as to all types of data sets in tabular and matrix format. To estimate the scalability with increasing workloads, we analyzed three synthetic COVID-19 datasets with the size of 90.0[Formula: see text]MB, 180.0[Formula: see text]MB, and 360.0[Formula: see text]MB. Experiments was performed on showing the amount of data that can be analyzed in a given amount of time increases almost linearly with the number of computing resources available. Instead, to perform communities detection, we employed the real data set.

SPECIALIZED MEASURES OF LEAF OUTLINES AND HIERARCHICAL PERCEPTRONS IN AN AUTOMATIC IDENTIFICATION OF PLANTS

2004 ◽  
Vol 15 (08) ◽  
pp. 1171-1186 ◽  
Author(s):  
WOJCIECH BORKOWSKI ◽  
LIDIA KOSTRZYŃSKA
Keyword(s):  
Neural Network ◽  
Nearest Neighbor ◽  
Real Data ◽  
Automatic Identification ◽  
Identification System ◽  
Data Sets ◽  
Identification Algorithm ◽  
Plant Identification ◽  
Neural Network Classifiers ◽  
Computer Identification

The development of an efficient image-based computer identification system for plants or other organisms is an important ambitious goal, which is still far from realization. This paper presents three new methods potentially usable for such a system: fractal-based measures of complexity of leaf outline, a heuristic algorithm for automatic detection of leaf parts — the blade and the petiole, and a hierarchical perceptron — a kind of neural network classifier. The next few sets of automatically extractable features of leaf blades, encompassed those presented and/or traditionally used, are compared in the task of plant identification using the simplest known "nearest neighbor" identification algorithm, and more realistic neural network classifiers, especially the hierarchical. We show on two real data sets that the presented techniques are really usable for automatic identification, and are worthy of further investigation.

scholarly journals Research on Outlier Detection Algorithm for Evaluation of Battery System Safety

2014 ◽  
Vol 6 ◽  
pp. 830402 ◽  
Author(s):  
Changhao Piao ◽  
Zhi Huang ◽  
Ling Su ◽  
Sheng Lu
Keyword(s):  
Outlier Detection ◽  
Data Stream ◽  
Concept Drift ◽  
Detection Algorithm ◽  
High Dimensional ◽  
Small Scale ◽  
Data Sets ◽  
Angle Distribution ◽  
System Safety ◽  
Battery System

Battery system is the key part of the electric vehicle. To realize outlier detection in the running process of battery system effectively, a new high-dimensional data stream outlier detection algorithm (DSOD) based on angle distribution is proposed. First, in order to improve the algorithm stability in high-dimensional space, the method of angle distribution-based outlier detection algorithm is employed. Second, to reduce the computational complexity, a small-scale calculation set of data stream is established, which is composed of normal set and border set. For the purpose of solving the problem of concept drift, an update mechanism for the normal set and border set is developed in this paper. By this way, these hidden abnormal points will be rapidly detected. The experimental results on real data sets and battery system simulation data sets demonstrate that DSOD is more efficient than Simple variance of angles (Simple VOA) and angle-based outlier detection (ABOD) and is very suitable for the evaluation of battery system safety.

Semi-Supervised Outlier Detection with Only Positive and Unlabeled Data Based on Fuzzy Clustering

2015 ◽  
Vol 24 (03) ◽  
pp. 1550003 ◽  
Author(s):  
Armin Daneshpazhouh ◽  
Ashkan Sami
Keyword(s):  
Intrusion Detection ◽  
Outlier Detection ◽  
Fuzzy Clustering ◽  
Real World ◽  
State Of The Art ◽  
Real Data ◽  
Experimental Results ◽  
The Other ◽  
Data Sets ◽  
Real World Applications

The task of semi-supervised outlier detection is to find the instances that are exceptional from other data, using some labeled examples. In many applications such as fraud detection and intrusion detection, this issue becomes more important. Most existing techniques are unsupervised. On the other hand, semi-supervised approaches use both negative and positive instances to detect outliers. However, in many real world applications, very few positive labeled examples are available. This paper proposes an innovative approach to address this problem. The proposed method works as follows. First, some reliable negative instances are extracted by a kNN-based algorithm. Afterwards, fuzzy clustering using both negative and positive examples is utilized to detect outliers. Experimental results on real data sets demonstrate that the proposed approach outperforms the previous unsupervised state-of-the-art methods in detecting outliers.

scholarly journals Gaussian Mixture Models Based on Principal Components and Applications

2020 ◽  
Vol 2020 ◽  
pp. 1-13
Author(s):  
Nada A. Alqahtani ◽  
Zakiah I. Kalantan
Keyword(s):  
Mixture Model ◽  
Principal Components ◽  
Gaussian Mixture Models ◽  
Expectation Maximization Algorithm ◽  
Large Data ◽  
Real Data ◽  
R Package ◽  
Gaussian Mixture ◽  
Machine Learning Algorithms ◽  
Data Sets

Data scientists use various machine learning algorithms to discover patterns in large data that can lead to actionable insights. In general, high-dimensional data are reduced by obtaining a set of principal components so as to highlight similarities and differences. In this work, we deal with the reduced data using a bivariate mixture model and learning with a bivariate Gaussian mixture model. We discuss a heuristic for detecting important components by choosing the initial values of location parameters using two different techniques: cluster means, k-means and hierarchical clustering, and default values in the “mixtools” R package. The parameters of the model are obtained via an expectation maximization algorithm. The criteria from Bayesian point are evaluated for both techniques, demonstrating that both techniques are efficient with respect to computation capacity. The effectiveness of the discussed techniques is demonstrated through a simulation study and using real data sets from different fields.

Sign in / Sign up

Export Citation Format

Share Document