Comparison on outlier detection methods using measured data from a long span tied-arch bridge

2021 ◽  
Author(s):  
Peng Ni ◽  
Haili Jiang ◽  
Wurong Fu ◽  
Ye Xia ◽  
Limin Sun
Keyword(s):  
Outlier Detection ◽  
Measured Data ◽  
Monitoring Data ◽  
Detection Methods ◽  
Data Set ◽  
Rule Based ◽  
Long Span ◽  
The Mean ◽  
Popular Rule ◽  
Tied Arch Bridge

<p>As the demand for the detections of outliers in the structural health monitoring data-set increases, numerous approaches are presented for it. However, the characteristics of the existing methods dealing with different kinds of measured data are not yet clear enough for practical use. Therefore, this paper conducts a comparative study of several popular rule-based methods based on monitoring data of an arch-tied bridge in China. For measured data, outliers are not known in advance. In this way, this study evaluates and compares the detection performances rely on two indicators: the quantity of the detected outliers and the extreme value of the outliers deviating from the mean of the data. Conclusions on the features and applicable situations of involved methods are given. Additionally, combining the results of different methods proves to be beneficial. Finally, a software incorporating the research results is developed for outlier detection.</p>

Outlier Detection Techniques for Data Mining

2011 ◽  
pp. 1483-1488
Author(s):  
Fabrizio Angiulli
Keyword(s):  
Data Mining ◽  
Outlier Detection ◽  
Credit Card ◽  
Detection Methods ◽  
Distribution Model ◽  
Main Task ◽  
Data Set ◽  
Homogeneous Groups ◽  
Definition Of ◽  
Dependency Detection

Data mining techniques can be grouped in four main categories: clustering, classification, dependency detection, and outlier detection. Clustering is the process of partitioning a set of objects into homogeneous groups, or clusters. Classification is the task of assigning objects to one of several predefined categories. Dependency detection searches for pairs of attribute sets which exhibit some degree of correlation in the data set at hand. The outlier detection task can be defined as follows: “Given a set of data points or objects, find the objects that are considerably dissimilar, exceptional or inconsistent with respect to the remaining data”. These exceptional objects as also referred to as outliers. Most of the early methods for outlier identification have been developed in the field of statistics (Hawkins, 1980; Barnett & Lewis, 1994). Hawkins’ definition of outlier clarifies the approach: “An outlier is an observation that deviates so much from other observations as to arouse suspicions that it was generated by a different mechanism”. Indeed, statistical techniques assume that the given data set has a distribution model. Outliers are those points that satisfy a discordancy test, that is, that are significantly far from what would be their expected position given the hypothesized distribution. Many clustering, classification and dependency detection methods produce outliers as a by-product of their main task. For example, in classification, mislabeled objects are considered outliers and thus they are removed from the training set to improve the accuracy of the resulting classifier, while in clustering, objects that do not strongly belong to any cluster are considered outliers. Nevertheless, it must be said that searching for outliers through techniques specifically designed for tasks different from outlier detection could not be advantageous. As an example, clusters can be distorted by outliers and, thus, the quality of the outliers returned is affected by their presence. Moreover, other than returning a solution of higher quality, outlier detection algorithms can be vastly more efficient than non ad-hoc algorithms. While in many contexts outliers are considered as noise that must be eliminated, as pointed out elsewhere, “one person’s noise could be another person’s signal”, and thus outliers themselves can be of great interest. Outlier mining is used in telecom or credit card frauds to detect the atypical usage of telecom services or credit cards, in intrusion detection for detecting unauthorized accesses, in medical analysis to test abnormal reactions to new medical therapies, in marketing and customer segmentations to identify customers spending much more or much less than average customer, in surveillance systems, in data cleaning, and in many other fields.

scholarly journals AN OUTLIER DETECTION METHOD IN GEODETIC NETWORKS BASED ON THE ORIGINAL OBSERVATIONS

2014 ◽  
Vol 20 (3) ◽  
pp. 578-589 ◽  
Author(s):  
BAHATTIN ERDOGAN
Keyword(s):  
Outlier Detection ◽  
Mean Value ◽  
Least Square ◽  
Detection Methods ◽  
Random Errors ◽  
Network Adjustment ◽  
Mean Values ◽  
Geodetic Networks ◽  
First Case ◽  
The Mean

The observations in geodetic networks are measured repetitively and in the network adjustment step, the mean values of these original observations are used. The mean operator is a kind of Least Square Estimation (LSE). LSE provides optimal results when random errors are normally distributed. If one of the original repetitive observations has outlier, the magnitude of this outlier will decrease because the mean value of these original observations is used in the network adjustment and outlier detection. In this case, the reliability of the outlier detection methods decreases, too. Since the original repetitive observations are independent, they can be used in the adjustment model instead of the estimating mean value of them. In this study, to show the effects of the estimating mean value of the original repetitive observations, a leveling network that contains both outward run and backward run observations were simulated. Tests for outlier, Huber and Danish methods were applied to two different cases. First, the mean values of the original observations (outward run and return run) were used; and then all original observations were considered in the outlier detection. The reliabilities of the methods were measured by Mean Succes Rate. According to the obtained results, the second case has more reliable results than first case.

scholarly journals Comparative Study of Outlier Detection Algorithms via Fundamental Analysis Variables

2015 ◽  
Vol 4 (4) ◽  
pp. 45-60 ◽  
Author(s):  
Senol Emir ◽  
Hasan Dincer ◽  
Umit Hacioglu ◽  
Serhat Yuksel
Keyword(s):  
Data Mining ◽  
Outlier Detection ◽  
Prediction Models ◽  
Novelty Detection ◽  
Detection Methods ◽  
Fundamental Analysis ◽  
Data Set ◽  
Detection Algorithms ◽  
Long Time ◽  
Application Specific

In a data set, an outlier refers to a data point that is considerably different from the others. Detecting outliers provides useful application-specific insights and leads to choosing right prediction models. Outlier detection (also known as anomaly detection or novelty detection) has been studied in statistics and machine learning for a long time. It is an essential preprocessing step of data mining process. In this study, outlier detection step in the data mining process is applied for identifying the top 20 outlier firms. Three outlier detection algorithms are utilized using fundamental analysis variables of firms listed in Borsa Istanbul for the 2011-2014 period. The results of each algorithm are presented and compared. Findings show that 15 different firms are identified by three different outlier detection methods. KCHOL and SAHOL have the greatest number of appearances with 12 observations among these firms. By investigating the results, it is concluded that each of three algorithms makes different outlier firm lists due to differences in their approaches for outlier detection.

scholarly journals Association Measures in Network Outlier Detection Methods

2019 ◽  
Vol 8 (4) ◽  
pp. 12218-12223
Keyword(s):  
Outlier Detection ◽  
Design Method ◽  
Robustness Analysis ◽  
Detection Methods ◽  
Multidimensional Data ◽  
Traditional Methods ◽  
Data Set ◽  
Network Intrusion ◽  
Entire Network ◽  
Better Than

Detecting outliers before they cause any damage to the data in the network is a important constraint. Outlier detection methods need to be applied on various applications like fraud detection, network robustness analysis. This paper mainly focuses on detailed measures of both proposed intrusion and outlier detection methods with traditional methods. In the proposed work, KDD CUP data set is used. In this work, we initially divide the entire network into individual nodes for efficient monitoring. Later, the proposed methodology is applied on networks which can easily handle high / multidimensional data. While detection of outliers, the proposed method divides the entire network into sub-networks and each network is formed with density based strategy and then outlier detection is applied on them using a Efficient Crossover Design method which identifies the outliers more accurately. Finally ,the proposed method is evaluated and compared with traditional method will all possible parameters in network intrusion detection and the results prove that the performance levels of the proposed method is far better than the traditional methods

scholarly journals Exploration of Outliers in If-Then Rule-Based Knowledge Bases

Entropy ◽  
2020 ◽  
Vol 22 (10) ◽  
pp. 1096
Author(s):  
Agnieszka Nowak-Brzezińska ◽  
Czesław Horyń
Keyword(s):  
Outlier Detection ◽  
Domain Knowledge ◽  
Clustering Algorithms ◽  
Knowledge Bases ◽  
Detection Methods ◽  
Complex Data ◽  
Rule Based ◽  
Detection Algorithms ◽  
The Subject

The article presents both methods of clustering and outlier detection in complex data, such as rule-based knowledge bases. What distinguishes this work from others is, first, the application of clustering algorithms to rules in domain knowledge bases, and secondly, the use of outlier detection algorithms to detect unusual rules in knowledge bases. The aim of the paper is the analysis of using four algorithms for outlier detection in rule-based knowledge bases: Local Outlier Factor (LOF), Connectivity-based Outlier Factor (COF), K-MEANS, and SMALLCLUSTERS. The subject of outlier mining is very important nowadays. Outliers in rules If-Then mean unusual rules, which are rare in comparing to others and should be explored by the domain expert as soon as possible. In the research, the authors use the outlier detection methods to find a given number of outliers in rules (1%, 5%, 10%), while in small groups, the number of outliers covers no more than 5% of the rule cluster. Subsequently, the authors analyze which of seven various quality indices, which they use for all rules and after removing selected outliers, improve the quality of rule clusters. In the experimental stage, the authors use six different knowledge bases. The best results (the most often the clusters quality was improved) are achieved for two outlier detection algorithms LOF and COF.

scholarly journals Building Damage Detection from Post-Event Aerial Imagery Using Single Shot Multibox Detector

2019 ◽  
Vol 9 (6) ◽  
pp. 1128 ◽  
Author(s):  
Yundong Li ◽  
Wei Hu ◽  
Han Dong ◽  
Xueyan Zhang
Keyword(s):  
Machine Learning ◽  
Data Augmentation ◽  
Hurricane Sandy ◽  
Training Data ◽  
Aerial Images ◽  
Detection Methods ◽  
Single Shot ◽  
Data Set ◽  
Augmentation Strategies ◽  
Post Disaster

Using aerial cameras, satellite remote sensing or unmanned aerial vehicles (UAV) equipped with cameras can facilitate search and rescue tasks after disasters. The traditional manual interpretation of huge aerial images is inefficient and could be replaced by machine learning-based methods combined with image processing techniques. Given the development of machine learning, researchers find that convolutional neural networks can effectively extract features from images. Some target detection methods based on deep learning, such as the single-shot multibox detector (SSD) algorithm, can achieve better results than traditional methods. However, the impressive performance of machine learning-based methods results from the numerous labeled samples. Given the complexity of post-disaster scenarios, obtaining many samples in the aftermath of disasters is difficult. To address this issue, a damaged building assessment method using SSD with pretraining and data augmentation is proposed in the current study and highlights the following aspects. (1) Objects can be detected and classified into undamaged buildings, damaged buildings, and ruins. (2) A convolution auto-encoder (CAE) that consists of VGG16 is constructed and trained using unlabeled post-disaster images. As a transfer learning strategy, the weights of the SSD model are initialized using the weights of the CAE counterpart. (3) Data augmentation strategies, such as image mirroring, rotation, Gaussian blur, and Gaussian noise processing, are utilized to augment the training data set. As a case study, aerial images of Hurricane Sandy in 2012 were maximized to validate the proposed method’s effectiveness. Experiments show that the pretraining strategy can improve of 10% in terms of overall accuracy compared with the SSD trained from scratch. These experiments also demonstrate that using data augmentation strategies can improve mAP and mF1 by 72% and 20%, respectively. Finally, the experiment is further verified by another dataset of Hurricane Irma, and it is concluded that the paper method is feasible.

scholarly journals Relationship between large-scale ionospheric field-aligned currents and electron/ion precipitations: DMSP observations

2020 ◽  
Vol 72 (1) ◽  
Author(s):  
Chao Xiong ◽  
Claudia Stolle ◽  
Patrick Alken ◽  
Jan Rauberg
Keyword(s):  
Time Distribution ◽  
Large Scale ◽  
Particle Flux ◽  
Particle Energy ◽  
Lower Latitude ◽  
Particle Precipitation ◽  
Data Set ◽  
The Mean ◽  
Two Parameters ◽  
Meteorological Satellite

Abstract In this study, we have derived field-aligned currents (FACs) from magnetometers onboard the Defense Meteorological Satellite Project (DMSP) satellites. The magnetic latitude versus local time distribution of FACs from DMSP shows comparable dependences with previous findings on the intensity and orientation of interplanetary magnetic field (IMF) By and Bz components, which confirms the reliability of DMSP FAC data set. With simultaneous measurements of precipitating particles from DMSP, we further investigate the relation between large-scale FACs and precipitating particles. Our result shows that precipitation electron and ion fluxes both increase in magnitude and extend to lower latitude for enhanced southward IMF Bz, which is similar to the behavior of FACs. Under weak northward and southward Bz conditions, the locations of the R2 current maxima, at both dusk and dawn sides and in both hemispheres, are found to be close to the maxima of the particle energy fluxes; while for the same IMF conditions, R1 currents are displaced further to the respective particle flux peaks. Largest displacement (about 3.5°) is found between the downward R1 current and ion flux peak at the dawn side. Our results suggest that there exists systematic differences in locations of electron/ion precipitation and large-scale upward/downward FACs. As outlined by the statistical mean of these two parameters, the FAC peaks enclose the particle energy flux peaks in an auroral band at both dusk and dawn sides. Our comparisons also found that particle precipitation at dawn and dusk and in both hemispheres maximizes near the mean R2 current peaks. The particle precipitation flux maxima closer to the R1 current peaks are lower in magnitude. This is opposite to the known feature that R1 currents are on average stronger than R2 currents.

scholarly journals Outlier Detection Transilience-Probabilistic Model for Wind Tunnels Based on Sensor Data

Sensors ◽  
2021 ◽  
Vol 21 (7) ◽  
pp. 2532
Author(s):  
Encarna Quesada ◽  
Juan J. Cuadrado-Gallego ◽  
Miguel Ángel Patricio ◽  
Luis Usero
Keyword(s):  
Anomaly Detection ◽  
Outlier Detection ◽  
Research Work ◽  
Industrial Sector ◽  
Sensor Data ◽  
Correct Selection ◽  
Wind Tunnels ◽  
Data Set ◽  
First Case ◽  
Anomaly Analysis

Anomaly Detection research is focused on the development and application of methods that allow for the identification of data that are different enough—compared with the rest of the data set that is being analyzed—and considered anomalies (or, as they are more commonly called, outliers). These values mainly originate from two sources: they may be errors introduced during the collection or handling of the data, or they can be correct, but very different from the rest of the values. It is essential to correctly identify each type as, in the first case, they must be removed from the data set but, in the second case, they must be carefully analyzed and taken into account. The correct selection and use of the model to be applied to a specific problem is fundamental for the success of the anomaly detection study and, in many cases, the use of only one model cannot provide sufficient results, which can be only reached by using a mixture model resulting from the integration of existing and/or ad hoc-developed models. This is the kind of model that is developed and applied to solve the problem presented in this paper. This study deals with the definition and application of an anomaly detection model that combines statistical models and a new method defined by the authors, the Local Transilience Outlier Identification Method, in order to improve the identification of outliers in the sensor-obtained values of variables that affect the operations of wind tunnels. The correct detection of outliers for the variables involved in wind tunnel operations is very important for the industrial ventilation systems industry, especially for vertical wind tunnels, which are used as training facilities for indoor skydiving, as the incorrect performance of such devices may put human lives at risk. In consequence, the use of the presented model for outlier detection may have a high impact in this industrial sector. In this research work, a proof-of-concept is carried out using data from a real installation, in order to test the proposed anomaly analysis method and its application to control the correct performance of wind tunnels.

Sign in / Sign up

Export Citation Format

Share Document