scholarly journals Outlier Detection Method for Flash Flood Disaster Monitoring Data based on Information Entropy

2021 ◽  
Vol 2138 (1) ◽  
pp. 012013
Author(s):  
Yongzhi Chen ◽  
Ziao Xu ◽  
Chaoqun Niu
Keyword(s):  
Outlier Detection ◽  
Information Entropy ◽  
Detection Method ◽  
Flash Flood ◽  
False Positive Rate ◽  
Flood Disaster ◽  
Detection Methods ◽  
Positive Rate ◽  
Disaster Monitoring ◽  
Local Outlier

Abstract In the research of flash flood disaster monitoring and early warning, the Internet of Things is widely used in real-time information collection. There are abnormal situations such as noise, repetition and errors in a large amount of data collected by sensors, which will lead to false alarm, lower prediction accuracy and other problems. Aiming at the characteristic that outliers flow of sensors will cause obvious fluctuation of information entropy, this paper proposes a local outlier detection method based on information entropy and optimized by sliding window and LOF (Local Outlier Factor). This method can be used to improve the data quality, thus improving the accuracy of disaster prediction. The method is applied to data stream processing of water sensor, and the experimental results show that the method can accurately detect outliers. Compared with the existing detection methods that only use data distance to determine, the test positive rate is improved and the false positive rate is reduced.

scholarly journals PRATD: A Phased Remote Access Trojan Detection Method with Double-Sided Features

Electronics ◽  
2020 ◽  
Vol 9 (11) ◽  
pp. 1894
Author(s):  
Chun Guo ◽  
Zihua Song ◽  
Yuan Ping ◽  
Guowei Shen ◽  
Yuhei Cui ◽  
...  
Keyword(s):  
False Positive ◽  
Detection Method ◽  
False Positive Rate ◽  
True Positive Rate ◽  
Remote Access ◽  
Detection Methods ◽  
Security Threats ◽  
True Positive ◽  
Trojan Detection ◽  
Positive Rate

Remote Access Trojan (RAT) is one of the most terrible security threats that organizations face today. At present, two major RAT detection methods are host-based and network-based detection methods. To complement one another’s strengths, this article proposes a phased RATs detection method by combining double-side features (PRATD). In PRATD, both host-side and network-side features are combined to build detection models, which is conducive to distinguishing the RATs from benign programs because that the RATs not only generate traffic on the network but also leave traces on the host at run time. Besides, PRATD trains two different detection models for the two runtime states of RATs for improving the True Positive Rate (TPR). The experiments on the network and host records collected from five kinds of benign programs and 20 famous RATs show that PRATD can effectively detect RATs, it can achieve a TPR as high as 93.609% with a False Positive Rate (FPR) as low as 0.407% for the known RATs, a TPR 81.928% and FPR 0.185% for the unknown RATs, which suggests it is a competitive candidate for RAT detection.

An efficient local outlier detection optimized by rough clustering

2021 ◽  
pp. 1-12
Author(s):  
Chunyan She ◽  
Shaohua Zeng
Keyword(s):  
Outlier Detection ◽  
False Positive Rate ◽  
Time Cost ◽  
Local Distribution ◽  
Running Speed ◽  
Local Outlier Factor ◽  
Real World Applications ◽  
Positive Rate ◽  
Nearest Neighborhood ◽  
Local Outlier

Outlier detection is a hot issue in data mining, which has plenty of real-world applications. LOF (Local Outlier Factor) can capture the abnormal degree of objects in the dataset with different density levels, and many extended algorithms have been proposed in recent years. However, the LOF needs to search the nearest neighborhood of each object on the whole dataset, which greatly increases the time cost. Most of these extended algorithms only consider the distance between an object and its neighborhood, but ignore the local distribution of an object within its neighborhood, resulting in a high false-positive rate. To improve the running speed, a rough clustering based on triple fusion is proposed, which divides a dataset into several subsets and outlier detection is performed only on each subset. Then, considering the local distribution of an object within its neighborhood, a new local outlier factor is constructed to estimate the abnormal degree of each object. Finally, the experimental results indicate that the proposed algorithm has better performance and lower running time than the others.

scholarly journals Identification of Influential Variants in Significant Aggregate Rare Variant Tests

2021 ◽  
pp. 1-13
Author(s):  
Rachel Z. Blumhagen ◽  
David A. Schwartz ◽  
Carl D. Langefeld ◽  
Tasha E. Fingerlin
Keyword(s):  
Outlier Detection ◽  
Rare Variant ◽  
False Positive ◽  
Rare Variants ◽  
False Positive Rate ◽  
Experimental Studies ◽  
Detection Methods ◽  
True Positive ◽  
Adaptive Combination ◽  
Positive Rate

<b><i>Introduction:</i></b> Studies that examine the role of rare variants in both simple and complex disease are increasingly common. Though the usual approach of testing rare variants in aggregate sets is more powerful than testing individual variants, it is of interest to identify the variants that are plausible drivers of the association. We present a novel method for prioritization of rare variants after a significant aggregate test by quantifying the influence of the variant on the aggregate test of association. <b><i>Methods:</i></b> In addition to providing a measure used to rank variants, we use outlier detection methods to present the computationally efficient Rare Variant Influential Filtering Tool (RIFT) to identify a subset of variants that influence the disease association. We evaluated several outlier detection methods that vary based on the underlying variance measure: interquartile range (Tukey fences), median absolute deviation, and SD. We performed 1,000 simulations for 50 regions of size 3 kb and compared the true and false positive rates. We compared RIFT using the Inner Tukey to 2 existing methods: adaptive combination of <i>p</i> values (ADA) and a Bayesian hierarchical model (BeviMed). Finally, we applied this method to data from our targeted resequencing study in idiopathic pulmonary fibrosis (IPF). <b><i>Results:</i></b> All outlier detection methods observed higher sensitivity to detect uncommon variants (0.001 &#x3c; minor allele frequency, MAF &#x3e; 0.03) compared to very rare variants (MAF &#x3c;0.001). For uncommon variants, RIFT had a lower median false positive rate compared to the ADA. ADA and RIFT had significantly higher true positive rates than that observed for BeviMed. When applied to 2 regions found previously associated with IPF including 100 rare variants, we identified 6 polymorphisms with the greatest evidence for influencing the association with IPF. <b><i>Discussion:</i></b> In summary, RIFT has a high true positive rate while maintaining a low false positive rate for identifying polymorphisms influencing rare variant association tests. This work provides an approach to obtain greater resolution of the rare variant signals within significant aggregate sets; this information can provide an objective measure to prioritize variants for follow-up experimental studies and insight into the biological pathways involved.

scholarly journals Structural similarity based common library detection method for Android

2021 ◽  
Vol 39 (2) ◽  
pp. 448-453
Author(s):  
Zhiying Mu ◽  
Zhihu Li ◽  
Xiaoyu Li
Keyword(s):  
Large Scale ◽  
Detection Method ◽  
False Positive Rate ◽  
Structural Similarity ◽  
Detection Methods ◽  
Fine Grained ◽  
Real World Applications ◽  
Positive Rate ◽  
Android Applications ◽  
Detection Speed

The correct classifying and filtering of common libraries in Android applications can effectively improve the accuracy of repackaged application detection. However, the existing common library detection methods barely meet the requirement of large-scale app markets due to the low detection speed caused by their classification rules. Aiming at this problem, a structural similarity based common library detection method for Android is presented. The sub-packages with weak association to main package are extracted as common library candidates from the decompiled APK (Android application package) by using PDG (program dependency graph) method. With package structures and API calls being used as features, the classifying of those candidates is accomplished through coarse and fine-grained filtering. The experimental results by using real-world applications as dataset show that the detection speed of the present method is higher while the accuracy and false positive rate are both ensured. The method is proved to be efficient and precise.

scholarly journals Identification of Influential Variants in Significant Aggregate Rare Variant Tests

2020 ◽  
Author(s):  
Rachel Z. Blumhagen ◽  
David A. Schwartz ◽  
Carl D. Langefeld ◽  
Tasha E. Fingerlin
Keyword(s):  
Outlier Detection ◽  
Rare Variant ◽  
False Positive ◽  
Rare Variants ◽  
False Positive Rate ◽  
Experimental Studies ◽  
Detection Methods ◽  
True Positive ◽  
Adaptive Combination ◽  
Positive Rate

AbstractIntroductionStudies that examine the role of rare variants in both simple and complex disease are increasingly common. Though the usual approach of testing rare variants in aggregate sets is more powerful than testing individual variants, it is of interest to identify the variants that are plausible drivers of the association. We present a novel method for prioritization of rare variants after a significant aggregate test by quantifying the influence of the variant on the aggregate test of association.MethodsIn addition to providing a measure used to rank variants, we use outlier detection methods to present the computationally efficient Rare Variant Influential Filtering Tool (RIFT) to identify a subset of variants that influence the disease association. We evaluated several outlier detection methods that vary based on the underlying variance measure: interquartile range (Tukey fences), median absolute deviation and standard deviation. We performed 1000 simulations for 50 regions of size 3kb and compared the true and false positive rates. We compared RIFT using the Inner Tukey to two existing methods: adaptive combination of p-values (ADA) and a Bayesian hierarchical model (BeviMed). Finally, we applied this method to data from our targeted resequencing study in idiopathic pulmonary fibrosis (IPF).ResultsAll outlier detection methods observed higher sensitivity to detect uncommon variants (0.001 < MAF > 0.03) compared to very rare variants (MAF < 0.001). For uncommon variants, RIFT had a lower median false positive rate compared to the ADA. ADA and RIFT had significantly higher true positive rates than that observed for BeviMed. When applied to two regions found previously associated with IPF including 100 rare variants, we identified six polymorphisms with the greatest evidence for influencing the association with IPF.DiscussionIn summary, RIFT has a high true positive rate while maintaining a low false positive rate for identifying polymorphisms influencing rare variant association tests. This work provides an approach to obtain greater resolution of the rare variant signals within significant aggregate sets; this information can provide an objective measure to prioritize variants for follow-up experimental studies and insight into the biological pathways involved.

Local outlier detection based on information entropy weighting

2019 ◽  
Vol 30 (4) ◽  
pp. 207
Author(s):  
Lina Wang ◽  
Chao Feng ◽  
Yongjun Ren ◽  
Jinyue Xia
Keyword(s):  
Outlier Detection ◽  
Information Entropy ◽  
Local Outlier

scholarly journals Outlier Detection in Growth Data: Beyond Biologically Implausible Values

2020 ◽  
Vol 4 (Supplement_2) ◽  
pp. 1174-1174
Author(s):  
Paraskevi Massara ◽  
Robert Bandsma ◽  
Celine Bourdon ◽  
Jonathon Maguire ◽  
Elena Comelli ◽  
...  
Keyword(s):  
Outlier Detection ◽  
Sensitivity And Specificity ◽  
Detection Method ◽  
Nutritional Assessment ◽  
Empirical Method ◽  
Child Growth ◽  
Detection Methods ◽  
Healthy Children ◽  
Growth Data ◽  
Growth Standards

Abstract Objectives Eliminating anthropometry measurement error and employing outlier and biological implausible values (BIV) detection methods adapted to longitudinal measurements is important for the study of growth. This work aimed to review and assess the accuracy of the available BIV and outlier detection methods and propose a growth trajectory outlier detection method. Methods We included 2354 infants from the Applied Research Group for Kids (TARGet Kids! ) cohort-based in Toronto (ON, Canada) that recruits healthy children from birth to 5 years of age. We considered infants with at least 8 length and weight measurements available between the 1st and the 24th month of age. Weight-for-length z-scores (wflz) were calculated using the WHO growth standards. Outlier measurements were randomly introduced in 5% of the wflz measurements using a normal distribution (μ = 0, σ = 1). We employed 4 outlier detection methods; an empirical detection method for BIV using the cut-offs derived from the WHO Child Growth Standards, a clustering method, a method based on cluster prototypes for individual outlier measurements and a method based on cluster prototypes for entire growth trajectories. Each method was applied individually and evaluated using the sensitivity and specificity indexes based on the manually introduced outliers. We also calculated the Kappa statistic to evaluate the agreement of each method against the manual outliers. Results After excluding premature (&lt;37 weeks), low birth weight (&lt;1500 g) neonates and children with missing length and weight measurements, we analyzed 393 children with a total of 3144 measurements. Sensitivity and specificity for the four methods ranged between 4.4%–55.0% and 83.7% −99.7%, respectively, with kappa being non-significant (P &gt; 0.05) only for the empirical. The clustering detection method reported a higher finding rate, while the empirical method found most of the BIV, but few of the rest of the outliers. Conclusions BIV account for a small portion of the possible outliers in growth datasets. We show that additional statistical or model-based methods are required for a more comprehensive outlier detection process, which has implications for growth analysis and nutritional assessment. Funding Sources Joannah and Brian Lawson Center for Child Nutrition, Connaught Fund, Onassis Foundation.

scholarly journals A Review on Outliers-Detection Methods for Multivariate Data

2021 ◽  
Vol 3 (1) ◽  
pp. 1-15
Author(s):  
Sharifah Sakinah Syed Abd Mutalib ◽  
Siti Zanariah Satari ◽  
Wan Nur Syahidah Wan Yusoff
Keyword(s):  
Cluster Analysis ◽  
Multivariate Analysis ◽  
Outlier Detection ◽  
High Dimension ◽  
Detection Method ◽  
Multivariate Data ◽  
Projection Pursuit ◽  
Detection Methods ◽  
And Cluster Analysis ◽  
Detection Of Outliers

Data in practice are often of high dimension and multivariate in nature. Detection of outliers has been one of the problems in multivariate analysis. Detecting outliers in multivariate data is difficult and it is not sufficient by using only graphical inspection. In this paper, a nontechnical and brief outlier detection method for multivariate data which are projection pursuit method, methods based on robust distance and cluster analysis are reviewed. The strengths and weaknesses of each method are briefly discussed.

Sign in / Sign up

Export Citation Format

Share Document