Error Distribution-based Anomaly Score for Forecasting-based Anomaly Detection of PV Systems

2021 ◽  
Author(s):  
HyunYong Lee ◽  
Nac-Woo Kim ◽  
Jun-Gi Lee ◽  
Byung-Tak Lee
Keyword(s):  
Anomaly Detection ◽  
Error Distribution ◽  
Pv Systems ◽  
Anomaly Score

scholarly journals Spectral-Spatial Anomaly Detection of Hyperspectral Data Based on Improved Isolation Forest

2021 ◽  
Author(s):  
Xiangyu Song ◽  
Sunil Aryal ◽  
Kai Ming Ting ◽  
zhen Liu ◽  
Bin He
Keyword(s):  
Anomaly Detection ◽  
Spatial Information ◽  
Hyperspectral Image ◽  
Gabor Filter ◽  
Poor Performance ◽  
Hyperspectral Data ◽  
Relative Mass ◽  
Anomaly Score ◽  
Spectral Anomaly ◽  
Isolation Forest

Anomaly detection in hyperspectral image is affected by redundant bands and the limited utilization capacity of spectral-spatial information. In this article, we propose a novel Improved Isolation Forest (IIF) algorithm based on the assumption that anomaly pixels are more susceptible to isolation than the background pixels. The proposed IIF is a modified version of the Isolation Forest (iForest) algorithm, which addresses the poor performance of iForest in detecting local anomalies and anomaly detection in high-dimensional data. Further, we propose a spectral-spatial anomaly detector based on IIF (SSIIFD) to make full use of global and local information, as well as spectral and spatial information. To be specific, first, we apply the Gabor filter to extract spatial features, which are then employed as input to the Relative Mass Isolation Forest (ReMass-iForest) detector to obtain the spatial anomaly score. Next, original images are divided into several homogeneous regions via the Entropy Rate Segmentation (ERS) algorithm, and the preprocessed images are then employed as input to the proposed IIF detector to obtain the spectral anomaly score. Finally, we fuse the spatial and spectral anomaly scores by combining them linearly to predict anomaly pixels. The experimental results on four real hyperspectral data sets demonstrate that the proposed detector outperforms other state-of-the-art methods.

scholarly journals Subjective Logic-Based Anomaly Detection Framework in Wireless Sensor Networks

2012 ◽  
Vol 8 (1) ◽  
pp. 482191 ◽  
Author(s):  
Jinhui Yuan ◽  
Hongwei Zhou ◽  
Hong Chen
Keyword(s):  
Wireless Sensor Networks ◽  
Sensor Networks ◽  
Anomaly Detection ◽  
Sensor Node ◽  
Wireless Sensor ◽  
Data Simulation ◽  
Subjective Logic ◽  
Simulation Results ◽  
Anomaly Score

In existing anomaly detection approaches, sensor node often turns to neighbors to further determine whether the data is normal while the node itself cannot decide. However, previous works consider neighbors' opinions being just normal and anomalous, and do not consider the uncertainty of neighbors to the data of the node. In this paper, we propose SLAD (subjective logic based anomaly detection) framework. It redefines opinion deriving from subjective logic theory which takes the uncertainty into account. Furthermore, it fuses the opinions of neighbors to get the quantitative anomaly score of the data. Simulation results show that SLAD framework improves the performance of anomaly detection compared with previous works.

scholarly journals Anomaly detection in multimodal MRI identifies rare individual phenotypes among 20,000 brains

2021 ◽  
Author(s):  
Zhiwei Ma ◽  
Daniel S. Reich ◽  
Sarah Dembling ◽  
Jeff H. Duyn ◽  
Alan P. Koretsky
Keyword(s):  
White Matter ◽  
Anomaly Detection ◽  
Large Scale ◽  
Mean Diffusivity ◽  
Large Sample Size ◽  
Lesion Volume ◽  
Resting State Functional Connectivity ◽  
Human Connectome Project ◽  
Multimodal Mri ◽  
Anomaly Score

The UK Biobank (UKB) is a large-scale epidemiological study and its imaging component focuses on the pre-symptomatic participants. Given its large sample size, rare imaging phenotypes within this unique cohort are of interest, as they are often clinically relevant and could be informative for discovering new processes and mechanisms. Identifying these rare phenotypes is often referred to as "anomaly detection", or "outlier detection". However, anomaly detection in neuroimaging has usually been applied in a supervised or semi-supervised manner for clinically defined cohorts of relatively small size. There has been much less work using anomaly detection on large unlabeled cohorts like the UKB. Here we developed a two-level anomaly screening methodology to systematically identify anomalies from ~19,000 UKB subjects. The same method was also applied to ~1,000 young healthy subjects from the Human Connectome Project (HCP). In primary screening, using ventricular, white matter, and gray matter-based imaging phenotypes derived from multimodal MRI, every subject was parameterized with an anomaly score per phenotype to quantitate the degree of abnormality. These anomaly scores were highly robust. Anomaly score distributions of the UKB cohort were all more outlier-prone than the HCP cohort of young adults. The approach enabled the assessments of test-retest reliability via the anomaly scores, which ranged from excellent reliability for ventricular volume, white matter lesion volume, and fractional anisotropy, to good reliability for mean diffusivity and cortical thickness. In secondary screening, the anomalies due to data collection/processing errors were eliminated. A subgroup of the remaining anomalies were radiologically reviewed, and a substantial percentage of them (UKB: 90.1%; HCP: 42.9%) had various brain pathologies such as masses, cysts, white matter lesions, infarcts, encephalomalacia, or prominent sulci. The remaining anomalies of the subgroup had unexplained causes and would be interesting for follow-up. Finally, we show that anomaly detection applied to resting-state functional connectivity did not identify any reliable anomalies, which was attributed to the confounding effects of brain-wide signal variation. Together, this study establishes an unsupervised framework for investigating rare individual imaging phenotypes within large heterogeneous cohorts.

scholarly journals Understanding the Effect of Bias in Deep Anomaly Detection

2021 ◽  
Author(s):  
Ziyu Ye ◽  
Yuxin Chen ◽  
Haitao Zheng
Keyword(s):  
Anomaly Detection ◽  
False Positive Rate ◽  
Score Function ◽  
Learning Task ◽  
Future Research ◽  
Finite Sample ◽  
Unique Challenge ◽  
Anomaly Detector ◽  
Real World Datasets ◽  
Anomaly Score

Anomaly detection presents a unique challenge in machine learning, due to the scarcity of labeled anomaly data. Recent work attempts to mitigate such problems by augmenting training of deep anomaly detection models with additional labeled anomaly samples. However, the labeled data often does not align with the target distribution and introduces harmful bias to the trained model. In this paper, we aim to understand the effect of a biased anomaly set on anomaly detection. Concretely, we view anomaly detection as a supervised learning task where the objective is to optimize the recall at a given false positive rate. We formally study the relative scoring bias of an anomaly detector, defined as the difference in performance with respect to a baseline anomaly detector. We establish the first finite sample rates for estimating the relative scoring bias for deep anomaly detection, and empirically validate our theoretical results on both synthetic and real-world datasets. We also provide an extensive empirical study on how a biased training anomaly set affects the anomaly score function and therefore the detection performance on different anomaly classes. Our study demonstrates scenarios in which the biased anomaly set can be useful or problematic, and provides a solid benchmark for future research.

Anomaly Detection Algorithm Based on Subspace Local Density Estimation

2019 ◽  
Vol 16 (3) ◽  
pp. 44-58 ◽  
Author(s):  
Chunkai Zhang ◽  
Ao Yin
Keyword(s):  
Theoretical Analysis ◽  
Anomaly Detection ◽  
Density Estimation ◽  
Local Density ◽  
Tree Height ◽  
Detection Algorithm ◽  
Density Estimator ◽  
Large Tree ◽  
Anomaly Score ◽  
Tree Evaluation

In this article, the authors propose a novel anomaly detection algorithm based on subspace local density estimation. The key insight of the proposed algorithm is to build multiple trident trees, which can implement the process of building subspace and local density estimation. Each trident tree (T-tree) is constructed recursively by splitting the data outside of 3 sigma into the left or right subtree and splitting the remaining data into the middle subtree. Each node in trident tree records the number of instances that falls on this node, so each trident tree can be used as a local density estimator. The density of each instance is the average of all trident tree evaluation instance densities, and it can be used as the anomaly score of instances. Since each trident tree is constructed according to 3 sigma principle, it can obtain good anomaly detection results without a large tree height. Theoretical analysis and experimental results show that the proposed algorithm is effective and efficient.

Anomaly Detection in Wireless Networks

2016 ◽  
pp. 108-118
Author(s):  
Yirui Hu
Keyword(s):  
Wireless Networks ◽  
Anomaly Detection ◽  
Initial Step ◽  
Training Data ◽  
Data Set ◽  
Detection Systems ◽  
Detection Analysis ◽  
Anomaly Score ◽  
Unsupervised Algorithms ◽  
Traffic Behavior

This chapter is an introduction to multi-cluster based anomaly detection analysis. Various anomalies present different behaviors in wireless networks. Not all anomalies are known to networks. Unsupervised algorithms are desirable to automatically characterize the nature of traffic behavior and detect anomalies from normal behaviors. Essentially all anomaly detection systems first learn a model of the normal patterns in training data set, and then determine the anomaly score of a given testing data point based on the deviations from the learned patterns. The initial step of learning a good model is the most crucial part in anomaly detection. Multi-cluster based analysis are valuable because they can obtain the insights of human behaviors and learn similar patterns in temporal traffic data. The anomaly threshold can be determined by quantitative analysis based on the trained model. A novel quantitative “Donut” algorithm of anomaly detection on the basis of model log-likelihood is proposed in this chapter.

scholarly journals Automated Detection of Ischemic Stroke and Subsequent Patient Triage in Routinely Acquired Head CT

2021 ◽  
Author(s):  
Tom Finck ◽  
David Schinz ◽  
Lioba Grundl ◽  
Rami Eisawy ◽  
Mehmet Yiğitsoy ◽  
...  
Keyword(s):  
Ischemic Stroke ◽  
Anomaly Detection ◽  
Tertiary Care ◽  
Ground Truth ◽  
Care Hospital ◽  
Predictive Values ◽  
Head Ct ◽  
Weakly Supervised ◽  
Anomaly Score ◽  
Subsequent Patient

Abstract Purpose Advanced machine-learning (ML) techniques can potentially detect the entire spectrum of pathology through deviations from a learned norm. We investigated the utility of a weakly supervised ML tool to detect characteristic findings related to ischemic stroke in head CT and provide subsequent patient triage. Methods Patients having undergone non-enhanced head CT at a tertiary care hospital in April 2020 with either no anomalies, subacute or chronic ischemia, lacunar infarcts of the deep white matter or hyperdense vessel signs were retrospectively analyzed. Anomaly detection was performed using a weakly supervised ML classifier. Findings were displayed on a voxel-level (heatmap) and pooled to an anomaly score. Thresholds for this score classified patients into i) normal, ii) inconclusive, iii) pathological. Expert-validated radiological reports were considered as ground truth. Test assessment was performed with ROC analysis; inconclusive results were pooled to pathological predictions for accuracy measurements. Results During the investigation period 208 patients were referred for head CT of which 111 could be included. Definite ratings into normal/pathological were feasible in 77 (69.4%) patients. Based on anomaly scores, the AUC to differentiate normal from pathological scans was 0.98 (95% CI 0.97–1.00). The sensitivity, specificity, positive and negative predictive values were 100%, 40.6%, 80.6% and 100%, respectively. Conclusion Our study demonstrates the potential of a weakly supervised anomaly-detection tool to detect stroke findings in head CT. Definite classification into normal/pathological was made with high accuracy in > 2/3 of patients. Anomaly heatmaps further provide guidance towards pathologies, also in cases with inconclusive ratings.

scholarly journals Spectral-Spatial Anomaly Detection of Hyperspectral Data Based on Improved Isolation Forest

2021 ◽  
Author(s):  
Xiangyu Song
Keyword(s):  
Anomaly Detection ◽  
Spatial Information ◽  
Hyperspectral Image ◽  
Gabor Filter ◽  
Poor Performance ◽  
Hyperspectral Data ◽  
Relative Mass ◽  
Anomaly Score ◽  
Spectral Anomaly ◽  
Isolation Forest

Anomaly detection in hyperspectral image is affected by redundant bands and the limited utilization capacity of spectral-spatial information. In this article, we propose a novel Improved Isolation Forest (IIF) algorithm based on the assumption that anomaly pixels are more susceptible to isolation than the background pixels. The proposed IIF is a modified version of the Isolation Forest (iForest) algorithm, which addresses the poor performance of iForest in detecting local anomalies and anomaly detection in high-dimensional data. Further, we propose a spectral-spatial anomaly detector based on IIF (SSIIFD) to make full use of global and local information, as well as spectral and spatial information. To be specific, first, we apply the Gabor filter to extract spatial features, which are then employed as input to the Relative Mass Isolation Forest (ReMass-iForest) detector to obtain the spatial anomaly score. Next, original images are divided into several homogeneous regions via the Entropy Rate Segmentation (ERS) algorithm, and the preprocessed images are then employed as input to the proposed IIF detector to obtain the spectral anomaly score. Finally, we fuse the spatial and spectral anomaly scores by combining them linearly to predict anomaly pixels. The experimental results on four real hyperspectral data sets demonstrate that the proposed detector outperforms other state-of-the-art methods.

scholarly journals Anomaly Detection in NFV Using Tree-Based Unsupervised Learning Method

2019 ◽  
Author(s):  
Girish L
Keyword(s):  
Anomaly Detection ◽  
Data Center ◽  
Learning Algorithm ◽  
Traffic Monitoring ◽  
Anomalous Behavior ◽  
Normal Operation ◽  
Machine Learning Algorithm ◽  
Memory Network ◽  
Anomaly Score ◽  
Isolation Forest

With the increased adoption of virtualized NFs in data center, it is crucial toaddress some of the challenges such as performance and availability of the applications invirtualized network environment. The normal operation of the network can be analyzed withrespect to the usage of various resources like, CPU, memory, network and disk. Inefficientusage or over usage of these resources leads to anomalous behavior. Anomalies are oftenpreceded by faults. It is important to detect anomalies before they occur. The detectedanomalies can be used for corrective and optimization actions. This paper presents that;unsupervised machine learning algorithm performs better compared to supervised machinelearning algorithms in detecting anomalies. Here we use isolation forest algorithm on timeseries dataset, which is collected using monitoring agent collectd. Stress is induced to thecomputer using traffic monitoring generator and stress-ng. The results show that isolationforest algorithm gives better performance in anomaly detection with good anomaly score.

Sign in / Sign up

Export Citation Format

Share Document