scholarly journals A flexible framework for anomaly Detection via dimensionality reduction

2021 ◽  
Author(s):  
Alireza Vafaei Sadr ◽  
Bruce A. Bassett ◽  
M. Kunz
Keyword(s):  
Anomaly Detection ◽  
Dimensionality Reduction ◽  
Dimensional Space ◽  
High Dimensions ◽  
Detection Algorithms ◽  
Latent Space ◽  
Wide Range ◽  
Flexible Framework ◽  
Online Anomaly Detection ◽  
Python Package

AbstractAnomaly detection is challenging, especially for large datasets in high dimensions. Here, we explore a general anomaly detection framework based on dimensionality reduction and unsupervised clustering. DRAMA is released as a general python package that implements the general framework with a wide range of built-in options. This approach identifies the primary prototypes in the data with anomalies detected by their large distances from the prototypes, either in the latent space or in the original, high-dimensional space. DRAMA is tested on a wide variety of simulated and real datasets, in up to 3000 dimensions, and is found to be robust and highly competitive with commonly used anomaly detection algorithms, especially in high dimensions. The flexibility of the DRAMA framework allows for significant optimization once some examples of anomalies are available, making it ideal for online anomaly detection, active learning, and highly unbalanced datasets. Besides, DRAMA naturally provides clustering of outliers for subsequent analysis.

scholarly journals Investigating the Effects of a Combined Spatial and Spectral Dimensionality Reduction Approach for Aerial Hyperspectral Target Detection Applications

2021 ◽  
Vol 13 (9) ◽  
pp. 1647
Author(s):  
Fraser Macfarlane ◽  
Paul Murray ◽  
Stephen Marshall ◽  
Henry White
Keyword(s):  
Dimensionality Reduction ◽  
Target Detection ◽  
Effective Means ◽  
Reduction Process ◽  
Hyperspectral Images ◽  
Detection Algorithms ◽  
Wide Range ◽  
Reduction Methods ◽  
Spectral Dimensionality ◽  
Spatial Redundancy

Target detection and classification is an important application of hyperspectral imaging in remote sensing. A wide range of algorithms for target detection in hyperspectral images have been developed in the last few decades. Given the nature of hyperspectral images, they exhibit large quantities of redundant information and are therefore compressible. Dimensionality reduction is an effective means of both compressing and denoising data. Although spectral dimensionality reduction is prevalent in hyperspectral target detection applications, the spatial redundancy of a scene is rarely exploited. By applying simple spatial masking techniques as a preprocessing step to disregard pixels of definite disinterest, the subsequent spectral dimensionality reduction process is simpler, less costly and more informative. This paper proposes a processing pipeline to compress hyperspectral images both spatially and spectrally before applying target detection algorithms to the resultant scene. The combination of several different spectral dimensionality reduction methods and target detection algorithms, within the proposed pipeline, are evaluated. We find that the Adaptive Cosine Estimator produces an improved F1 score and Matthews Correlation Coefficient when compared to unprocessed data. We also show that by using the proposed pipeline the data can be compressed by over 90% and target detection performance is maintained.

scholarly journals LC-IDS: Loci-Constellation-Based Intrusion Detection for Reconfigurable Wireless Networks

Electronics ◽  
2021 ◽  
Vol 10 (24) ◽  
pp. 3053
Author(s):  
Jaime Zuniga-Mejia ◽  
Rafaela Villalpando-Hernandez ◽  
Cesar Vargas-Rosales ◽  
Mahdi Zareei
Keyword(s):  
Wireless Networks ◽  
Intrusion Detection ◽  
Anomaly Detection ◽  
Dimensional Space ◽  
Temporal Structure ◽  
Detection Performance ◽  
Detection Accuracy ◽  
Routing Attacks ◽  
Network Intrusion ◽  
Wide Range

Detection accuracy of current machine-learning approaches to intrusion detection depends heavily on feature engineering and dimensionality-reduction techniques (e.g., variational autoencoder) applied to large datasets. For many use cases, a tradeoff between detection performance and resource requirements must be considered. In this paper, we propose Loci-Constellation-based Intrusion Detection System (LC-IDS), a general framework for network intrusion detection (detection of already known and previously unknown routing attacks) for reconfigurable wireless networks (e.g., vehicular ad hoc networks, unmanned aerial vehicle networks). We introduce the concept of ‘attack-constellation’, which allows us to represent all the relevant information for intrusion detection (misuse detection and anomaly detection) on a latent 2-dimensional space that arises naturally by considering the temporal structure of the input data. The attack/anomaly-detection performance of LC-IDS is analyzed through simulations in a wide range of network conditions. We show that for all the analyzed network scenarios, we can detect known attacks, with a good detection accuracy, and anomalies with low false positive rates. We show the flexibility and scalability of LC-IDS that allow us to consider a dynamic number of neighboring nodes and routing attacks in the ‘attack-constellation’ in a distributed fashion and with low computational requirements.

scholarly journals GUV-Net for high fidelity shoeprint generation

2021 ◽  
Author(s):  
Muhammad Hassan ◽  
Yan Wang ◽  
Wei Pang ◽  
Di Wang ◽  
Daixi Li ◽  
...  
Keyword(s):  
Deep Learning ◽  
Dimensional Space ◽  
Super Resolution ◽  
Network Models ◽  
Ground Truth ◽  
Special Treatment ◽  
High Fidelity ◽  
Proposed Model ◽  
Latent Space ◽  
Wide Range

AbstractShoeprints contain valuable information for tracing evidence in forensic scenes, and they need to be generated into cleaned, sharp, and high-fidelity images. Most of the acquired shoeprints are found with low quality and/or in distorted forms. The high-fidelity shoeprint generation is of great significance in forensic science. A wide range of deep learning models has been suggested for super-resolution, being either generalized approaches or application specific. Considering the crucial challenges in shoeprint based processing and lacking specific algorithms, we proposed a deep learning based GUV-Net model for high-fidelity shoeprint generation. GUV-Net imitates learning features from VAE, U-Net, and GAN network models with special treatment of absent ground truth shoeprints. GUV-Net encodes efficient probabilistic distributions in the latent space and decodes variants of samples together with passed key features. GUV-Net forwards the learned samples to a refinement-unit proceeded to the generation of high-fidelity output. The refinement-unit receives low-level features from the decoding module at distinct levels. Furthermore, the refinement process is made more efficient by inverse-encoded in high dimensional space through a parallel inverse encoding network. The objective functions at different levels enable the model to efficiently optimize the parameters by mapping a low quality image to a high-fidelity one by maintaining salient features which are important to forensics. Finally, the performance of the proposed model is evaluated against state-of-the-art super-resolution network models.

Anomaly Detection via Mining Numerical Workflow Relations from Logs

2020 ◽  
Author(s):  
Bo Zhang ◽  
Hongyu Zhang ◽  
Pablo Moscato
Keyword(s):  
Distributed Systems ◽  
Anomaly Detection ◽  
Text Messages ◽  
Distributed File System ◽  
Log Data ◽  
Sliding Windows ◽  
Novel Approach ◽  
Hadoop Distributed File System ◽  
Blue Gene ◽  
Online Anomaly Detection

<div>Complex software intensive systems, especially distributed systems, generate logs for troubleshooting. The logs are text messages recording system events, which can help engineers determine the system's runtime status. This paper proposes a novel approach named ADR (stands for Anomaly Detection by workflow Relations) that employs matrix nullspace to mine numerical relations from log data. The mined relations can be used for both offline and online anomaly detection and facilitate fault diagnosis. We have evaluated ADR on log data collected from two distributed systems, HDFS (Hadoop Distributed File System) and BGL (IBM Blue Gene/L supercomputers system). ADR successfully mined 87 and 669 numerical relations from the logs and used them to detect anomalies with high precision and recall. For online anomaly detection, ADR employs PSO (Particle Swarm Optimization) to find the optimal sliding windows' size and achieves fast anomaly detection.</div><div>The experimental results confirm that ADR is effective for both offline and online anomaly detection. </div>

Unsupervised Online Anomaly Detection to Identify Cyber-Attacks on Internet Connected Photovoltaic System Inverters

2021 ◽  
Author(s):  
C. Birk Jones ◽  
Adrian Chavez ◽  
Shamina Hossain-McKenzie ◽  
Nicholas Jacobs ◽  
Adam Summers ◽  
...  
Keyword(s):  
Anomaly Detection ◽  
Cyber Attacks ◽  
Photovoltaic System ◽  
Online Anomaly Detection

scholarly journals Anomaly detection with convolutional Graph Neural Networks

2021 ◽  
Vol 2021 (8) ◽  
Author(s):  
Oliver Atkinson ◽  
Akanksha Bhardwaj ◽  
Christoph Englert ◽  
Vishal S. Ngairangbam ◽  
Michael Spannowsky
Keyword(s):  
Neural Networks ◽  
Anomaly Detection ◽  
Top Quarks ◽  
New Physics ◽  
Broad Applicability ◽  
Latent Space ◽  
Edge Features ◽  
Graph Neural Networks ◽  
Promising Avenue ◽  
Do So

Abstract We devise an autoencoder based strategy to facilitate anomaly detection for boosted jets, employing Graph Neural Networks (GNNs) to do so. To overcome known limitations of GNN autoencoders, we design a symmetric decoder capable of simultaneously reconstructing edge features and node features. Focusing on latent space based discriminators, we find that such setups provide a promising avenue to isolate new physics and competing SM signatures from sensitivity-limiting QCD jet contributions. We demonstrate the flexibility and broad applicability of this approach using examples of W bosons, top quarks, and exotic hadronically-decaying exotic scalar bosons.

scholarly journals Intrusion Detection based on Sequential Information preserving Log Embedding Methods and Anomaly Detection Algorithms

IEEE Access ◽  
2021 ◽  
pp. 1-1
Author(s):  
Czangyeob Kim ◽  
Myeongjun Jang ◽  
Seungwan Seo ◽  
Kyeongchan Park ◽  
Pilsung Kang
Keyword(s):  
Intrusion Detection ◽  
Anomaly Detection ◽  
Detection Algorithms ◽  
Sequential Information ◽  
Embedding Methods
Sign in / Sign up

Export Citation Format

Share Document