scholarly journals Automated Feature Selection for Anomaly Detection in Network Traffic Data

2021 ◽  
Vol 12 (3) ◽  
pp. 1-28
Author(s):  
Makiya Nakashima ◽  
Alex Sim ◽  
Youngsoo Kim ◽  
Jonghyun Kim ◽  
Jinoh Kim
Keyword(s):  
Feature Selection ◽  
Anomaly Detection ◽  
Network Traffic ◽  
Selection Process ◽  
Traffic Data ◽  
Ensemble Techniques ◽  
Building Models ◽  
Comparable Performance ◽  
Network Anomaly Detection ◽  
Feature Selection Techniques

Variable selection (also known as feature selection ) is essential to optimize the learning complexity by prioritizing features, particularly for a massive, high-dimensional dataset like network traffic data. In reality, however, it is not an easy task to effectively perform the feature selection despite the availability of the existing selection techniques. From our initial experiments, we observed that the existing selection techniques produce different sets of features even under the same condition (e.g., a static size for the resulted set). In addition, individual selection techniques perform inconsistently, sometimes showing better performance but sometimes worse than others, thereby simply relying on one of them would be risky for building models using the selected features. More critically, it is demanding to automate the selection process, since it requires laborious efforts with intensive analysis by a group of experts otherwise. In this article, we explore challenges in the automated feature selection with the application of network anomaly detection. We first present our ensemble approach that benefits from the existing feature selection techniques by incorporating them, and one of the proposed ensemble techniques based on greedy search works highly consistently showing comparable results to the existing techniques. We also address the problem of when to stop to finalize the feature elimination process and present a set of methods designed to determine the number of features for the reduced feature set. Our experimental results conducted with two recent network datasets show that the identified feature sets by the presented ensemble and stopping methods consistently yield comparable performance with a smaller number of features to conventional selection techniques.

scholarly journals Model fusion of deep neural networks for anomaly detection

2021 ◽  
Vol 8 (1) ◽  
Author(s):  
Nouar AlDahoul ◽  
Hezerul Abdul Karim ◽  
Abdulaziz Saleh Ba Wazir
Keyword(s):  
Neural Networks ◽  
Anomaly Detection ◽  
Network Traffic ◽  
Large Scale ◽  
Deep Neural Networks ◽  
Denial Of Service ◽  
Traffic Data ◽  
Model Fusion ◽  
Class Weight ◽  
Network Anomaly Detection

AbstractNetwork Anomaly Detection is still an open challenging task that aims to detect anomalous network traffic for security purposes. Usually, the network traffic data are large-scale and imbalanced. Additionally, they have noisy labels. This paper addresses the previous challenges and utilizes million-scale and highly imbalanced ZYELL’s dataset. We propose to train deep neural networks with class weight optimization to learn complex patterns from rare anomalies observed from the traffic data. This paper proposes a novel model fusion that combines two deep neural networks including binary normal/attack classifier and multi-attacks classifier. The proposed solution can detect various network attacks such as Distributed Denial of Service (DDOS), IP probing, PORT probing, and Network Mapper (NMAP) probing. The experiments conducted on a ZYELL’s real-world dataset show promising performance. It was found that the proposed approach outperformed the baseline model in terms of average macro Fβ score and false alarm rate by 17% and 5.3%, respectively.

A Hybrid Technique Using PCA and Wavelets in Network Traffic Anomaly Detection

2014 ◽  
Vol 6 (1) ◽  
pp. 17-53 ◽  
Author(s):  
Stevan Novakov ◽  
Chung-Horng Lung ◽  
Ioannis Lambadaris ◽  
Nabil Seddigh
Keyword(s):  
Spectral Analysis ◽  
Statistical Analysis ◽  
Wavelet Analysis ◽  
Anomaly Detection ◽  
Network Traffic ◽  
Hybrid Approach ◽  
Haar Wavelet ◽  
Wavelet Filtering ◽  
Analysis Technique ◽  
Network Anomaly Detection

Research into network anomaly detection has become crucial as a result of a significant increase in the number of computer attacks. Many approaches in network anomaly detection have been reported in the literature, but data or solutions typically are not freely available. Recently, a labeled network traffic flow dataset, Kyoto2006+, has been created and is publicly available. Most existing approaches using Kyoto2006+ for network anomaly detection apply various clustering techniques. This paper leverages existing well known statistical analysis and spectral analysis techniques for network anomaly detection. The first popular approach is a statistical analysis technique called Principal Component Analysis (PCA). PCA describes data in a new dimension to unlock otherwise hidden characteristics. The other well known spectral analysis technique is Haar Wavelet filtering analysis. It measures the amount and magnitude of abrupt changes in data. Both approaches have strengths and limitations. In response, this paper proposes a Hybrid PCA–Haar Wavelet Analysis. The hybrid approach first applies PCA to describe the data and then Haar Wavelet filtering for analysis. Based on prototyping and measurement, an investigation of the Hybrid PCA–Haar Wavelet Analysis technique is performed using the Kyoto2006+ dataset. The authors consider a number of parameters and present experimental results to demonstrate the effectiveness of the hybrid approach as compared to the two algorithms individually.

scholarly journals Learning Representations of Network Traffic Using Deep Neural Networks for Network Anomaly Detection: A Perspective towards Oil and Gas IT Infrastructures

Symmetry ◽  
2020 ◽  
Vol 12 (11) ◽  
pp. 1882
Author(s):  
Sheraz Naseer ◽  
Rao Faizan Ali ◽  
P.D.D Dominic ◽  
Yasir Saleem
Keyword(s):  
Anomaly Detection ◽  
Network Traffic ◽  
Oil And Gas ◽  
Resource Planning ◽  
Machine Learning Algorithms ◽  
Data Driven ◽  
Network Data ◽  
It Infrastructure ◽  
Industrial Automation ◽  
Network Anomaly Detection

Oil and Gas organizations are dependent on their IT infrastructure, which is a small part of their industrial automation infrastructure, to function effectively. The oil and gas (O&G) organizations industrial automation infrastructure landscape is complex. To perform focused and effective studies, Industrial systems infrastructure is divided into functional levels by The Instrumentation, Systems and Automation Society (ISA) Standard ANSI/ISA-95:2005. This research focuses on the ISA-95:2005 level-4 IT infrastructure to address network anomaly detection problem for ensuring the security and reliability of Oil and Gas resource planning, process planning and operations management. Anomaly detectors try to recognize patterns of anomalous behaviors from network traffic and their performance is heavily dependent on extraction time and quality of network traffic features or representations used to train the detector. Creating efficient representations from large volumes of network traffic to develop anomaly detection models is a time and resource intensive task. In this study we propose, implement and evaluate use of Deep learning to learn effective Network data representations from raw network traffic to develop data driven anomaly detection systems. Proposed methodology provides an automated and cost effective replacement of feature extraction which is otherwise a time and resource intensive task for developing data driven anomaly detectors. The ISCX-2012 dataset is used to represent ISA-95 level-4 network traffic because the O&G network traffic at this level is not much different than normal internet traffic. We trained four representation learning models using popular deep neural network architectures to extract deep representations from ISCX 2012 traffic flows. A total of sixty anomaly detectors were trained by authors using twelve conventional Machine Learning algorithms to compare the performance of aforementioned deep representations with that of a human-engineered handcrafted network data representation. The comparisons were performed using well known model evaluation parameters. Results showed that deep representations are a promising feature in engineering replacement to develop anomaly detection models for IT infrastructure security. In our future research, we intend to investigate the effectiveness of deep representations, extracted using ISA-95:2005 Level 2-3 traffic comprising of SCADA systems, for anomaly detection in critical O&G systems.

Network anomaly detection based on signal processing techniques

2013 ◽  
Vol 18 (1) ◽  
pp. 15-21
Author(s):  
Tomasz Andrysiak ◽  
Łukasz Saganowski ◽  
Mirosław Maszewski
Keyword(s):  
Signal Processing ◽  
Anomaly Detection ◽  
Network Traffic ◽  
Detection Method ◽  
Matching Pursuit ◽  
Matching Pursuit Decomposition ◽  
Network Anomaly Detection ◽  
Processing Techniques ◽  
Signal Processing Techniques

Abstract The article depicts possibility of using Matching Pursuit decomposition in order to recognize unspecified hazards in network traffic. Furthermore, the work aims to present feasible enhancements to the anomaly detection method, as well as their efficiency on the basis of a wide collection of pattern test traces.

Feature selection by multi-objective optimisation: Application to network anomaly detection by hierarchical self-organising maps

2014 ◽  
Vol 71 ◽  
pp. 322-338 ◽  
Author(s):  
Emiro de la Hoz ◽  
Eduardo de la Hoz ◽  
Andrés Ortiz ◽  
Julio Ortega ◽  
Antonio Martínez-Álvarez
Keyword(s):  
Feature Selection ◽  
Anomaly Detection ◽  
Multi Objective ◽  
Network Anomaly Detection

scholarly journals A New Design of Custom Optimized Cnn-Lstm Assists to Detect Network Anomaly Using Categorical Data

2021 ◽  
Author(s):  
Kanmani R ◽  
A.Christy Jeba Malar ◽  
Roopa V ◽  
Ranjani D ◽  
Suganya R
Keyword(s):  
Feature Selection ◽  
Anomaly Detection ◽  
Categorical Data ◽  
Imbalanced Data ◽  
Experimental Result ◽  
Training Dataset ◽  
Detection Model ◽  
Effective Performance ◽  
Network Anomaly Detection ◽  
System Effectiveness

Abstract For traditional intrusion detection model, the system effectiveness is fully based on training dataset and feature selection. During feature selection, it needs more labour charge and trusted mainly on expert’s knowledge. Moreover, the training dataset contains more imbalanced data which in terms model tends to be biased. Here, an automatic approach is introduced to correct deficiency in the system. In this paper, the author proposes novel network anomaly detection (NID) build using categorical data. A model has to be designed with modified form of deep neural network primarily utilized for detecting anomaly within the network. Custom CNN-LSTM with Harris Hawks Optimization (named as custom optimized CNN-LSTM) is designed as a new classifier majorly used to detect the anomaly from word cloud to distinguish the data with effective performance. The experimental result shows that the proposed method achieves a promising output for network anomaly detection.

Sign in / Sign up

Export Citation Format

Share Document