scholarly journals On the proper choice of datasets and traffic features for real-time anomaly detection

2021 ◽  
Vol 2091 (1) ◽  
pp. 012001
Author(s):  
C Callegari ◽  
S Giordano ◽  
M Pagano
Keyword(s):  
Intrusion Detection ◽  
Anomaly Detection ◽  
Real Time ◽  
Detection System ◽  
Detection Algorithm ◽  
Performance Comparison ◽  
Critical Element ◽  
Correct Identification ◽  
Data Set ◽  
On Line

Abstract Thanks to its ability to face unknown attacks, Anomaly-based Intrusion Detection is a key research topic in network security and different statistical methods, fed by suitable traffic features, have been proposed in the literature. The choice of a proper dataset is a critical element not only for performance comparison, but also for the correct identification of the normal traffic behaviour. In this paper we address the general problem of selecting traffic features from recent real traffic traces (MAWI data set) and verify how the real-time constraint impacts on the general performance. Although a state-of-the-art IDS (Intrusion Detection System) based on deep neural networks is considered, our conclusions can be extended to any anomaly detection algorithm and advocate for a fair comparison of IDSs using representative datasets and traffic features that can be extracted on-line (and do not depend on the entire dataset).

scholarly journals Anomaly Intrusion Detection System in Real Time Environment using Ensemble Learning Model

2019 ◽  
Vol 8 (4) ◽  
pp. 4908-4917
Keyword(s):  
Intrusion Detection ◽  
Anomaly Detection ◽  
Real Time ◽  
Network Traffic ◽  
Intrusion Detection System ◽  
Detection System ◽  
False Negative ◽  
Ensemble Classification ◽  
Data Set ◽  
Rule Set

System security is of essential part now days for huge organizations. The Intrusion Detection System (IDS) are getting to be irreplaceable for successful assurance against intrusions that are continually changing in size and intricacy. With information honesty, privacy and accessibility, they must be solid, simple to oversee and with low upkeep cost. Different adjustments are being connected to IDS consistently to recognize new intrusions and handle them. This paper proposes model based on combination of ensemble classification for network traffic anomaly detection. Intrusion detection system is try to perform in real time, but they cannot improved due to the network connections. This research paper is trying to implement intrusion detection system (IDS) using ensemble method for misuse as well anomaly detection for HIDS and NIDS based also. This system used various individual classification methods and its ensemble model on KDD99 and NSL-KDD data set to check the performance of model. It also check the performance on creating real time network traffic using own attack creator and send this to the remote machine which has our proposed IDS system. This system used training rule set as a background knowledge which are generated by genetic algorithm. Ensemble approach contains three algorithms as Naive Bayes, Artificial Neural Network and J48. Ensemble classifiers apply on network packets mapping with GA rule set and generate the result. Finally our proposed model produces highest detection rate and lower false negative ratio compare to others. Also find the accuracy of each attack types.

The Study of Invasion Examination Algorithm Based on Improvement Fuzzy C Means

2011 ◽  
Vol 267 ◽  
pp. 720-725
Author(s):  
Ke Chen ◽  
Wen De Ke
Keyword(s):  
Intrusion Detection ◽  
Anomaly Detection ◽  
Intrusion Detection System ◽  
Detection System ◽  
Detection Algorithm ◽  
Initial Value ◽  
Data Set ◽  
Fuzzy C Means ◽  
Improvement Algorithm ◽  
Selection Of

This paper put forward intrusion detection algorithm based on improved fuzzy C means (FCM) algorithm and execute the anomaly detection on KDDCUP data set, build intrusion detection system based improved algorithm and analyze the feasibility of the system. Through the fuzzy C means value's improvement algorithm, solve the fuzzy C means value algorithm problem that the algorithm sensitive to selection of the initial values and easily to fall in the local best solution. Thereby under the condition guarantee integrality and consistency of data attribute values, get rid of blindness of selecting initial value and reduce clustering time and algorithm complexity, enhance speed of the algorithm.

Intrusion detection algorithm based on image enhanced convolutional neural network

2021 ◽  
pp. 1-12
Author(s):  
Qian Wang ◽  
Wenfang Zhao ◽  
Jiadong Ren
Keyword(s):  
Neural Network ◽  
Intrusion Detection ◽  
Convolutional Neural Network ◽  
Detection System ◽  
Binary Classification ◽  
Original Data ◽  
Detection Algorithm ◽  
Machine Learning Algorithms ◽  
Data Set ◽  
Softmax Classifier

Intrusion Detection System (IDS) can reduce the losses caused by intrusion behaviors and protect users’ information security. The effectiveness of IDS depends on the performance of the algorithm used in identifying intrusions. And traditional machine learning algorithms are limited to deal with the intrusion data with the characteristics of high-dimensionality, nonlinearity and imbalance. Therefore, this paper proposes an Intrusion Detection algorithm based on Image Enhanced Convolutional Neural Network (ID-IE-CNN). Firstly, based on the image processing technology of deep learning, oversampling method is used to increase the amount of original data to achieve data balance. Secondly, the one-dimensional data is converted into two-dimensional image data, the convolutional layer and the pooling layer are used to extract the main features of the image to reduce the data dimensionality. Third, the Tanh function is introduced as an activation function to fit nonlinear data, a fully connected layer is used to integrate local information, and the generalization ability of the prediction model is improved by the Dropout method. Finally, the Softmax classifier is used to predict the behavior of intrusion detection. This paper uses the KDDCup99 data set and compares with other competitive algorithms. Both in the performance of binary classification and multi-classification, ID-IE-CNN is better than the compared algorithms, which verifies its superiority.

scholarly journals Anomaly Detection using Optimized Features using Genetic Algorithm and MultiEnsemble Classifier

IJOSTHE ◽  
2018 ◽  
Vol 5 (6) ◽  
pp. 7
Author(s):  
Apoorva Deshpande ◽  
Ramnaresh Sharma
Keyword(s):  
Machine Learning ◽  
Genetic Algorithm ◽  
Intrusion Detection ◽  
Anomaly Detection ◽  
Detection System ◽  
Research Work ◽  
Ensemble Classifier ◽  
Machine Learning Algorithms ◽  
Data Set ◽  
Machine Learning Classification

Anomaly detection system plays an important role in network security. Anomaly detection or intrusion detection model is a predictive model used to predict the network data traffic as normal or intrusion. Machine Learning algorithms are used to build accurate models for clustering, classification and prediction. In this paper classification and predictive models for intrusion detection are built by using machine learning classification algorithms namely Random Forest. These algorithms are tested with KDD-99 data set. In this research work the model for anomaly detection is based on normalized reduced feature and multilevel ensemble classifier. The work is performed in divided into two stages. In the first stage data is normalized using mean normalization. In second stage genetic algorithm is used to reduce number of features and further multilevel ensemble classifier is used for classification of data into different attack groups. From result analysis it is analysed that with reduced feature intrusion can be classified more efficiently.

An Adaptive Intrusion Detection Scheme for Cloud Computing

2019 ◽  
Vol 10 (4) ◽  
pp. 53-70
Author(s):  
Nurudeen Mahmud Ibrahim ◽  
Anazida Zainal
Keyword(s):  
Cloud Computing ◽  
Intrusion Detection ◽  
Anomaly Detection ◽  
Virtual Machine ◽  
Detection System ◽  
Performance Comparison ◽  
Change Point Detection ◽  
Virtual Machine Migration ◽  
Detection Scheme ◽  
Adaptive Scheme

To provide dynamic resource management, live virtual machine migration is used to move a virtual machine from one host to another. However, virtual machine migration poses challenges to cloud intrusion detection systems because movement of VMs from one host to another makes it difficult to create a consistent normal profile for anomaly detection. Hence, there is a need to provide an adaptive anomaly detection system capable of adapting to changes that occur in the cloud data during VM migration. To achieve this, the authors proposed a scheme for adaptive IDS for Cloud computing. The proposed adaptive scheme is comprised of four components: an ant colony optimization-based feature selection component, a statistical time series change point detection component, adaptive classification, and model update component, and a detection component. The proposed adaptive scheme was evaluated using simulated datasets collected from vSphere and performance comparison shows improved performance over existing techniques.

scholarly journals Network Abnormal Data Detection Based on Deep Learning Model

CONVERTER ◽  
2021 ◽  
pp. 64-73
Author(s):  
Yang Dong
Keyword(s):  
Deep Learning ◽  
Real Time ◽  
Detection System ◽  
Feature Learning ◽  
Original Data ◽  
Detection Algorithm ◽  
Training Data ◽  
Data Detection ◽  
Classification Problems ◽  
Data Set

To improve intrusion detection system performance,many algorithms are used to improve the performance of IDS systems, especially deep learning models. This paper presents an algorithm based on the model MLP, the training data set is the KDD99 data set, and the original data of the data set is vectorized by one-hot encoding, and the feature data is processed by Z-Score, and then the feature vector is encoded, and then the multi-layer perception is used The machine network performs feature learning, and finally trains the classifier model for detection. Traditional network anomaly detection algorithm models mainly use manual selection methods, and the accuracy and efficiency of classification problems are not high. This article first proposed the role of multilayer perceptron in Adam optimizer. The test of the KDD99 data set has been completed. The algorithm accuracy rate can reach 99%. For future network abnormal data detection work, an algorithm model that can realize real-time online detection is provided, which will have higher accuracy and better real-time performance.

scholarly journals Intrusion Detection Based on Gray-Level Co-Occurrence Matrix and 2D Dispersion Entropy

2021 ◽  
Vol 11 (12) ◽  
pp. 5567
Author(s):  
Gianmarco Baldini ◽  
Jose Luis Hernandez Ramos ◽  
Irene Amerini
Keyword(s):  
Intrusion Detection ◽  
Network Flows ◽  
Learning Algorithm ◽  
Detection System ◽  
Detection Algorithm ◽  
Entropy Measure ◽  
Detection Accuracy ◽  
Gray Level ◽  
Data Set ◽  
Occurrence Matrix

The Intrusion Detection System (IDS) is an important tool to mitigate cybersecurity threats in an Information and Communication Technology (ICT) infrastructure. The function of the IDS is to detect an intrusion to an ICT system or network so that adequate countermeasures can be adopted. Desirable features of IDS are computing efficiency and high intrusion detection accuracy. This paper proposes a new anomaly detection algorithm for IDS, where a machine learning algorithm is applied to detect deviations from legitimate traffic, which may indicate an intrusion. To improve computing efficiency, a sliding window approach is applied where the analysis is applied on large sequences of network flows statistics. This paper proposes a novel approach based on the transformation of the network flows statistics to gray images on which Gray level Co-occurrence Matrix (GLCM) are applied together with an entropy measure recently proposed in literature: the 2D Dispersion Entropy. This approach is applied to the recently public IDS data set CIC-IDS2017. The results show that the proposed approach is competitive in comparison to other approaches proposed in literature on the same data set. The approach is applied to two attacks of the CIC-IDS2017 data set: DDoS and Port Scan achieving respectively an Error Rate of 0.0016 and 0.0048.

scholarly journals Anomaly Detection for Individual Sequences with Applications in Identifying Malicious Tools

Entropy ◽  
2020 ◽  
Vol 22 (6) ◽  
pp. 649 ◽  
Author(s):  
Shachar Siboni ◽  
Asaf Cohen
Keyword(s):  
New York ◽  
Computer Security ◽  
Anomaly Detection ◽  
Statistical Model ◽  
Detection System ◽  
Detection Algorithm ◽  
Normal Behaviour ◽  
Data Set ◽  
Probability Of Error ◽  
Individual Sequences

Anomaly detection refers to the problem of identifying abnormal behaviour within a set of measurements. In many cases, one has some statistical model for normal data, and wishes to identify whether new data fit the model or not. However, in others, while there are normal data to learn from, there is no statistical model for this data, and there is no structured parameter set to estimate. Thus, one is forced to assume an individual sequences setup, where there is no given model or any guarantee that such a model exists. In this work, we propose a universal anomaly detection algorithm for one-dimensional time series that is able to learn the normal behaviour of systems and alert for abnormalities, without assuming anything on the normal data, or anything on the anomalies. The suggested method utilizes new information measures that were derived from the Lempel–Ziv (LZ) compression algorithm in order to optimally and efficiently learn the normal behaviour (during learning), and then estimate the likelihood of new data (during operation) and classify it accordingly. We apply the algorithm to key problems in computer security, as well as a benchmark anomaly detection data set, all using simple, single-feature time-indexed data. The first is detecting Botnets Command and Control (C&C) channels without deep inspection. We then apply it to the problems of malicious tools detection via system calls monitoring and data leakage identification.We conclude with the New York City (NYC) taxi data. Finally, while using information theoretic tools, we show that an attacker’s attempt to maliciously fool the detection system by trying to generate normal data is bound to fail, either due to a high probability of error or because of the need for huge amounts of resources.

scholarly journals Development of Scalable On-Line Anomaly Detection System for Autonomous and Adaptive Manufacturing Processes

2019 ◽  
Vol 9 (21) ◽  
pp. 4502 ◽  
Author(s):  
Seunghyun Choi ◽  
Sekyoung Youm ◽  
Yong-Shin Kang
Keyword(s):  
Anomaly Detection ◽  
Real Time ◽  
Large Scale ◽  
Detection System ◽  
Manufacturing Processes ◽  
Alignment Algorithm ◽  
Architecture Framework ◽  
On Line ◽  
Smart Factories ◽  
Anomaly Detection System

Factories of the future are foreseen to evolve into smart factories with autonomous and adaptive manufacturing processes. However, the increasing complexity of the network of manufacturing processes is expected to complicate the rapid detection of process anomalies in real time. This paper proposes an architecture framework and method for the implementation of the Scalable On-line Anomaly Detection System (SOADS), which can detect process anomalies via real-time processing and analyze large amounts of process execution data in the context of autonomous and adaptive manufacturing processes. The design of this system architecture framework entailed the derivation of standard subsequence patterns using the PrefixSpan algorithm, a sequential pattern algorithm. The anomalies of the real-time event streams and derived subsequence patterns were scored using the Smith-Waterman algorithm, a sequence alignment algorithm. The excellence of the proposed system was verified by measuring the time for deriving subsequence patterns and by obtaining the anomaly scoring time from large event logs. The proposed system succeeded in large-scale data processing and analysis, one of the requirements for a smart factory, by using Apache Spark streaming and Apache Hbase, and is expected to become the basis of anomaly detection systems of smart factories.

Sign in / Sign up

Export Citation Format

Share Document