scholarly journals Anomaly Detection for the Centralised Elasticsearch Service at CERN

2021 ◽  
Vol 4 ◽  
Author(s):  
Jennifer R. Andersson ◽  
Jose Alonso Moya ◽  
Ulrich Schwickerath
Keyword(s):  
Anomaly Detection ◽  
Real Time ◽  
Distributed System ◽  
Detection System ◽  
Current System ◽  
Moving Average ◽  
Data Retrieval ◽  
Huge Amount ◽  
Future System ◽  
User Communities

For several years CERN has been offering a centralised service for Elasticsearch, a popular distributed system for search and analytics of user provided data. The service offered by CERN IT is better described as a service of services, delivering centrally managed and maintained Elasticsearch instances to CERN users who have a justified need for it. This dynamic infrastructure currently consists of about 30 distinct and independent Elasticsearch installations, in the following referred to as Elasticsearch clusters, some of which are shared between different user communities. The service is used by several hundred users mainly for logs and service analytics. Due to its size and complexity, the installation produces a huge amount of internal monitoring data which can be difficult to process in real time with limited available person power. Early on, an idea was therefore born to process this data automatically, aiming to extract anomalies and possible issues building up in real time, allowing the experts to address them before they start to cause an issue for the users of the service. Both deep learning and traditional methods have been applied to analyse the data in order to achieve this goal. This resulted in the current deployment of an anomaly detection system based on a one layer multi dimensional LSTM neural network, coupled with applying a simple moving average to the data to validate the results. This paper will describe which methods were investigated and give an overview of the current system, including data retrieval, data pre-processing and analysis. In addition, reports on experiences gained when applying the system to actual data will be provided. Finally, weaknesses of the current system will be briefly discussed, and ideas for future system improvements will be sketched out.

scholarly journals Anomaly Detection and Classification of Physiological Signals in IoT- Based Healthcare Framework

2021 ◽  
Author(s):  
Menaa Nawaz ◽  
Jameel Ahmed
Keyword(s):  
Deep Learning ◽  
Anomaly Detection ◽  
Real Time ◽  
Signal Analysis ◽  
Detection System ◽  
Muscular Activity ◽  
Physiological Signals ◽  
Time Data ◽  
Cloud Data ◽  
Medical Sensors

Abstract Physiological signals retrieve the information from sensors implanted or attached to the human body. These signals are vital data sources that can assist in predicting the disease well before time and thus proper treatment can be made possible. With the addition of Internet of Things in healthcare, real-time data collection and pre-processing for signal analysis has reduced burden of in-person appointments and decision making on healthcare. Recently, Deep learning-based algorithms have been implemented by researchers for recognition, realization and prediction of diseases by extracting and analyzing the important features. In this research real-time 1-D timeseries data of on-body non-invasive bio-medical sensors have been acquired and pre-processed and analyzed for anomaly detection. Feature engineered parameters of large and diverse dataset have been used to train the data to make the anomaly detection system more reliable. For comprehensive real-time monitoring the implemented system uses wavelet time scattering features for classification and deep learning based autoencoder for anomaly detection of time series signals for assisting the clinical diagnosis of cardiovascular and muscular activity. In this research, an implementation of IoT based healthcare system using bio-medical sensors has been presented. This paper also aims to provide the analysis of cloud data acquired through bio-medical sensors using signal analysis techniques for anomaly detection and timeseries classification has been done for the disease prognosis in real-time. Wavelet time scattering based signals classification accuracy of 99.88% is achieved. In real time signals anomaly detection, 98% accuracy is achieved. The average Mean Absolute Error loss of 0.0072 for normal signals and 0.078 is achieved for anomaly signals.

scholarly journals Anomaly Intrusion Detection System in Real Time Environment using Ensemble Learning Model

2019 ◽  
Vol 8 (4) ◽  
pp. 4908-4917
Keyword(s):  
Intrusion Detection ◽  
Anomaly Detection ◽  
Real Time ◽  
Network Traffic ◽  
Intrusion Detection System ◽  
Detection System ◽  
False Negative ◽  
Ensemble Classification ◽  
Data Set ◽  
Rule Set

System security is of essential part now days for huge organizations. The Intrusion Detection System (IDS) are getting to be irreplaceable for successful assurance against intrusions that are continually changing in size and intricacy. With information honesty, privacy and accessibility, they must be solid, simple to oversee and with low upkeep cost. Different adjustments are being connected to IDS consistently to recognize new intrusions and handle them. This paper proposes model based on combination of ensemble classification for network traffic anomaly detection. Intrusion detection system is try to perform in real time, but they cannot improved due to the network connections. This research paper is trying to implement intrusion detection system (IDS) using ensemble method for misuse as well anomaly detection for HIDS and NIDS based also. This system used various individual classification methods and its ensemble model on KDD99 and NSL-KDD data set to check the performance of model. It also check the performance on creating real time network traffic using own attack creator and send this to the remote machine which has our proposed IDS system. This system used training rule set as a background knowledge which are generated by genetic algorithm. Ensemble approach contains three algorithms as Naive Bayes, Artificial Neural Network and J48. Ensemble classifiers apply on network packets mapping with GA rule set and generate the result. Finally our proposed model produces highest detection rate and lower false negative ratio compare to others. Also find the accuracy of each attack types.

Predicting Imminent Crash Risk with Simulated Traffic from Distant Sensors

2018 ◽  
Vol 2672 (38) ◽  
pp. 12-21
Author(s):  
Zhi Chen ◽  
Xiao Qin ◽  
Renxin Zhong ◽  
Pan Liu ◽  
Yang Cheng
Keyword(s):  
Real Time ◽  
Prediction Models ◽  
Detection System ◽  
Data Retrieval ◽  
Transmission Model ◽  
Traffic Data ◽  
Crash Prediction ◽  
Station Data ◽  
Virtual Station ◽  
Crash Prediction Models

The aim of this research was to investigate the performance of simulated traffic data for real-time crash prediction when loop detector stations are distant from the actual crash location. Nearly all contemporary real-time crash prediction models use traffic data from physical detector stations; however, the distance between a crash location and its nearest detector station can vary considerably from site to site, creating inconsistency in detector data retrieval and subsequent crash prediction. Moreover, large distances between crash locations and detector stations imply that traffic data from these stations may not truly reflect crash-prone conditions. Crash and noncrash events were identified for a freeway section on I-94 EB in Wisconsin. The cell transmission model (CTM), a macroscopic simulation model, was applied in this study to instrument segments with virtual detector stations when physical stations were not available near the crash location. Traffic data produced from the virtual stations were used to develop crash prediction models. A comparison revealed that the predictive accuracy of models developed with virtual station data was comparable to those developed with physical station data. The finding demonstrates that simulated traffic data are a viable option for real-time crash prediction given distant detector stations. The proposed approach can be used in the real-time crash detection system or in a connected vehicle environment with different settings.

Sign in / Sign up

Export Citation Format

Share Document