Robust Anomaly Detection for Large-Scale Sensor Data

2016 ◽  
Author(s):  
Aniket Chakrabarti ◽  
Manish Marwah ◽  
Martin Arlitt
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Sensor Data

scholarly journals Multi-task learning based Encoder-Decoder: A comprehensive detection and diagnosis system for multi-sensor data

2021 ◽  
Vol 13 (5) ◽  
pp. 168781402110131
Author(s):  
Junfeng Wu ◽  
Li Yao ◽  
Bin Liu ◽  
Zheyuan Ding ◽  
Lei Zhang
Keyword(s):  
Anomaly Detection ◽  
Event Detection ◽  
Large Scale ◽  
Multivariate Time Series ◽  
Sensor Data ◽  
Unified Framework ◽  
Diagnosis System ◽  
Learning Framework ◽  
Task Learning ◽  
Detection And Diagnosis

As more and more sensor data have been collected, automated detection, and diagnosis systems are urgently needed to lessen the increasing monitoring burden and reduce the risk of system faults. A plethora of researches have been done on anomaly detection, event detection, anomaly diagnosis respectively. However, none of current approaches can explore all these respects in one unified framework. In this work, a Multi-Task Learning based Encoder-Decoder (MTLED) which can simultaneously detect anomalies, diagnose anomalies, and detect events is proposed. In MTLED, feature matrix is introduced so that features are extracted for each time point and point-wise anomaly detection can be realized in an end-to-end way. Anomaly diagnosis and event detection share the same feature matrix with anomaly detection in the multi-task learning framework and also provide important information for system monitoring. To train such a comprehensive detection and diagnosis system, a large-scale multivariate time series dataset which contains anomalies of multiple types is generated with simulation tools. Extensive experiments on the synthetic dataset verify the effectiveness of MTLED and its multi-task learning framework, and the evaluation on a real-world dataset demonstrates that MTLED can be used in other application scenarios through transfer learning.

Scalable Anomaly Detection for Large-Scale Heterogeneous Data in Cloud Using Optimal Elliptic Curve Cryptography and Gaussian Kernel Fuzzy C-Means Clustering

2019 ◽  
Vol 29 (05) ◽  
pp. 2050074 ◽  
Author(s):  
P. Santhosh Kumar ◽  
Latha Parthiban
Keyword(s):  
Anomaly Detection ◽  
Elliptic Curve ◽  
Elliptic Curve Cryptography ◽  
Large Scale ◽  
Privacy Preserving ◽  
Sensor Data ◽  
Gaussian Kernel ◽  
Cloud Platform ◽  
Privacy And Security ◽  
Cloud Data

In most systems, a smart functionality is enabled through an essential vital service such as detecting anomalies from complex, large-scale and dynamic data. However, ensuring the privacy and security for the cloud data is the most crucial and challenging task in the present world. Moreover, it is important to safeguard the security of sensitive data and its privacy from unauthorized parties who are trying to access the data. Therefore, to accomplish this task, several encryption, decryption and key generation mechanisms were introduced in the existing works for privacy preserving in cloud platform. But, there still remain open issues such as increased communication overhead, reduced security and increased time consumption. Also, these existing works followed the symmetric key cryptographic mechanism for privacy preservation of data; hence, a single secret key is shared by several users for accessing the original data. Due to this fact, a high security risk arises and it allows unauthorized parties to access the data. Thus, this work introduces a cloud-based privacy preserving model for offering a scalable and reliable anomaly detection service for sensor data through holding the benefits of cloud resources. Also, this paper aims to impose a newly developed Elliptic Curve Cryptography-based Collective Decision Optimization (ECDO) approach over the proposed framework for improving the privacy and security of the data. Furthermore, to perform the data clustering computation we used the Gaussian kernel fuzzy [Formula: see text]-means clustering (GKFCM) algorithm within the cloud platform, especially for data partitioning and to classify the anomalies. Thus, the computational difficulties are limited by adopting this suitable privacy preserving model which collaborates a private server and a set of public servers through a cloud data center. Moreover, on encrypted data the granular anomaly detection operations are performed by the virtual nodes executed over public servers. Experimental validation was performed on four datasets resulting from Intel Labs publicly available sensor data. The experimental outcomes demonstrate the ability of the proposed framework in providing high anomaly detection accuracy without any degradation in data privacy.

Anomaly Detection in Sensor Data

2018 ◽  
Vol 18 (1) ◽  
pp. 20-32 ◽  
Author(s):  
Jong-Min Kim ◽  
Jaiwook Baik
Keyword(s):  
Anomaly Detection ◽  
Sensor Data

scholarly journals An edge-cloud collaboration architecture for pattern anomaly detection of time series in wireless sensor networks

2021 ◽  
Author(s):  
Cong Gao ◽  
Ping Yang ◽  
Yanping Chen ◽  
Zhongmin Wang ◽  
Yue Wang
Keyword(s):  
Time Series ◽  
Wireless Sensor Networks ◽  
Sensor Networks ◽  
Anomaly Detection ◽  
Estimation Method ◽  
Feature Representation ◽  
Sensor Data ◽  
Wireless Sensor ◽  
Data Sets ◽  
Edge Node

AbstractWith large deployment of wireless sensor networks, anomaly detection for sensor data is becoming increasingly important in various fields. As a vital data form of sensor data, time series has three main types of anomaly: point anomaly, pattern anomaly, and sequence anomaly. In production environments, the analysis of pattern anomaly is the most rewarding one. However, the traditional processing model cloud computing is crippled in front of large amount of widely distributed data. This paper presents an edge-cloud collaboration architecture for pattern anomaly detection of time series. A task migration algorithm is developed to alleviate the problem of backlogged detection tasks at edge node. Besides, the detection tasks related to long-term correlation and short-term correlation in time series are allocated to cloud and edge node, respectively. A multi-dimensional feature representation scheme is devised to conduct efficient dimension reduction. Two key components of the feature representation trend identification and feature point extraction are elaborated. Based on the result of feature representation, pattern anomaly detection is performed with an improved kernel density estimation method. Finally, extensive experiments are conducted with synthetic data sets and real-world data sets.

scholarly journals Outlier Detection Transilience-Probabilistic Model for Wind Tunnels Based on Sensor Data

Sensors ◽  
2021 ◽  
Vol 21 (7) ◽  
pp. 2532
Author(s):  
Encarna Quesada ◽  
Juan J. Cuadrado-Gallego ◽  
Miguel Ángel Patricio ◽  
Luis Usero
Keyword(s):  
Anomaly Detection ◽  
Outlier Detection ◽  
Research Work ◽  
Industrial Sector ◽  
Sensor Data ◽  
Correct Selection ◽  
Wind Tunnels ◽  
Data Set ◽  
First Case ◽  
Anomaly Analysis

Anomaly Detection research is focused on the development and application of methods that allow for the identification of data that are different enough—compared with the rest of the data set that is being analyzed—and considered anomalies (or, as they are more commonly called, outliers). These values mainly originate from two sources: they may be errors introduced during the collection or handling of the data, or they can be correct, but very different from the rest of the values. It is essential to correctly identify each type as, in the first case, they must be removed from the data set but, in the second case, they must be carefully analyzed and taken into account. The correct selection and use of the model to be applied to a specific problem is fundamental for the success of the anomaly detection study and, in many cases, the use of only one model cannot provide sufficient results, which can be only reached by using a mixture model resulting from the integration of existing and/or ad hoc-developed models. This is the kind of model that is developed and applied to solve the problem presented in this paper. This study deals with the definition and application of an anomaly detection model that combines statistical models and a new method defined by the authors, the Local Transilience Outlier Identification Method, in order to improve the identification of outliers in the sensor-obtained values of variables that affect the operations of wind tunnels. The correct detection of outliers for the variables involved in wind tunnel operations is very important for the industrial ventilation systems industry, especially for vertical wind tunnels, which are used as training facilities for indoor skydiving, as the incorrect performance of such devices may put human lives at risk. In consequence, the use of the presented model for outlier detection may have a high impact in this industrial sector. In this research work, a proof-of-concept is carried out using data from a real installation, in order to test the proposed anomaly analysis method and its application to control the correct performance of wind tunnels.

scholarly journals Using autonomous video to estimate the bottom-contact area of longline trap gear and presence–absence of sensitive benthic habitat

2018 ◽  
Vol 75 (5) ◽  
pp. 797-812 ◽  
Author(s):  
Beau Doherty ◽  
Samuel D.N. Johnson ◽  
Sean P. Cox
Keyword(s):  
Contact Area ◽  
Large Scale ◽  
Marine Protected Area ◽  
Cold Water ◽  
Sensor Data ◽  
Benthic Habitat ◽  
Commercial Fishing ◽  
Depth Sensor ◽  
Quality Of Data ◽  
Bottom Contact

Bottom longline hook and trap fishing gear can potentially damage sensitive benthic areas (SBAs) in the ocean; however, the large-scale risks to these habitats are poorly understood because of the difficulties in mapping SBAs and in measuring the bottom-contact area of longline gear. In this paper, we describe a collaborative academic–industry–government approach to obtaining direct presence–absence data for SBAs and to measuring gear interactions with seafloor habitats via a novel deepwater trap camera and motion-sensing systems on commercial longline traps for sablefish (Anoplopoma fimbria) within SGaan Kinghlas – Bowie Seamount Marine Protected Area. We obtained direct presence–absence observations of cold-water corals (Alcyonacea, Antipatharia, Pennatulacea, Stylasteridae) and sponges (Hexactinellida, Demospongiae) at 92 locations over three commercial fishing trips. Video, accelerometer, and depth sensor data were used to estimate a mean bottom footprint of 53 m2 for a standard sablefish trap, which translates to 3200 m2 (95% CI = 2400–3900 m2) for a 60-trap commercial sablefish longline set. Our successful collaboration demonstrates how research partnerships with commercial fisheries have potential for massive improvements in the quantity and quality of data needed for conducting SBA risk assessments over large spatial and temporal scales.

Sign in / Sign up

Export Citation Format

Share Document