scholarly journals Time Series Anomaly Detection for KPIs Based on Correlation Analysis and HMM

2021 ◽  
Vol 11 (23) ◽  
pp. 11353
Author(s):  
Zijing Shang ◽  
Yingjun Zhang ◽  
Xiuguo Zhang ◽  
Yun Zhao ◽  
Zhiying Cao ◽  
...  
Keyword(s):  
Time Series ◽  
Correlation Analysis ◽  
Anomaly Detection ◽  
Correlation Matrix ◽  
Time Series Prediction ◽  
Original Data ◽  
Data Sets ◽  
Public Data ◽  
Close Relationship ◽  
Residual Sequence

KPIs (Key Performance Indicators) in distributed systems may involve a variety of anomalies, which will lead to system failure and huge losses. Detecting KPI anomalies in the system is very important. This paper presents a time series anomaly detection method based on correlation analysis and HMM. Correlation analysis is used to obtain the correlation between abnormal KPIs in the system, thereby reducing the false alarm rate of anomaly detection. The HMM (Hidden Markov Model) is used for anomaly detection by finding the close relationship between abnormal KPIs. In our correlation analysis of abnormal KPIs, firstly, the time series prediction model (1D-CNN-TCN) is proposed. The residual sequence is obtained by calculating the residual between the predicted value and the actual value. The residual sequence can highlight the abnormal segment in each data point and improve the accuracy of anomaly screening. According to the obtained residual sequence, these abnormal KPIs are preliminarily screened out from the historical data. Next, KPI correlation analysis is performed, and the correlation score is obtained by adding a sliding window onto the obtained anomaly index residual sequence. The correlation analysis based on the residual sequence can eliminate the interference of the original data fluctuation itself. Then, a correlation matrix of abnormal KPIs is constructed using the obtained correlation scores. In anomaly detection, the constructed correlation matrix is processed to obtain the adaptive parameters of the HMM model, and the trained HMM is used to quickly discover the abnormal KPI that may cause a KPI anomaly. Experiments on public data sets show that the method obtains good results.

scholarly journals An edge-cloud collaboration architecture for pattern anomaly detection of time series in wireless sensor networks

2021 ◽  
Author(s):  
Cong Gao ◽  
Ping Yang ◽  
Yanping Chen ◽  
Zhongmin Wang ◽  
Yue Wang
Keyword(s):  
Time Series ◽  
Wireless Sensor Networks ◽  
Sensor Networks ◽  
Anomaly Detection ◽  
Estimation Method ◽  
Feature Representation ◽  
Sensor Data ◽  
Wireless Sensor ◽  
Data Sets ◽  
Edge Node

AbstractWith large deployment of wireless sensor networks, anomaly detection for sensor data is becoming increasingly important in various fields. As a vital data form of sensor data, time series has three main types of anomaly: point anomaly, pattern anomaly, and sequence anomaly. In production environments, the analysis of pattern anomaly is the most rewarding one. However, the traditional processing model cloud computing is crippled in front of large amount of widely distributed data. This paper presents an edge-cloud collaboration architecture for pattern anomaly detection of time series. A task migration algorithm is developed to alleviate the problem of backlogged detection tasks at edge node. Besides, the detection tasks related to long-term correlation and short-term correlation in time series are allocated to cloud and edge node, respectively. A multi-dimensional feature representation scheme is devised to conduct efficient dimension reduction. Two key components of the feature representation trend identification and feature point extraction are elaborated. Based on the result of feature representation, pattern anomaly detection is performed with an improved kernel density estimation method. Finally, extensive experiments are conducted with synthetic data sets and real-world data sets.

Hybrid Neural Models For Rice Yields Times Forecasting

2012 ◽  
Author(s):  
Ruhaidah Samsudin ◽  
Puteh Saad ◽  
Ani Shabri
Keyword(s):  
Time Series ◽  
Hybrid Model ◽  
Moving Average ◽  
Time Series Prediction ◽  
Arima Model ◽  
Data Sets ◽  
Ann Model ◽  
Yield Data ◽  
Rice Yields ◽  
Artificial Neural Network Ann

In this paper, time series prediction is considered as a problem of missing value. A model for the determination of the missing time series value is presented. The hybrid model integrating autoregressive intergrated moving average (ARIMA) and artificial neural network (ANN) model is developed to solve this problem. The developed models attempts to incorporate the linear characteristics of an ARIMA model and nonlinear patterns of ANN to create a hybrid model. In this study, time series modeling of rice yield data in Muda Irrigation area. Malaysia from 1995 to 2003 are considered. Experimental results with rice yields data sets indicate that the hybrid model improve the forecasting performance by either of the models used separately. Key words: ARIMA; Box and Jenkins; neural networks; rice yields; hybrid ANN model

scholarly journals Fuzzy Transfer Learning in Time Series Forecasting for Stock Market Prices

2021 ◽  
Author(s):  
Shanoli Samui Pal ◽  
Samarjit Kar
Keyword(s):  
Time Series ◽  
Stock Market ◽  
Prior Knowledge ◽  
Transfer Learning ◽  
Prediction Accuracy ◽  
Time Series Data ◽  
Time Series Prediction ◽  
Series Data ◽  
Data Sets ◽  
Market Data

Abstract Transfer learning involves transferring prior knowledge of solving similar problems in order to achieve quick and efficient solution. The aim of fuzzy transfer learning is to transfer prior knowledge in an imprecise environment. Time series like stock market data are non-linear in nature and movement of stock is uncertain, so it is quite difficult following the stock market and in decision making. In this study, we propose a method to forecast stock market time series in the situation when we can use prior experience to make decisions. Fuzzy transfer learning (FuzzyTL) is based on knowledge transfer in that and adapting rules obtained domain. Three different stock market time series data sets are used for comparative study. It is observed that the effect of knowledge transferring works well together with smoothing of dependent attributes as the stock market data fluctuate with time. Finally, we give an empirical application in Shenzhen stock market with larger data sets to demonstrate the performance of the model. We have explored FuzzyTL in time series prediction to unerstand the essence of FuzzyTL. We were working on the question of the capability of FuzzyTL in improving prediction accuracy. From the comparisons, it can be said fuzzy transfer learning with smoothing improves prediction accuracy efficiently.

scholarly journals Grid Partitioning for Anomaly Detection (Gpad) in High Density Distributed Environment for Mining Techniques

2019 ◽  
Vol 8 (6S3) ◽  
pp. 1156-1161
Keyword(s):  
Anomaly Detection ◽  
Original Data ◽  
Extraction Process ◽  
High Density ◽  
Data Sets ◽  
Distributed Environment ◽  
Data Set ◽  
Detection Scheme ◽  
Grid Partitioning ◽  
Data Points

Anomaly detection is the most important task in data mining techniques. This helps to increase the scalability, accuracy and efficiency. During the extraction process, the outsource may damage their original data set and that will be defined as the intrusion. To avoid the intrusion and maintain the anomaly detection in a high densely populated environment is another difficult task. For that purpose, Grid Partitioning for Anomaly Detection (GPAD) has been proposed for high density environment. This technique will detect the outlier using the grid partitioning approach and density based outlier detection scheme. Initially, all the data sets will be split in the grid format. Allocate the equal amount of data points to each grid. Compare the density of each grid to their neighbor grid in a zigzag manner. Based on the response, lesser density grid will be detected as outlier function as well as that grid will be eliminated. This proposed Grid Partitioning for Anomaly Detection (GPAD) has reduced the complexity and increases the accuracy and these will be proven in simulation part.

Sign in / Sign up

Export Citation Format

Share Document