scholarly journals Minimum Message Length in Hybrid ARMA and LSTM Model Forecasting

Entropy ◽  
2021 ◽  
Vol 23 (12) ◽  
pp. 1601
Author(s):  
Zheng Fang ◽  
David L. Dowe ◽  
Shelton Peiris ◽  
Dedi Rosadi
Keyword(s):  
Time Series ◽  
Environmental Science ◽  
Short Term Memory ◽  
Arima Model ◽  
Theoretic Approach ◽  
Arma Models ◽  
Information Theoretic ◽  
Minimum Message Length ◽  
Message Length ◽  
Real World Datasets

Modeling and analysis of time series are important in applications including economics, engineering, environmental science and social science. Selecting the best time series model with accurate parameters in forecasting is a challenging objective for scientists and academic researchers. Hybrid models combining neural networks and traditional Autoregressive Moving Average (ARMA) models are being used to improve the accuracy of modeling and forecasting time series. Most of the existing time series models are selected by information-theoretic approaches, such as AIC, BIC, and HQ. This paper revisits a model selection technique based on Minimum Message Length (MML) and investigates its use in hybrid time series analysis. MML is a Bayesian information-theoretic approach and has been used in selecting the best ARMA model. We utilize the long short-term memory (LSTM) approach to construct a hybrid ARMA-LSTM model and show that MML performs better than AIC, BIC, and HQ in selecting the model—both in the traditional ARMA models (without LSTM) and with hybrid ARMA-LSTM models. These results held on simulated data and both real-world datasets that we considered. We also develop a simple MML ARIMA model.

scholarly journals Minimum Message Length in Hybrid ARMA and LSTM Model Forecasting

2021 ◽  
Author(s):  
Zheng Fang ◽  
David L. Dowe ◽  
Shelton Peiris ◽  
Dedi Rosadi
Keyword(s):  
Time Series ◽  
Deep Learning ◽  
Short Term Memory ◽  
Moving Average ◽  
Arma Model ◽  
Autoregressive Moving Average ◽  
Information Theoretic ◽  
Minimum Message Length ◽  
Message Length ◽  
Long Short Term Memory

We investigate the power of time series analysis based on a variety of information-theoretic approaches from statistics (AIC, BIC) and machine learning (Minimum Message Length) - and we then compare their efficacy with traditional time series model and with hybrids involving deep learning. More specifically, we develop AIC, BIC and Minimum Message Length (MML) ARMA (autoregressive moving average) time series models - with this Bayesian information-theoretic MML ARMA modelling already being new work. We then study deep learning based algorithms in time series forecasting, using Long Short Term Memory (LSTM), and we then combine this with the ARMA modelling to produce a hybrid ARMA-LSTM prediction. Part of the purpose of the use of LSTM is to seek capture any hidden information in the residuals left from the traditional ARMA model. We show that MML not only outperforms earlier statistical approaches to ARMA modelling, but we further show that the hybrid MML ARMA-LSTM models outperform both ARMA models and LSTM models.

scholarly journals Minimum Message Length in Hybrid ARMA and LSTM Model Forecasting

2021 ◽  
Author(s):  
Zheng Fang ◽  
David L. Dowe ◽  
Shelton Peiris ◽  
Dedi Rosadi
Keyword(s):  
Time Series ◽  
Deep Learning ◽  
Short Term Memory ◽  
Moving Average ◽  
Arma Model ◽  
Autoregressive Moving Average ◽  
Information Theoretic ◽  
Minimum Message Length ◽  
Message Length ◽  
Long Short Term Memory

We investigate the power of time series analysis based on a variety of information-theoretic approaches from statistics (AIC, BIC) and machine learning (Minimum Message Length) - and we then compare their efficacy with traditional time series model and with hybrids involving deep learning. More specifically, we develop AIC, BIC and Minimum Message Length (MML) ARMA (autoregressive moving average) time series models - with this Bayesian information-theoretic MML ARMA modelling already being new work. We then study deep learning based algorithms in time series forecasting, using Long Short Term Memory (LSTM), and we then combine this with the ARMA modelling to produce a hybrid ARMA-LSTM prediction. Part of the purpose of the use of LSTM is to seek capture any hidden information in the residuals left from the traditional ARMA model. We show that MML not only outperforms earlier statistical approaches to ARMA modelling, but we further show that the hybrid MML ARMA-LSTM models outperform both ARMA models and LSTM models.

scholarly journals SIMIT: Subjectively Interesting Motifs in Time Series

Entropy ◽  
2019 ◽  
Vol 21 (6) ◽  
pp. 566 ◽  
Author(s):  
Junning Deng ◽  
Jefrey Lijffijt ◽  
Bo Kang ◽  
Tijl De Bie
Keyword(s):  
Time Series ◽  
Time Series Data ◽  
Synthetic Data ◽  
Series Data ◽  
Theoretic Approach ◽  
Information Theoretic ◽  
Novel Approach ◽  
Approximate Form ◽  
Real World Datasets ◽  
Information Theoretic Approach

Numerical time series data are pervasive, originating from sources as diverse as wearable devices, medical equipment, to sensors in industrial plants. In many cases, time series contain interesting information in terms of subsequences that recur in approximate form, so-called motifs. Major open challenges in this area include how one can formalize the interestingness of such motifs and how the most interesting ones can be found. We introduce a novel approach that tackles these issues. We formalize the notion of such subsequence patterns in an intuitive manner and present an information-theoretic approach for quantifying their interestingness with respect to any prior expectation a user may have about the time series. The resulting interestingness measure is thus a subjective measure, enabling a user to find motifs that are truly interesting to them. Although finding the best motif appears computationally intractable, we develop relaxations and a branch-and-bound approach implemented in a constraint programming solver. As shown in experiments on synthetic data and two real-world datasets, this enables us to mine interesting patterns in small or mid-sized time series.

scholarly journals An Information Theoretic Approach for Creating 3D Spatial Images from 4D Time Series Data

2017 ◽  
Vol 23 (S1) ◽  
pp. 100-101
Author(s):  
Willy Wriggers ◽  
Julio Kovacs ◽  
Federica Castellani ◽  
P. Thomas Vernier ◽  
Dean J. Krusienski
Keyword(s):  
Time Series ◽  
Time Series Data ◽  
Series Data ◽  
Theoretic Approach ◽  
Information Theoretic ◽  
Information Theoretic Approach

scholarly journals Information Guided Exploration of Scalar Values and Isocontours in Ensemble Datasets

Entropy ◽  
2018 ◽  
Vol 20 (7) ◽  
pp. 540 ◽  
Author(s):  
Subhashis Hazarika ◽  
Ayan Biswas ◽  
Soumya Dutta ◽  
Han-Wei Shen
Keyword(s):  
Weather Forecasting ◽  
Conditional Entropy ◽  
Specific Information ◽  
Theoretic Approach ◽  
Information Theoretic ◽  
Information Measures ◽  
New Information ◽  
Real World Datasets ◽  
The Individual ◽  
Information Theoretic Approach

Uncertainty of scalar values in an ensemble dataset is often represented by the collection of their corresponding isocontours. Various techniques such as contour-boxplot, contour variability plot, glyphs and probabilistic marching-cubes have been proposed to analyze and visualize ensemble isocontours. All these techniques assume that a scalar value of interest is already known to the user. Not much work has been done in guiding users to select the scalar values for such uncertainty analysis. Moreover, analyzing and visualizing a large collection of ensemble isocontours for a selected scalar value has its own challenges. Interpreting the visualizations of such large collections of isocontours is also a difficult task. In this work, we propose a new information-theoretic approach towards addressing these issues. Using specific information measures that estimate the predictability and surprise of specific scalar values, we evaluate the overall uncertainty associated with all the scalar values in an ensemble system. This helps the scientist to understand the effects of uncertainty on different data features. To understand in finer details the contribution of individual members towards the uncertainty of the ensemble isocontours of a selected scalar value, we propose a conditional entropy based algorithm to quantify the individual contributions. This can help simplify analysis and visualization for systems with more members by identifying the members contributing the most towards overall uncertainty. We demonstrate the efficacy of our method by applying it on real-world datasets from material sciences, weather forecasting and ocean simulation experiments.

scholarly journals Heterogeneous Graphical Granger Causality by Minimum Message Length

Entropy ◽  
2020 ◽  
Vol 22 (12) ◽  
pp. 1400
Author(s):  
Kateřina Hlaváčková-Schindler ◽  
Claudia Plant
Keyword(s):  
Time Series ◽  
Causal Inference ◽  
Granger Causality ◽  
Real Data ◽  
Maximum Likelihood Estimates ◽  
Target Time ◽  
Minimum Message Length ◽  
Message Length ◽  
Comparison Methods ◽  
Statistical Model Selection

The heterogeneous graphical Granger model (HGGM) for causal inference among processes with distributions from an exponential family is efficient in scenarios when the number of time observations is much greater than the number of time series, normally by several orders of magnitude. However, in the case of “short” time series, the inference in HGGM often suffers from overestimation. To remedy this, we use the minimum message length principle (MML) to determinate the causal connections in the HGGM. The minimum message length as a Bayesian information-theoretic method for statistical model selection applies Occam’s razor in the following way: even when models are equal in their measure of fit-accuracy to the observed data, the one generating the most concise explanation of data is more likely to be correct. Based on the dispersion coefficient of the target time series and on the initial maximum likelihood estimates of the regression coefficients, we propose a minimum message length criterion to select the subset of causally connected time series with each target time series and derive its form for various exponential distributions. We propose two algorithms—the genetic-type algorithm (HMMLGA) and exHMML to find the subset. We demonstrated the superiority of both algorithms in synthetic experiments with respect to the comparison methods Lingam, HGGM and statistical framework Granger causality (SFGC). In the real data experiments, we used the methods to discriminate between pregnancy and labor phase using electrohysterogram data of Islandic mothers from Physionet databasis. We further analysed the Austrian climatological time measurements and their temporal interactions in rain and sunny days scenarios. In both experiments, the results of HMMLGA had the most realistic interpretation with respect to the comparison methods. We provide our code in Matlab. To our best knowledge, this is the first work using the MML principle for causal inference in HGGM.

scholarly journals PRESEE: An MDL/MML Algorithm to Time-Series Stream Segmenting

2013 ◽  
Vol 2013 ◽  
pp. 1-11 ◽  
Author(s):  
Kaikuo Xu ◽  
Yexi Jiang ◽  
Mingjie Tang ◽  
Changan Yuan ◽  
Changjie Tang
Keyword(s):  
Time Series ◽  
Real Time ◽  
Processing Speed ◽  
Data Stream ◽  
Minimum Description Length ◽  
Data Types ◽  
Minimum Message Length ◽  
Message Length ◽  
Different Types ◽  
State Of Art

Time-series stream is one of the most common data types in data mining field. It is prevalent in fields such as stock market, ecology, and medical care. Segmentation is a key step to accelerate the processing speed of time-series stream mining. Previous algorithms for segmenting mainly focused on the issue of ameliorating precision instead of paying much attention to the efficiency. Moreover, the performance of these algorithms depends heavily on parameters, which are hard for the users to set. In this paper, we proposePRESEE(parameter-free, real-time, and scalable time-series stream segmenting algorithm), which greatly improves the efficiency of time-series stream segmenting. PRESEE is based on both MDL (minimum description length) and MML (minimum message length) methods, which could segment the data automatically. To evaluate the performance of PRESEE, we conduct several experiments on time-series streams of different types and compare it with the state-of-art algorithm. The empirical results show that PRESEE is very efficient for real-time stream datasets by improving segmenting speed nearly ten times. The novelty of this algorithm is further demonstrated by the application of PRESEE in segmenting real-time stream datasets from ChinaFLUX sensor networks data stream.

scholarly journals A Novel LSTM for Multivariate Time Series with Massive Missingness

Sensors ◽  
2020 ◽  
Vol 20 (10) ◽  
pp. 2832
Author(s):  
Nazanin Fouladgar ◽  
Kary Främling
Keyword(s):  
Time Series ◽  
Short Term Memory ◽  
Time Series Prediction ◽  
Multivariate Time Series ◽  
Streaming Data ◽  
Short Term ◽  
Time Intervals ◽  
Long Short Term Memory ◽  
Real World Datasets ◽  
Novel Model

Multivariate time series with missing data is ubiquitous when the streaming data is collected by sensors or any other recording instruments. For instance, the outdoor sensors gathering different meteorological variables may encounter low material sensitivity to specific situations, leading to incomplete information gathering. This is problematic in time series prediction with massive missingness and different missing rate of variables. Contribution addressing this problem on the regression task of meteorological datasets by employing Long Short-Term Memory (LSTM), capable of controlling the information flow with its memory unit, is still missing. In this paper, we propose a novel model called forward and backward variable-sensitive LSTM (FBVS-LSTM) consisting of two decay mechanisms and some informative data. The model inputs are mainly the missing indicator, time intervals of missingness in both forward and backward direction and missing rate of each variable. We employ this information to address the so-called missing not at random (MNAR) mechanism. Separately learning the features of each parameter, the model becomes adapted to deal with massive missingness. We conduct our experiment on three real-world datasets for the air pollution forecasting. The results demonstrate that our model performed well along with other LSTM-derivation models in terms of prediction accuracy.

Sign in / Sign up

Export Citation Format

Share Document