A New Attention Mechanism to Classify Multivariate Time Series

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2020/277 ◽

2020 ◽

Author(s):

Yifan Hao ◽

Huiping Cao

Keyword(s):

Neural Network ◽

Time Series ◽

State Of The Art ◽

Multivariate Time Series ◽

Attention Mechanism ◽

Classification Methods ◽

Short Term ◽

Time Step ◽

Relevant Variables ◽

Multiple Variables

Classifying multivariate time series (MTS), which record the values of multiple variables over a continuous period of time, has gained a lot of attention. However, existing techniques suffer from two major issues. First, the long-range dependencies of the time-series sequences are not well captured. Second, the interactions of multiple variables are generally not represented in features. To address these aforementioned issues, we propose a novel Cross Attention Stabilized Fully Convolutional Neural Network (CA-SFCN) to classify MTS data. First, we introduce a temporal attention mechanism to extract long- and short-term memories across all time steps. Second, variable attention is designed to select relevant variables at each time step. CA-SFCN is compared with 16 approaches using 14 different MTS datasets. The extensive experimental results show that the CA-SFCN outperforms state-of-the-art classification methods, and the cross attention mechanism achieves better performance than other attention mechanisms.

Download Full-text

Compound Autoregressive Network for Prediction of Multivariate Time Series

Complexity ◽

10.1155/2019/9107167 ◽

2019 ◽

Vol 2019 ◽

pp. 1-11 ◽

Cited By ~ 13

Author(s):

Yuting Bai ◽

Xuebo Jin ◽

Xiaoyi Wang ◽

Tingli Su ◽

Jianlei Kong ◽

...

Keyword(s):

Time Series ◽

Time Series Prediction ◽

Multivariate Time Series ◽

Prediction Algorithm ◽

Short Term ◽

Advanced Control ◽

Multiple Variables ◽

Nonlinear Autoregressive Model ◽

Interactive Relation ◽

Nonlinear Autoregressive

The prediction information has effects on the emergency prevention and advanced control in various complex systems. There are obvious nonlinear, nonstationary, and complicated characteristics in the time series. Moreover, multiple variables in the time-series impact on each other to make the prediction more difficult. Then, a solution of time-series prediction for the multivariate was explored in this paper. Firstly, a compound neural network framework was designed with the primary and auxiliary networks. The framework attempted to extract the change features of the time series as well as the interactive relation of multiple related variables. Secondly, the structures of the primary and auxiliary networks were studied based on the nonlinear autoregressive model. The learning method was also introduced to obtain the available models. Thirdly, the prediction algorithm was concluded for the time series with multiple variables. Finally, the experiments on environment-monitoring data were conducted to verify the methods. The results prove that the proposed method can obtain the accurate prediction value in the short term.

Download Full-text

Stacked Boosters Network Architecture for Short-Term Load Forecasting in Buildings

Energies ◽

10.3390/en13092370 ◽

2020 ◽

Vol 13 (9) ◽

pp. 2370 ◽

Cited By ~ 1

Author(s):

Tuukka Salmi ◽

Jussi Kiljander ◽

Daniel Pakkala

Keyword(s):

Time Series ◽

Network Architecture ◽

State Of The Art ◽

Multivariate Time Series ◽

Load Forecasting ◽

Building Energy ◽

Short Term ◽

Univariate Time Series ◽

Short Term Load Forecasting ◽

Base Learner

This paper presents a novel deep learning architecture for short-term load forecasting of building energy loads. The architecture is based on a simple base learner and multiple boosting systems that are modelled as a single deep neural network. The architecture transforms the original multivariate time series into multiple cascading univariate time series. Together with sparse interactions, parameter sharing and equivariant representations, this approach makes it possible to combat against overfitting while still achieving good presentation power with a deep network architecture. The architecture is evaluated in several short-term load forecasting tasks with energy data from an office building in Finland. The proposed architecture outperforms state-of-the-art load forecasting model in all the tasks.

Download Full-text

A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2017/366 ◽

2017 ◽

Cited By ~ 131

Author(s):

Yao Qin ◽

Dongjin Song ◽

Haifeng Chen ◽

Wei Cheng ◽

Guofei Jiang ◽

...

Keyword(s):

Neural Network ◽

Time Series ◽

Recurrent Neural Network ◽

Empirical Studies ◽

Time Series Prediction ◽

Attention Mechanism ◽

Time Step ◽

Narx Models ◽

Dual Stage ◽

Hidden States

The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them can capture the long-term temporal dependencies appropriately and select the relevant driving series to make predictions. In this paper, we propose a dual-stage attention-based recurrent neural network (DA-RNN) to address these two issues. In the first stage, we introduce an input attention mechanism to adaptively extract relevant driving series (a.k.a., input features) at each time step by referring to the previous encoder hidden state. In the second stage, we use a temporal attention mechanism to select relevant encoder hidden states across all time steps. With this dual-stage attention scheme, our model can not only make predictions effectively, but can also be easily interpreted. Thorough empirical studies based upon the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can outperform state-of-the-art methods for time series prediction.

Download Full-text

XCM: An Explainable Convolutional Neural Network for Multivariate Time Series Classification

Mathematics ◽

10.3390/math9233137 ◽

2021 ◽

Vol 9 (23) ◽

pp. 3137

Author(s):

Kevin Fauvel ◽

Tao Lin ◽

Véronique Masson ◽

Élisa Fromont ◽

Alexandre Termier

Keyword(s):

Neural Network ◽

Time Series ◽

Deep Learning ◽

Convolutional Neural Network ◽

Input Data ◽

State Of The Art ◽

Multivariate Time Series ◽

Learning Approach ◽

Multiple Domains ◽

Post Hoc

Multivariate Time Series (MTS) classification has gained importance over the past decade with the increase in the number of temporal datasets in multiple domains. The current state-of-the-art MTS classifier is a heavyweight deep learning approach, which outperforms the second-best MTS classifier only on large datasets. Moreover, this deep learning approach cannot provide faithful explanations as it relies on post hoc model-agnostic explainability methods, which could prevent its use in numerous applications. In this paper, we present XCM, an eXplainable Convolutional neural network for MTS classification. XCM is a new compact convolutional neural network which extracts information relative to the observed variables and time directly from the input data. Thus, XCM architecture enables a good generalization ability on both large and small datasets, while allowing the full exploitation of a faithful post hoc model-specific explainability method (Gradient-weighted Class Activation Mapping) by precisely identifying the observed variables and timestamps of the input data that are important for predictions. We first show that XCM outperforms the state-of-the-art MTS classifiers on both the large and small public UEA datasets. Then, we illustrate how XCM reconciles performance and explainability on a synthetic dataset and show that XCM enables a more precise identification of the regions of the input data that are important for predictions compared to the current deep learning MTS classifier also providing faithful explainability. Finally, we present how XCM can outperform the current most accurate state-of-the-art algorithm on a real-world application while enhancing explainability by providing faithful and more informative explanations.

Download Full-text

MULTIVARIATE TIME SERIES MODELING IN A CONNECTIONIST APPROACH

International Journal of Modern Physics C ◽

10.1142/s0129183100000146 ◽

2000 ◽

Vol 11 (01) ◽

pp. 159-173

Author(s):

D. R. KULKARNI ◽

J. C. PARIKH

Keyword(s):

Neural Network ◽

Time Series ◽

Time Series Data ◽

Multivariate Time Series ◽

Series Data ◽

Time Series Modeling ◽

Multivariate Models ◽

Short Term ◽

Several Variables ◽

Connectionist Approach

Multivariate models in the framework of artificial neural network have been constructed for systems where time series data of several variables is known. The models have been tested using computer generated data for the Lorenz and Henon systems. They are found to be robust and give accurate short term predictions. Analysis of the models is able to throw some light on theoretical questions related to multivariate "embedding" and removal of redundancy in the embedding.

Download Full-text

Tensorized LSTM with Adaptive Shared Memory for Learning Trends in Multivariate Time Series

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i02.5496 ◽

2020 ◽

Vol 34 (02) ◽

pp. 1395-1402

Author(s):

Dongkuan Xu ◽

Wei Cheng ◽

Bo Zong ◽

Dongjin Song ◽

Jingchao Ni ◽

...

Keyword(s):

Time Series ◽

Shared Memory ◽

Multivariate Time Series ◽

Temporal Patterns ◽

Series Data ◽

Short Term ◽

Trend Prediction ◽

Multiple Variables ◽

Parameter Sharing

The problem of learning and forecasting underlying trends in time series data arises in a variety of applications, such as traffic management, energy optimization, etc. In literature, a trend in time series is characterized by the slope and duration, and its prediction is then to forecast the two values of the subsequent trend given historical data of the time series. For this problem, existing approaches mainly deal with the case in univariate time series. However, in many real-world applications, there are multiple variables at play, and handling all of them at the same time is crucial for an accurate prediction. A natural way is to employ multi-task learning (MTL) techniques in which the trend learning of each time series is treated as a task. The key point of MTL is to learn task relatedness to achieve better parameter sharing, which however is challenging in trend prediction task. First, effectively modeling the complex temporal patterns in different tasks is hard as the temporal and spatial dimensions are entangled. Second, the relatedness among tasks may change over time. In this paper, we propose a neural network, DeepTrends, for multivariate time series trend prediction. The core module of DeepTrends is a tensorized LSTM with adaptive shared memory (TLASM). TLASM employs the tensorized LSTM to model the temporal patterns of long-term trend sequences in an MTL setting. With an adaptive shared memory, TLASM is able to learn the relatedness among tasks adaptively, based upon which it can dynamically vary degrees of parameter sharing among tasks. To further consider short-term patterns, DeepTrends utilizes a multi-task 1dCNN to learn the local time series features, and employs a task-specific sub-network to learn a mixture of long-term and short-term patterns for trend prediction. Extensive experiments on real datasets demonstrate the effectiveness of the proposed model.

Download Full-text

VS-GRU: A Variable Sensitive Gated Recurrent Neural Network for Multivariate Time Series with Massive Missing Values

Applied Sciences ◽

10.3390/app9153041 ◽

2019 ◽

Vol 9 (15) ◽

pp. 3041 ◽

Cited By ~ 5

Author(s):

Qianting Li ◽

Yong Xu

Keyword(s):

Neural Network ◽

Time Series ◽

Recurrent Neural Network ◽

Real World ◽

Missing Values ◽

State Of The Art ◽

Multivariate Time Series ◽

Training Procedure ◽

Model Training ◽

Mimic Iii

Multivariate time series are often accompanied with missing values, especially in clinical time series, which usually contain more than 80% of missing data, and the missing rates between different variables vary widely. However, few studies address these missing rate differences and extract univariate missing patterns simultaneously before mixing them in the model training procedure. In this paper, we propose a novel recurrent neural network called variable sensitive GRU (VS-GRU), which utilizes the different missing rate of each variable as another input and learns the feature of different variables separately, reducing the harmful impact of variables with high missing rates. Experiments show that VS-GRU outperforms the state-of-the-art method in two real-world clinical datasets (MIMIC-III, PhysioNet).

Download Full-text