scholarly journals Explaining Bad Forecasts in Global Time Series Models

2021 ◽  
Vol 11 (19) ◽  
pp. 9243
Author(s):  
Jože Rožanec ◽  
Elena Trajkova ◽  
Klemen Kenda ◽  
Blaž Fortuna ◽  
Dunja Mladenić
Keyword(s):  
Time Series ◽  
Model Performance ◽  
Time Series Forecasting ◽  
Modular Architecture ◽  
Detection Algorithms ◽  
Training Samples ◽  
Forecasting Performance ◽  
Global Time ◽  
Real World Datasets ◽  
Value Changes

While increasing empirical evidence suggests that global time series forecasting models can achieve better forecasting performance than local ones, there is a research void regarding when and why the global models fail to provide a good forecast. This paper uses anomaly detection algorithms and explainable artificial intelligence (XAI) to answer when and why a forecast should not be trusted. To address this issue, a dashboard was built to inform the user regarding (i) the relevance of the features for that particular forecast, (ii) which training samples most likely influenced the forecast outcome, (iii) why the forecast is considered an outlier, and (iv) provide a range of counterfactual examples to understand how value changes in the feature vector can lead to a different outcome. Moreover, a modular architecture and a methodology were developed to iteratively remove noisy data instances from the train set, to enhance the overall global time series forecasting model performance. Finally, to test the effectiveness of the proposed approach, it was validated on two publicly available real-world datasets.

scholarly journals Explaining Bad Forecasts in Global Time Series Models

2021 ◽  
Author(s):  
Jože Rožanec ◽  
Elena Trajkova ◽  
Klemen Kenda ◽  
Blaž Fortuna ◽  
Dunja Mladenić
Keyword(s):  
Time Series ◽  
Model Performance ◽  
Time Series Forecasting ◽  
Modular Architecture ◽  
Detection Algorithms ◽  
Training Samples ◽  
Forecasting Performance ◽  
Global Time ◽  
Real World Datasets ◽  
Value Changes

While increasing empirical evidence suggests that global time series forecasting models can achieve better forecasting performance than local ones, there is a research void regarding when and why the global models fail to provide a good forecast. This paper uses anomaly detection algorithms and Explainable Artificial Intelligence (XAI) to answer when and why a forecast should not be trusted. To address this issue, a dashboard was built to inform the user regarding (i) the relevance of the features for that particular forecast, (ii) which training samples most likely influenced the forecast outcome, (iii) why the forecast is considered an outlier, and (iv) provide a range of counterfactual examples to understand value changes, in the feature vector or the predicted value, can lead to a different outcome. Moreover, a modular architecture and a methodology were developed to iteratively remove noisy data instances from the train set, to enhance the overall global time series forecasting model performance. Finally, to test the effectiveness of the proposed approach, it was validated on two publicly available real-world datasets.

scholarly journals Cogra: Concept-Drift-Aware Stochastic Gradient Descent for Time-Series Forecasting

2019 ◽  
Vol 33 ◽  
pp. 4594-4601 ◽  
Author(s):  
Kohei Miyaguchi ◽  
Hiroshi Kajino
Keyword(s):  
Time Series ◽  
Gradient Descent ◽  
Concept Drift ◽  
Time Series Forecasting ◽  
Learning Rate ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Automatic Learning ◽  
The Mean ◽  
Real World Datasets

We approach the time-series forecasting problem in the presence of concept drift by automatic learning rate tuning of stochastic gradient descent (SGD). The SGD-based approach is preferable to other concept drift algorithms in that it can be applied to any model and it can keep learning efficiently whilst predicting online. Among a number of SGD algorithms, the variance-based SGD (vSGD) can successfully handle concept drift by automatic learning rate tuning, which is reduced to an adaptive mean estimation problem. However, its performance is still limited because of its heuristic mean estimator. In this paper, we present a concept-drift-aware stochastic gradient descent (Cogra), equipped with more theoretically-sound mean estimator called sequential mean tracker (SMT). Our key contribution is that we define a goodness criterion for the mean estimators; SMT is designed to be optimal according to this criterion. As a result of comprehensive experiments, we find that (i) our SMT can estimate the mean better than vSGD’s estimator in the presence of concept drift, and (ii) in terms of predictive performance, Cogra reduces the predictive loss by 16–67% for real-world datasets, indicating that SMT improves the prediction accuracy significantly.

The Effects of Predictor Variables and Interval Partition on Fuzzy Time Series Forecasting

2011 ◽  
Vol 145 ◽  
pp. 143-148
Author(s):  
Hsien Lun Wong ◽  
Chi Chen Wang ◽  
Tsung Yi Shen
Keyword(s):  
Time Series ◽  
Fuzzy Model ◽  
Predictive Performance ◽  
Dynamic Parameter ◽  
Time Series Forecasting ◽  
Fuzzy Time Series ◽  
Predictor Variables ◽  
Fuzzy Models ◽  
Forecasting Error ◽  
Forecasting Performance

Fuzzy time series methods have been applied to social forecasting for over a decade; however, little research has been done to discuss the decision of an optimal fuzzy model for time series. In the paper, we evaluate the forecasting performance of three listed multivariate fuzzy models by comparing forecasting MSE of model. The data obtained from AEROM, Taiwan, includes Taiwan’s exports and foreign exchange rate for models’ test. The algorithm for predictive value of the models has three-stage computation procedure: First, calibrating time series correlation, deciding window base and interval partition; second, solving the static forecasting value of each model; third, comparing the dynamic parameter to impact of the forecasting error. The empirical results indicate that increasing predictor variables has no significant effect on predictive performance of the models; increasing length of interval would not improve the prediction performance of the models. Moreover, Fuzzy model is better for short-term time series forecasting. For forecasting purpose, Heuristic model has best forecasting performance among three fuzzy models. The findings of the paper represent a significant contribution to our understanding of the applicability of fuzzy models to predict.

scholarly journals The Wisdom of the Data: Getting the Most Out of Univariate Time Series Forecasting

Forecasting ◽  
2021 ◽  
Vol 3 (3) ◽  
pp. 478-497
Author(s):  
Fotios Petropoulos ◽  
Evangelos Spiliotis
Keyword(s):  
Time Series ◽  
Computational Cost ◽  
Forecast Accuracy ◽  
Time Series Forecasting ◽  
Data Manipulation ◽  
Combining Forecasts ◽  
Univariate Time Series ◽  
Forecasting Performance ◽  
Wisdom Of The Crowds ◽  
Theta Method

Forecasting is a challenging task that typically requires making assumptions about the observed data but also the future conditions. Inevitably, any forecasting process will result in some degree of inaccuracy. The forecasting performance will further deteriorate as the uncertainty increases. In this article, we focus on univariate time series forecasting and we review five approaches that one can use to enhance the performance of standard extrapolation methods. Much has been written about the “wisdom of the crowds” and how collective opinions will outperform individual ones. We present the concept of the “wisdom of the data” and how data manipulation can result in information extraction which, in turn, translates to improved forecast accuracy by aggregating (combining) forecasts computed on different perspectives of the same data. We describe and discuss approaches that are based on the manipulation of local curvatures (theta method), temporal aggregation, bootstrapping, sub-seasonal and incomplete time series. We compare these approaches with regards to how they extract information from the data, their computational cost, and their performance.

Absolute and relative measures for evaluating the forecasting performance of time series models for daily streamflows

2006 ◽  
Vol 37 (3) ◽  
pp. 205-215 ◽  
Author(s):  
T. Astatkie
Keyword(s):  
Time Series ◽  
Mean Squared Error ◽  
Model Performance ◽  
Absolute Error ◽  
Time Series Models ◽  
Streamflow Forecasting ◽  
Daily Streamflow ◽  
Forecasting Performance ◽  
Competing Models

Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) are widely used measures for evaluating the forecasting performance of time series models. Although these absolute measures can be used to compare the performance of competing models, one needs a reference to judge the goodness of the forecasts. In this paper, two relative measures, coefficient of efficiency (E) and index of agreement (d), and their modified versions (EM, EMP, dM and dMP) with desired values of closer to one are presented. These measures are illustrated by comparing the modeling ability and validation forecasting performance of a Nonlinear Additive Autoregressive with Exogenous variables (NAARX), Nested Threshold Autoregressive (NeTAR), and Multiple Nonlinear Inputs Transfer Function (MNITF) models developed for the Jökulsá eystri daily streamflow data. The results suggest that NeTAR describes the system best, and gives better 1- and 2-day ahead validation forecasts. MNITF gives better forecasts for 3-day ahead, and NeTAR and NAARX give comparable performance for 4- and 5-day ahead forecasting. The values of E and d were larger than those of the modified versions, giving a false sense of model performance, and unlike the modified versions, they decreased as forecast lead times increased. Differences among the values of these six relative measures can reveal the sensitiveness of competing models to outliers, and their potential for long-term forecasting. Accordingly, NeTAR was the least sensitive to outliers and NAARX was the most sensitive, with MNITF in between; and NAARX showed the most potential for long-term streamflow forecasting.

A Weighted Window Approach to Neural Network Time Series Forecasting

2011 ◽  
pp. 251-265 ◽  
Author(s):  
Bradley H. Morantz ◽  
Thomas Whalen ◽  
G. Peter Zhang
Keyword(s):  
Neural Network ◽  
Time Series ◽  
Time Series Forecasting ◽  
Forecast Errors ◽  
Data Sets ◽  
Economic Data ◽  
Network Time ◽  
Forecasting Performance ◽  
Window Approach ◽  
Traditional Approaches

In this chapter, we propose a neural network based weighted window approach to time series forecasting. We compare the weighted window approach with two commonly used methods of rolling and moving windows in modeling time series. Seven economic data sets are used to compare the performance of these three data windowing methods on observed forecast errors. We find that the proposed approach can improve forecasting performance over traditional approaches.

Forecasting performance of support vector machine for the Poyang Lake's water level

2014 ◽  
Vol 70 (9) ◽  
pp. 1488-1495 ◽  
Author(s):  
Yingying Lan
Keyword(s):  
Time Series ◽  
Support Vector Machine ◽  
Model Performance ◽  
Optimal Solution ◽  
Kernel Functions ◽  
Support Vector ◽  
Hydrologic Time Series ◽  
Forecasting Performance ◽  
Linear Svm

The growth of forecasting models has resulted in the development of an excellent model known as the support vector machine (SVM). SVMs can find a global optimal solution equipped with kernel functions. This research trains and tests the SVM network and constructs the support vector regression prediction model by using hydrologic data. Six hydrologic time series were calculated by different kernel functions (namely, linear, polynomial, radial basis function (RBF)), to determine which kernel is the more suitable hydrologic time series in practice. A new solution is presented to identify the good parameter (C; g) by using grid-search and cross-validation. Results prove that linear SVM is a superior model to polynomial and RBF and produced the most accurate results for modeling hydrologic time series behavior as complex hydrologic phenomena. The case study also shows that the calculation errors were correlated with data characteristics. More stable raw data will result in a more accurate result, whereas more random data will result in a more inaccurate result. Model performance could also be dependent on base data nonlinearity.

A new weighted MSE loss for wind speed forecasting based on deep learning models

2020 ◽  
Author(s):  
Xi Chen ◽  
Ruyi Yu ◽  
Sajid Ullah ◽  
Dianming Wu ◽  
Min Liu ◽  
...  
Keyword(s):  
Time Series ◽  
Deep Learning ◽  
Wind Speed ◽  
Loss Function ◽  
Time Series Data ◽  
Series Data ◽  
Learning Models ◽  
Wind Speed Forecasting ◽  
Training Samples ◽  
Forecasting Performance

<p>Wind speed forecasting is very important for a lot of real-life applications, especially for controlling and monitoring of wind power plants. Owing to the non-linearity of wind speed time series, it is hard to improve the accuracy of runoff forecasting, especially several days ahead. In order to improve the forecasting performance, many forecasting models have been proposed. Recently, deep learning models have been paid great attention, since they excel the conventional machine learning models. The majority of existing deep learning models take the mean squared error (MSE) loss as the loss function for forecasting. MSE loss is linear. Consequently, it hinders further improvement of forecasting performance over nonlinear wind speed time series data.   <br> <br>In this work, we propose a new weighted MSE loss function for wind speed forecasting based on deep learning. As is well known, the training procedure is dominated by easy-training samples in applications. The domination will cause the ineffectiveness and inefficiency of computation. In the new weighted MSE loss function, loss weights of samples can be automatically reduced, according to the contribution of easy-training samples. Thus, the total loss mainly focuses on hard-training samples. To verify the new loss function, Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have been used as base models. <br> <br>A number of experiments have been carried out by using open wind speed time series data collected from China and Unites states to demonstrate the effectiveness of the new loss function with three popular models. The performances of the models have been evaluated through the statistical error measures, such as Mean Absolute Error (MAE). MAE of the proposed weighted MSE loss are at most 55% lower than traditional MSE loss. The experimental results indicate that the new weighted loss function can outperform the popular MSE loss function in wind speed forecasting. </p>

scholarly journals EVALUATION OF COMPUTATIONAL INTELLIGENCE TECHNIQUES FOR DAILY PRODUCT SALES FORECASTING

2015 ◽  
pp. 157-164
Author(s):  
Gediminas Gediminas Žylius ◽  
Rimvydas Simutis ◽  
Vygandas Vaitkus
Keyword(s):  
Time Series ◽  
Computational Intelligence ◽  
Real Life ◽  
Time Series Forecasting ◽  
Support Vector ◽  
Historical Period ◽  
Support Vector Regression Model ◽  
Forecasting Performance ◽  
Intelligence Models ◽  
Product Sales

Product sales forecasting is crucial task in inventory control and whole supply chain management. Accuracy of sales forecasting determines product logistics performance. In this paper we present study that aims to answer three questions: what input set is most informative for daily sales time series forecasting; do weather input features improve forecasting performance; what computational intelligence model is most appropriate for daily sales forecasting. In order to answer those questions we selected three computational intelligence models that are used for regression task together with various input sets for daily time series forecasting. Data collected consist of 89 real life product sales time series from various stores with historical period of 15 months. Results show that most useful input set is extracted from time series itself. Secondly, research results show that weather features do not improve forecasting performance. And finally, best forecasting results are achieved using support vector regression model.

A Combined ARIMA and Neural Network Approach for Time Series Forecasting

2011 ◽  
pp. 213-225 ◽  
Author(s):  
G. Peter Zhang
Keyword(s):  
Neural Network ◽  
Time Series ◽  
Neural Network Model ◽  
Arima Model ◽  
Time Series Forecasting ◽  
Network Approach ◽  
Neural Network Approach ◽  
Individual Model ◽  
The Neural Network ◽  
Forecasting Performance

This chapter presents a combined ARIMA and neural network approach for time series forecasting. The model contains three steps: (1) fitting a linear ARIMA model to the time series under study, (2) building a neural network model based on the residuals from the ARIMA model, and (3) combine the ARIMA prediction and the neural network result to form the final forecast. By combining different models, we aim to take advantage of the unique modeling capability of each individual model and improve forecasting performance dramatically. The effectiveness of the combining approach is demonstrated and discussed with three applications.

Sign in / Sign up

Export Citation Format

Share Document