Absolute and relative measures for evaluating the forecasting performance of time series models for daily streamflows

2006 ◽  
Vol 37 (3) ◽  
pp. 205-215 ◽  
Author(s):  
T. Astatkie
Keyword(s):  
Time Series ◽  
Mean Squared Error ◽  
Model Performance ◽  
Absolute Error ◽  
Time Series Models ◽  
Streamflow Forecasting ◽  
Daily Streamflow ◽  
Forecasting Performance ◽  
Competing Models

Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) are widely used measures for evaluating the forecasting performance of time series models. Although these absolute measures can be used to compare the performance of competing models, one needs a reference to judge the goodness of the forecasts. In this paper, two relative measures, coefficient of efficiency (E) and index of agreement (d), and their modified versions (EM, EMP, dM and dMP) with desired values of closer to one are presented. These measures are illustrated by comparing the modeling ability and validation forecasting performance of a Nonlinear Additive Autoregressive with Exogenous variables (NAARX), Nested Threshold Autoregressive (NeTAR), and Multiple Nonlinear Inputs Transfer Function (MNITF) models developed for the Jökulsá eystri daily streamflow data. The results suggest that NeTAR describes the system best, and gives better 1- and 2-day ahead validation forecasts. MNITF gives better forecasts for 3-day ahead, and NeTAR and NAARX give comparable performance for 4- and 5-day ahead forecasting. The values of E and d were larger than those of the modified versions, giving a false sense of model performance, and unlike the modified versions, they decreased as forecast lead times increased. Differences among the values of these six relative measures can reveal the sensitiveness of competing models to outliers, and their potential for long-term forecasting. Accordingly, NeTAR was the least sensitive to outliers and NAARX was the most sensitive, with MNITF in between; and NAARX showed the most potential for long-term streamflow forecasting.

scholarly journals Evaluation of Random Forest for short-term daily streamflow forecast in rainfall and snowmelt driven watersheds

2020 ◽  
Author(s):  
Leo T. Pham ◽  
Lifeng Luo ◽  
Andrew O. Finley
Keyword(s):  
Random Forest ◽  
Mean Squared Error ◽  
Absolute Error ◽  
Climatic Conditions ◽  
Coefficient Of Determination ◽  
Streamflow Forecasting ◽  
Short Term ◽  
Streamflow Forecast ◽  
Daily Streamflow ◽  
Rf Performance

Abstract. In the past decades, data-driven Machine Learning (ML) models have emerged as promising tools for short-term streamflow forecasts. Among other qualities, the popularity of ML for such applications is due to the methods' competitive performance compared with alternative approaches, ease of application, and relative lack of strict distributional assumptions. Despite the encouraging results, most applications of ML for streamflow forecast have been limited to watersheds where rainfall is the major source of runoff. In this study, we evaluate the potential of Random Forest (RF), a popular ML method, to make streamflow forecast at 1-day lead time at 86 watersheds in the Pacific Northwest. These watersheds span climatic conditions and physiographic settings and exhibit varied contributions of rainfall and snowmelt to their streamflow. Watersheds are classified into three hydrologic regimes: rainfall-dominated, transisent, and snowmelt-dominated based on the timing of center of annual flow volume. RF performance is benchmarked against Naive and multiple linear regression (MLR) models, and evaluated using four metrics Coefficient of determination, Root mean squared error, Mean absolute error, and Kling-Gupta efficiency. Model evaluation metrics suggest RF performs better in snowmelt-driven watersheds. Largest improvement in forecasts, compared to benchmark models, are found among rainfall-driven watersheds. We obtain Kling–Gupta Efficiency (KGE) scores in the range of 0.62–0.99. RF performance deteriorates with increase in catchment slope and increase in soil sandiness. We note disagreement between two popular measures of RF variable importance and recommend jointly considering these measures with the physical processes under study. These and other results presented provide new insights for effective application of RF-based streamflow forecasting.

Application of Fuzzy Time Series Techniques in Wind and Wave Data Forecasting

2014 ◽  
Author(s):  
Christos N. Stefanakos ◽  
Orestis Schinas ◽  
Grim Eidnes
Keyword(s):  
Time Series ◽  
Stock Prices ◽  
Mean Squared Error ◽  
Offshore Structures ◽  
Fuzzy Time Series ◽  
Great Success ◽  
Peak Period ◽  
Wave Data ◽  
Forecasting Performance

This work explores the applicability of widely known fuzzy time series forecasting techniques for the prediction of wind and wave data. These techniques have extensively been used with great success to the forecasting of stock prices. In the present work, long-term time series of wind speed, significant wave height, and peak period are examined and used for the verification of the forecasting performance of the fuzzy models. To examine the forecasting accuracy, the root mean squared error (RMSE) is used as an evaluation criterion to compare the forecasting performance of the listing models. As the importance of quality of wind and wave data increases, effective forecasting could further benefit designers of offshore structures and environmental researchers.

scholarly journals Statistical modelling of agrometeorological time series by exponential smoothing

2016 ◽  
Vol 30 (1) ◽  
pp. 57-65 ◽  
Author(s):  
Małgorzata Murat ◽  
Iwona Malinowska ◽  
Holger Hoffmann ◽  
Piotr Baranowski
Keyword(s):  
Time Series ◽  
Mean Squared Error ◽  
Statistical Modelling ◽  
Absolute Error ◽  
Exponential Smoothing ◽  
Growth And Yield ◽  
Climate Change Scenarios ◽  
Autocorrelation Functions ◽  
Using Data

Abstract Meteorological time series are used in modelling agrophysical processes of the soil-plant-atmosphere system which determine plant growth and yield. Additionally, long-term meteorological series are used in climate change scenarios. Such studies often require forecasting or projection of meteorological variables, eg the projection of occurrence of the extreme events. The aim of the article was to determine the most suitable exponential smoothing models to generate forecast using data on air temperature, wind speed, and precipitation time series in Jokioinen (Finland), Dikopshof (Germany), Lleida (Spain), and Lublin (Poland). These series exhibit regular additive seasonality or non-seasonality without any trend, which is confirmed by their autocorrelation functions and partial autocorrelation functions. The most suitable models were indicated by the smallest mean absolute error and the smallest root mean squared error.

scholarly journals Evaluation of random forests for short-term daily streamflow forecasting in rainfall- and snowmelt-driven watersheds

2021 ◽  
Vol 25 (6) ◽  
pp. 2997-3015
Author(s):  
Leo Triet Pham ◽  
Lifeng Luo ◽  
Andrew Finley
Keyword(s):  
Random Forests ◽  
Mean Squared Error ◽  
Predictive Performance ◽  
Absolute Error ◽  
Climatic Conditions ◽  
Coefficient Of Determination ◽  
Streamflow Forecasting ◽  
Short Term ◽  
Daily Streamflow ◽  
Rf Performance

Abstract. In the past decades, data-driven machine-learning (ML) models have emerged as promising tools for short-term streamflow forecasting. Among other qualities, the popularity of ML models for such applications is due to their relative ease in implementation, less strict distributional assumption, and competitive computational and predictive performance. Despite the encouraging results, most applications of ML for streamflow forecasting have been limited to watersheds in which rainfall is the major source of runoff. In this study, we evaluate the potential of random forests (RFs), a popular ML method, to make streamflow forecasts at 1 d of lead time at 86 watersheds in the Pacific Northwest. These watersheds cover diverse climatic conditions and physiographic settings and exhibit varied contributions of rainfall and snowmelt to their streamflow. Watersheds are classified into three hydrologic regimes based on the timing of center-of-annual flow volume: rainfall-dominated, transient, and snowmelt-dominated. RF performance is benchmarked against naïve and multiple linear regression (MLR) models and evaluated using four criteria: coefficient of determination, root mean squared error, mean absolute error, and Kling–Gupta efficiency (KGE). Model evaluation scores suggest that the RF performs better in snowmelt-driven watersheds compared to rainfall-driven watersheds. The largest improvements in forecasts compared to benchmark models are found among rainfall-driven watersheds. RF performance deteriorates with increases in catchment slope and soil sandiness. We note disagreement between two popular measures of RF variable importance and recommend jointly considering these measures with the physical processes under study. These and other results presented provide new insights for effective application of RF-based streamflow forecasting.

scholarly journals Daily flow forecasting of perennial rivers in an arid watershed: a hybrid ensemble decomposition approach integrated with computational intelligence techniques

2020 ◽  
Vol 69 (6) ◽  
pp. 555-577
Author(s):  
Seyed Alireza Torabi ◽  
Reza Mastouri ◽  
Mohsen Najarchi
Keyword(s):  
Mean Squared Error ◽  
Search Algorithm ◽  
Absolute Error ◽  
Ensemble Empirical Mode Decomposition ◽  
Alternative Methods ◽  
Streamflow Forecasting ◽  
Decomposition Approach ◽  
Integrated Method ◽  
Daily Streamflow ◽  
Mode Decomposition

Abstract Accurate estimating of daily streamflow forecasting is one of the prominent topics in water resources activities. In this paper, an integrated method including decomposition technique based on the ensemble empirical mode decomposition (EEMD) combined with multivariate adaptive regression spline (MARS) was carried out to predict daily streamflow values. Daily streamflow value datasets collected from two stations in Iran (Gachsar and Kordkheyl) were selected. After dividing into calibration and validation datasets, each of them was decomposed by EEMD. Crow search algorithm (CSA) was used to optimize the MARS parameters (MARS-CSA). The performance of the integrated model (EEMD-MARS-CSA) was investigated by error indices (correlation coefficient (R), root mean squared error (RMSE), mean absolute error (MAE), Nash–Sutcliffe efficiency (NSE), as well as RMSE to standard deviation ratio (RSR)). From the results, EEMD was an important tool for increasing model accuracy and EEMD-MARS-CSA outperformed other alternative methods for daily streamflow estimation. According to one-day-ahead flow forecasting, EEMD-MARS-CSA (R = 0.94, RMSE = 5.94 m3/s (Kordkheyl) and R = 0.98, RMSE = 0.71 m3/s (Gachsar)) outperformed EEMD-MT/MARS, MT, and MARS models. Furthermore, RSR criterion of EEMD-MARS-CSA was reduced by 18%, 16%, and 17% for 3-days, 1-week, and 2-weeks-ahead streamflow forecasting compared to MARS-CSA model, respectively, for Gachsar station.

scholarly journals Explaining Bad Forecasts in Global Time Series Models

2021 ◽  
Vol 11 (19) ◽  
pp. 9243
Author(s):  
Jože Rožanec ◽  
Elena Trajkova ◽  
Klemen Kenda ◽  
Blaž Fortuna ◽  
Dunja Mladenić
Keyword(s):  
Time Series ◽  
Model Performance ◽  
Time Series Forecasting ◽  
Modular Architecture ◽  
Detection Algorithms ◽  
Training Samples ◽  
Forecasting Performance ◽  
Global Time ◽  
Real World Datasets ◽  
Value Changes

While increasing empirical evidence suggests that global time series forecasting models can achieve better forecasting performance than local ones, there is a research void regarding when and why the global models fail to provide a good forecast. This paper uses anomaly detection algorithms and explainable artificial intelligence (XAI) to answer when and why a forecast should not be trusted. To address this issue, a dashboard was built to inform the user regarding (i) the relevance of the features for that particular forecast, (ii) which training samples most likely influenced the forecast outcome, (iii) why the forecast is considered an outlier, and (iv) provide a range of counterfactual examples to understand how value changes in the feature vector can lead to a different outcome. Moreover, a modular architecture and a methodology were developed to iteratively remove noisy data instances from the train set, to enhance the overall global time series forecasting model performance. Finally, to test the effectiveness of the proposed approach, it was validated on two publicly available real-world datasets.

scholarly journals Investigation and Optimization of the C-ANN Structure in Predicting the Compressive Strength of Foamed Concrete

Materials ◽  
2020 ◽  
Vol 13 (5) ◽  
pp. 1072 ◽  
Author(s):  
Dong Van Dao ◽  
Hai-Bang Ly ◽  
Huong-Lan Thi Vu ◽  
Tien-Thinh Le ◽  
Binh Thai Pham
Keyword(s):  
Compressive Strength ◽  
Civil Engineering ◽  
Mean Squared Error ◽  
Model Performance ◽  
Absolute Error ◽  
Machine Learning Algorithms ◽  
Coefficient Of Determination ◽  
Dry Density ◽  
Statistical Measures ◽  
Foamed Concrete

Development of Foamed Concrete (FC) and incessant increases in fabrication technology have paved the way for many promising civil engineering applications. Nevertheless, the design of FC requires a large number of experiments to determine the appropriate Compressive Strength (CS). Employment of machine learning algorithms to take advantage of the existing experiments database has been attempted, but model performance can still be improved. In this study, the performance of an Artificial Neural Network (ANN) was fully analyzed to predict the 28 days CS of FC. Monte Carlo simulations (MCS) were used to statistically analyze the convergence of the modeled results under the effect of random sampling strategies and the network structures selected. Various statistical measures such as Coefficient of Determination (R2), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE) were used for validation of model performance. The results show that ANN is a highly efficient predictor of the CS of FC, achieving a maximum R2 value of 0.976 on the training part and an R2 of 0.972 on the testing part, using the optimized C-ANN-[3–4–5–1] structure, which compares with previous published studies. In addition, a sensitivity analysis using Partial Dependence Plots (PDP) over 1000 MCS was also performed to interpret the relationship between the input parameters and 28 days CS of FC. Dry density was found as the variable with the highest impact to predict the CS of FC. The results presented could facilitate and enhance the use of C-ANN in other civil engineering-related problems.

scholarly journals Novel FTLRNN with Gamma Memory for Short-Term and Long-Term Predictions of Chaotic Time Series

2009 ◽  
Vol 2009 ◽  
pp. 1-21
Author(s):  
Sanjay L. Badjate ◽  
Sanjay V. Dudul
Keyword(s):  
Neural Network ◽  
Time Series ◽  
Mean Squared Error ◽  
Chaotic Time Series ◽  
Nonstationary Time Series ◽  
Medium Term ◽  
Short Term ◽  
Squared Error ◽  
Wide Range

Multistep ahead prediction of a chaotic time series is a difficult task that has attracted increasing interest in the recent years. The interest in this work is the development of nonlinear neural network models for the purpose of building multistep chaotic time series prediction. In the literature there is a wide range of different approaches but their success depends on the predicting performance of the individual methods. Also the most popular neural models are based on the statistical and traditional feed forward neural networks. But it is seen that this kind of neural model may present some disadvantages when long-term prediction is required. In this paper focused time-lagged recurrent neural network (FTLRNN) model with gamma memory is developed for different prediction horizons. It is observed that this predictor performs remarkably well for short-term predictions as well as medium-term predictions. For coupled partial differential equations generated chaotic time series such as Mackey Glass and Duffing, FTLRNN-based predictor performs consistently well for different depths of predictions ranging from short term to long term, with only slight deterioration after k is increased beyond 50. For real-world highly complex and nonstationary time series like Sunspots and Laser, though the proposed predictor does perform reasonably for short term and medium-term predictions, its prediction ability drops for long term ahead prediction. However, still this is the best possible prediction results considering the facts that these are nonstationary time series. As a matter of fact, no other NN configuration can match the performance of FTLRNN model. The authors experimented the performance of this FTLRNN model on predicting the dynamic behavior of typical Chaotic Mackey-Glass time series, Duffing time series, and two real-time chaotic time series such as monthly sunspots and laser. Static multi layer perceptron (MLP) model is also attempted and compared against the proposed model on the performance measures like mean squared error (MSE), Normalized mean squared error (NMSE), and Correlation Coefficient (r). The standard back-propagation algorithm with momentum term has been used for both the models.

Sign in / Sign up

Export Citation Format

Share Document