On the use of cross-validation for time series predictor evaluation

Forecasting reservoir water levels is essential in water supply management, impacting both operations and intervention strategies. This paper examines the short-term and long-term forecasting performance of several statistical and machine learning-based methods for predicting the water levels of the Angat Dam in the Philippines. A total of six forecasting methods are compared: naïve/persistence; seasonal mean; autoregressive integrated moving average (ARIMA); gradient boosting machines (GBM); and two deep neural networks (DNN) using a long short-term memory-based (LSTM) encoder-decoder architecture: a univariate model (DNN-U) and a multivariate model (DNN-M). Daily historical water levels from 2001 to 2021 are used in predicting future water levels. In addition, we include meteorological data (rainfall and the Oceanic Niño Index) and irrigation data as exogenous variables. To evaluate the forecast accuracy of our methods, we use a time series cross-validation approach to establish a more robust estimate of the error statistics. Our results show that our DNN-U model has the best accuracy in the 1-day-ahead scenario with a mean absolute error (MAE) and root mean square error (RMSE) of 0.2 m. In the 30-day-, 90-day-, and 180-day-ahead scenarios, the DNN-M shows the best performance with MAE (RMSE) scores of 2.9 (3.3), 5.1 (6.0), and 6.7 (8.1) meters, respectively. Additionally, we demonstrate that further improvements in performance are possible by scanning over all possible combinations of the exogenous variables and only using a subset of them as features. In summary, we provide a comprehensive framework for evaluating water level forecasting by defining a baseline accuracy, analyzing performance across multiple prediction horizons, using time series cross-validation to assess accuracy and uncertainty, and examining the effects of exogenous variables on forecasting performance. In the process, our work addresses several notable gaps in the methodologies of previous works.

Download Full-text

SUBSIDENCE EVALUATION OF HIGH-SPEED RAILWAY IN SHENYANG BASED ON TIME-SERIES INSAR

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-iv-3-273-2018 ◽

2018 ◽

Vol IV-3 ◽

pp. 273-276

Author(s):

Yun Zhang ◽

Lianhuan Wei ◽

Jiayu Li ◽

Shanjun Liu ◽

Yachun Mao ◽

...

Keyword(s):

Time Series ◽

High Speed ◽

Cross Validation ◽

Surface Deformation ◽

Local Environment ◽

Local Area ◽

Safe Operation ◽

High Speed Rail ◽

High Speed Railway ◽

Under Construction

More and more high-speed railway are under construction in China. The slow settlement along high-speed railway tracks and newly-built stations would lead to inhomogeneous deformation of local area, and the accumulation may be a threat to the safe operation of high-speed rail system. In this paper, surface deformation of the newly-built high-speed railway station as well as the railway lines in Shenyang region will be retrieved by time series InSAR analysis using multi-orbit COSMO-SkyMed images. This paper focuses on the non-uniform subsidence caused by the changing of local environment along the railway. The accuracy of the settlement results can be verified by cross validation of the results obtained from two different orbits during the same period.

Download Full-text

A SMOOTHING ALGORITHM FOR NONLINEAR TIME SERIES

International Journal of Bifurcation and Chaos ◽

10.1142/s021812740400965x ◽

2004 ◽

Vol 14 (03) ◽

pp. 1037-1051 ◽

Cited By ~ 6

Author(s):

S. A. BILLINGS ◽

K. L. LEE

Keyword(s):

Time Series ◽

Noise Reduction ◽

Cross Validation ◽

User Interaction ◽

Nonlinear Time Series ◽

Smoothing Algorithm ◽

Wavelet Thresholding ◽

Numerical Examples ◽

Validation Method

A new NARMA based smoothing algorithm is introduced for chaotic and nonchaotic time series. The new algorithm employs a cross-validation method to determine the smoother structure, requires very little user interaction, and can be combined with wavelet thresholding to further enhance the noise reduction. Numerical examples are included to illustrate the application of the new algorithm.

Download Full-text

High Order Lambda Measure Based Choquet Integral Composition Forecasting Model

Applied Mechanics and Materials ◽

10.4028/www.scientific.net/amm.284-287.3111 ◽

2013 ◽

Vol 284-287 ◽

pp. 3111-3114

Author(s):

Hsiang Chuan Liu ◽

Wei Sung Chen ◽

Ben Chang Shia ◽

Chia Chen Lee ◽

Shang Ling Ou ◽

...

Keyword(s):

Time Series ◽

Linear Regression ◽

Cross Validation ◽

Choquet Integral ◽

Real Data ◽

High Order ◽

Forecasting Model ◽

Mean Square ◽

Weighted Composition ◽

Fold Cross Validation

In this paper, a novel fuzzy measure, high order lambda measure, was proposed, based on the Choquet integral with respect to this new measure, a novel composition forecasting model which composed the GM(1,1) forecasting model, the time series model and the exponential smoothing model was also proposed. For evaluating the efficiency of this improved composition forecasting model, an experiment with a real data by using the 5 fold cross validation mean square error was conducted. The performances of Choquet integral composition forecasting model with the P-measure, Lambda-measure, L-measure and high order lambda measure, respectively, a ridge regression composition forecasting model and a multiple linear regression composition forecasting model and the traditional linear weighted composition forecasting model were compared. The experimental results showed that the Choquet integral composition forecasting model with respect to the high order lambda measure has the best performance.

Download Full-text

Statistical Stability and Spatial Instability in Mapping Forest Tree Species by Comparing 9 Years of Satellite Image Time Series

Remote Sensing ◽

10.3390/rs11212512 ◽

2019 ◽

Vol 11 (21) ◽

pp. 2512 ◽

Cited By ~ 7

Author(s):

Nicolas Karasiak ◽

Jean-François Dejoux ◽

Mathieu Fauvel ◽

Jérôme Willm ◽

Claude Monteil ◽

...

Keyword(s):

Time Series ◽

Tree Species ◽

Cross Validation ◽

Satellite Image ◽

Support Vector ◽

Validation Data ◽

Spatial Instability ◽

Classification Errors ◽

Leave One Out ◽

The Impact

Mapping forest composition using multiseasonal optical time series remains a challenge. Highly contrasted results are reported from one study to another suggesting that drivers of classification errors are still under-explored. We evaluated the performances of single-year Formosat-2 time series to discriminate tree species in temperate forests in France and investigated how predictions vary statistically and spatially across multiple years. Our objective was to better estimate the impact of spatial autocorrelation in the validation data on measurement accuracy and to understand which drivers in the time series are responsible for classification errors. The experiments were based on 10 Formosat-2 image time series irregularly acquired during the seasonal vegetation cycle from 2006 to 2014. Due to lot of clouds in the year 2006, an alternative 2006 time series using only cloud-free images has been added. Thirteen tree species were classified in each single-year dataset based on the Support Vector Machine (SVM) algorithm. The performances were assessed using a spatial leave-one-out cross validation (SLOO-CV) strategy, thereby guaranteeing full independence of the validation samples, and compared with standard non-spatial leave-one-out cross-validation (LOO-CV). The results show relatively close statistical performances from one year to the next despite the differences between the annual time series. Good agreements between years were observed in monospecific tree plantations of broadleaf species versus high disparity in other forests composed of different species. A strong positive bias in the accuracy assessment (up to 0.4 of Overall Accuracy (OA)) was also found when spatial dependence in the validation data was not removed. Using the SLOO-CV approach, the average OA values per year ranged from 0.48 for 2006 to 0.60 for 2013, which satisfactorily represents the spatial instability of species prediction between years.

Download Full-text