The comprehensive differential split-sample test: A stress-test for hydrological model robustness under climate variability

2019 ◽  
Vol 573 ◽  
pp. 501-515 ◽  
Author(s):  
D.F. Motavita ◽  
R. Chow ◽  
A. Guthke ◽  
W. Nowak
Keyword(s):  
Climate Variability ◽  
Hydrological Model ◽  
Stress Test ◽  
Split Sample ◽  
Sample Test ◽  
Model Robustness

RAT (Robustness Assessment Test): a straightforward evaluation of hydrological model robustness to a changing climate

2021 ◽  
Author(s):  
Vazken Andréassian ◽  
Léonard Santos ◽  
Torben Sonnenborg ◽  
Alban de Lavenne ◽  
Göran Lindström ◽  
...  
Keyword(s):  
Climate Change ◽  
Hydrological Model ◽  
Hydrological Models ◽  
Climate Variables ◽  
Data Set ◽  
Hydrological Characteristics ◽  
Changing Climate ◽  
Split Sample ◽  
Sample Test ◽  
Model Robustness

<p>Hydrological models are increasingly used under evolving climatic conditions. They should thus be evaluated regarding their temporal transferability (application in different time periods) and extrapolation capacity (application beyond the range of known past conditions). In theory, parameters of hydrological models are independent of climate. In practice, however, many published studies based on the Split-Sample Test (Klemeš, 1986), have shown that model performances decrease systematically when it is used out of its calibration period. The RAT test proposed here aims at evaluating model robustness to a changing climate by assessing potential undesirable dependencies of hydrological model performances to climate variables. The test compares, over a long data period, the annual value of several climate variables (temperature, precipitation and aridity index) and the bias of the model over each year. If a significant relation exists between the climatic variable and the bias, the model is not considered to be robust to climate change on the catchment. The test has been compared to the Generalized Split-Sample Test (Coron et al., 2012) and showed similar results.</p><p>Here, we report on a large scale application of the test for three hydrological models with different level of complexity (GR6J, HYPE, MIKE-SHE) on a data set of 352 catchments in Denmark, France and Sweden. The results show that the test behaves differently given the evaluated variable (be temperature, precipitation or aridity) and the hydrological characteristics of each catchment. They also show that, although of different level of complexity, the robustness of the three models is similar on the overall data set. However, they are not robust on the same catchments and, then, are not sensitive to the same hydrological characteristics. This example highlights the applicability of the RAT test regardless of the model set-up and calibration procedure and its ability to provide a first evaluation of the model robustness to climate change.</p><p> </p><p><strong>References</strong></p><p>Coron, L., V. Andréassian, C. Perrin, J. Lerat, J. Vaze, M. Bourqui, and F. Hendrickx, 2012. Crash testing hydrological models in contrasted climate conditions: An experiment on 216 Australian catchments, Water Resour. Res., 48, W05552, doi:10.1029/2011WR011721</p><p>Klemeš, V., 1986. Operational testing of hydrological simulation models, Hydrol. Sci. J., 31, 13–24, doi:10.1080/02626668609491024</p><p> </p>

scholarly journals Technical note: PMR – a proxy metric to assess hydrological model robustness in a changing climate

2021 ◽  
Author(s):  
Paul Royer-Gaspard ◽  
Vazken Andréassian ◽  
Guillaume Thirel
Keyword(s):  
Hydrological Model ◽  
Evaluation Studies ◽  
Computational Cost ◽  
Climatic Conditions ◽  
Technical Note ◽  
Performance Metric ◽  
Reliable Assessment ◽  
Sample Test ◽  
The Impact ◽  
Model Robustness

Abstract. The ability of hydrological models to perform in climatic conditions different from those encountered in calibration is crucial to ensure a reliable assessment of the impact of climate change in water management sectors. However, most evaluation studies based on the Differential Split-Sample Test (DSST) endorsed the consensus that rainfall-runoff models lack climatic robustness. Models typically exhibit substantial errors on streamflow volumes applied under climatologically different conditions. In this technical note, we propose a new performance metric to evaluate model robustness without applying the DSST and which performs with a single hydrological model calibration. The Proxy for Model Robustness (PMR) is based on the systematic computation of model error on sliding sub-periods of the whole streamflow time series. We demonstrate that the metric shows patterns similar to those obtained with the DSST for a conceptual model on a set of 377 French catchments. An analysis of sensitivity to the length of the sub-periods shows that this length influences the values of the PMR and its adequation with DSST biases. We recommend a range of a few years for the choice of sub-period lengths, although this should be context-dependent. Our work makes it possible to evaluate the temporal transferability of any hydrological model, including uncalibrated models, at a very low computational cost.

Integration of Hidden Markov States in a hydrological model calibration/validation protocol

2021 ◽  
Author(s):  
Étienne Guilpart ◽  
Vahid Espanmanesh ◽  
Amaury Tilmant ◽  
François Anctil
Keyword(s):  
Time Series ◽  
Hidden Markov Models ◽  
Model Calibration ◽  
Hydrological Model ◽  
Markov Models ◽  
Hidden Markov ◽  
Change Points ◽  
Climate Trend ◽  
Split Sample ◽  
Sample Test

<p>Due to climate changes, the stationary assumption in hydrology has become obsolete. Moreover, the uncertainty regarding the future evolution of the Earth's climate and its impact on flow regimes is still large. Over the last decade, new risk management approaches have been proposed to support water resources planning under deep uncertainty. Those approaches rely at some point on a hydrological model to derive time series of streamflows for various hydro-climatic scenarios. One of the key issue is to make sure that the hydrological model is robust, i.e. that it performs well over contrasted hydro-climatic conditions. The differential split-sample test principle proposed by Klemes in 1986 recommends partitioning the time series into numerous and independent subperiods with different stationary climate features. Then, the hydrological model calibration is achieved on a specific climate period, and the validation on other(s). Classical detection methods commonly used to partition the times series, such as Mann-Kendall test or Pettitt test, can only detect a single change point, and thus are unable to handle complex climate sequences with multiple change points. We propose a calibration/validation protocol of hydrological models which rely on both the differential split-sample test and on an Hidden Markov Model to identify a succession of subsequences in a time series based on the state of the underlying process. We applied the proposed protocol on the Senegal River (West Africa). The hydrological model used is the conceptual GR2M model. Results show that (i) when the river discharges time series does not display a clear climate trend, and have multiple change points, classical rupture tests are not suitable. Hidden Markov Models are a good alternative as long as the climate sub-sequences are long enough (typically around 30 years or more); (ii) including a Hidden Markov Models in such protocol open up the range of possibilities for calibrate/validate, which can lead to an enhancement of the criterion function (but not necessarily).</p><p>Klemes, V.: Operational testing of hydrological simulation models, Hydrological Sciences Journal, 31, 13-24, 415 https://doi.org/10.1080/02626668609491024, 1986.</p>

scholarly journals The transferability of hydrological models under nonstationary climatic conditions

2012 ◽  
Vol 16 (4) ◽  
pp. 1239-1254 ◽  
Author(s):  
C. Z. Li ◽  
L. Zhang ◽  
H. Wang ◽  
Y. Q. Zhang ◽  
F. L. Yu ◽  
...  
Keyword(s):  
Monte Carlo Simulation ◽  
Monte Carlo ◽  
Hydrological Model ◽  
Climate Scenario ◽  
Climatic Conditions ◽  
Performance Criteria ◽  
Future Climate Change ◽  
Hydrological Models ◽  
Split Sample ◽  
Sample Test

Abstract. This paper investigates issues involved in calibrating hydrological models against observed data when the aim of the modelling is to predict future runoff under different climatic conditions. To achieve this objective, we tested two hydrological models, DWBM and SIMHYD, using data from 30 unimpaired catchments in Australia which had at least 60 yr of daily precipitation, potential evapotranspiration (PET), and streamflow data. Nash-Sutcliffe efficiency (NSE), modified index of agreement (d1) and water balance error (WBE) were used as performance criteria. We used a differential split-sample test to split up the data into 120 sub-periods and 4 different climatic sub-periods in order to assess how well the calibrated model could be transferred different periods. For each catchment, the models were calibrated for one sub-period and validated on the other three. Monte Carlo simulation was used to explore parameter stability compared to historic climatic variability. The chi-square test was used to measure the relationship between the distribution of the parameters and hydroclimatic variability. The results showed that the performance of the two hydrological models differed and depended on the model calibration. We found that if a hydrological model is set up to simulate runoff for a wet climate scenario then it should be calibrated on a wet segment of the historic record, and similarly a dry segment should be used for a dry climate scenario. The Monte Carlo simulation provides an effective and pragmatic approach to explore uncertainty and equifinality in hydrological model parameters. Some parameters of the hydrological models are shown to be significantly more sensitive to the choice of calibration periods. Our findings support the idea that when using conceptual hydrological models to assess future climate change impacts, a differential split-sample test and Monte Carlo simulation should be used to quantify uncertainties due to parameter instability and non-uniqueness.

scholarly journals Technical Note – RAT: a Robustness Assessment Test for calibrated and uncalibrated hydrological models

2021 ◽  
Author(s):  
Pierre Nicolle ◽  
Vazken Andréassian ◽  
Paul Royer-Gaspard ◽  
Charles Perrin ◽  
Guillaume Thirel ◽  
...  
Keyword(s):  
Climate Change ◽  
Climate Change Impact ◽  
Hydrological Model ◽  
Technical Note ◽  
New Method ◽  
Hydrological Models ◽  
Split Sample ◽  
Sample Test ◽  
Change Impact ◽  
Robustness Assessment

Abstract. In this note, we present RAT, a new method to assess the robustness of hydrological models. RAT can be seen as an alternative to the classical split-sample test widely used in hydrology. And because the RAT method does not require multiple calibrations, we suggest that it can be applied even to uncalibrated models. The RAT method can be used to determine whether a hydrological model is "safe" for being used for climate change impact studies.

scholarly journals Technical note: PMR – a proxy metric to assess hydrological model robustness in a changing climate

2021 ◽  
Vol 25 (11) ◽  
pp. 5703-5716
Author(s):  
Paul Royer-Gaspard ◽  
Vazken Andréassian ◽  
Guillaume Thirel
Keyword(s):  
Hydrological Model ◽  
Evaluation Studies ◽  
Computational Cost ◽  
Climatic Conditions ◽  
Technical Note ◽  
Performance Metric ◽  
Reliable Assessment ◽  
Sample Test ◽  
The Impact ◽  
Model Robustness

Abstract. The ability of hydrological models to perform in climatic conditions different from those encountered in calibration is crucial to ensure a reliable assessment of the impact of climate change on river regimes and water availability. However, most evaluation studies based on the differential split-sample test (DSST) endorsed the consensus that rainfall–runoff models lack climatic robustness. Models applied under climatologically different conditions typically exhibit substantial errors in streamflow volumes. In this technical note, we propose a new performance metric to evaluate model robustness without applying the DSST, and it can be performed with a single hydrological model calibration. The proxy for model robustness (PMR) is based on the systematic computation of model error on sliding sub-periods of the whole streamflow time series. We demonstrate that the PMR metric shows patterns similar to those obtained with the DSST for a conceptual model on a set of 377 French catchments. An analysis of the sensitivity to the length of the sub-periods shows that this length influences the values of the PMR and its equivalency with DSST biases. We recommend a range of a few years for the choice of sub-period lengths, although this should be context dependent. Our work makes it possible to evaluate the temporal transferability of any hydrological model, including uncalibrated models, at a very low computational cost.

scholarly journals The transferability of hydrological models under nonstationary climatic conditions

2011 ◽  
Vol 8 (5) ◽  
pp. 8701-8736 ◽  
Author(s):  
C. Z. Li ◽  
L. Zhang ◽  
H. Wang ◽  
Y. Q. Zhang ◽  
F. L. Yu ◽  
...  
Keyword(s):  
Monte Carlo Simulation ◽  
Monte Carlo ◽  
Hydrological Model ◽  
Climate Scenario ◽  
Climatic Conditions ◽  
Performance Criteria ◽  
Future Climate Change ◽  
Hydrological Models ◽  
Split Sample ◽  
Sample Test

Abstract. This paper investigates issues involved in calibrating hydrological models against observed data when the aim of the modelling is to predict future runoff under different climatic conditions. To achieve this objective, we tested two hydrological models, DWBM and SIMHYD, using data from 30 unimpaired catchments in Australia which had at least 60 years of daily precipitation, potential evapotranspiration (PET), and streamflow data. Nash-Sutcliffe efficiency (NSE) and absolute percentage water balance error (WBE) were used as performance criteria. We used a differential split-sample test to split up the data into 120 sub-periods and 4 different climatic sub-periods in order to assess how well the calibrated model could be transferred different periods. For each catchment, the models were calibrated for one sub-period and validated on the other three. Monte Carlo simulation was used to explore parameter stability compared to historic climatic variability. The chi-square test was used to measure the relationship between the distribution of the parameters and hydroclimatic variability. The results showed that the performance of the two hydrological models differed and depended on the model calibration. We found that if a hydrological model is set up to simulate runoff for a wet climate scenario then it should be calibrated on a wet segment of the historic record, and similarly a dry segment should be used for a dry climate scenario. The Monte Carlo simulation provides an effective and pragmatic approach to explore uncertainty and equifinality in hydrological model parameters. Some parameters of the hydrological models are shown to be significantly more sensitive to the choice of calibration periods. Our findings support the idea that when using conceptual hydrological models to assess future climate change impacts, a differential split-sample test and Monte Carlo simulation can reduce uncertainties due to parameter instability and non-uniqueness.

Sign in / Sign up

Export Citation Format

Share Document