scholarly journals On the importance of multiple-component evaluation of spatial patterns for optimization of earth system models – A case study using mHM v5.6 at catchment scale

2017 ◽  
Author(s):  
Julian Koch ◽  
Mehmet C. Demirel ◽  
Simon Stisen
Keyword(s):  
Spatial Pattern ◽  
Spatial Patterns ◽  
Model Calibration ◽  
Performance Metrics ◽  
Hydrologic Model ◽  
System Modelling ◽  
Earth System ◽  
Spatial Efficiency ◽  
Spatial Performance ◽  
Multiple Component

Abstract. The process of model evaluation is not only an integral part of model development and calibration but also of paramount importance when communicating modelling results to the scientific community and stakeholders. The modelling community has a large and well tested toolbox of metrics to evaluate temporal model performance. On the contrary, spatial performance evaluation is not corresponding to the grand availability of spatial observations readily available and to the sophisticate model codes simulating the spatial variability of complex earth system processes. This study makes a contribution towards advancing spatial pattern oriented model calibration by rigorously testing a multiple-component performance metric. The promoted SPAtial EFficiency (SPAEF) metric reflects three equally weighted components: correlation, coefficient of variation and histogram overlap. This multiple-component approach is found to be advantageous in order to achieve the complex task of comparing spatial patterns. SPAEF, its three components individually and two alternative spatial performance metrics, i.e. connectivity analysis and fractions skill score, are applied in a spatial pattern oriented model calibration of a catchment model in Denmark. Results suggest the importance of multiple-component metrics, because stand-alone metrics tend to fail to provide holistic pattern information to the optimizer. The three SPAEF components are found to be independent which allows them to complement each other in a meaningful way. In order to optimally exploit spatial observations made available by remote sensing platforms this study suggests applying bias insensitive metrics which further allow comparing variables which are related but may differ in unit. This study applies SPAEF in the hydrological context using the mesoscale Hydrologic Model (mHM; version 5.6), but we see great potential across disciplines related to spatial distributed earth system modelling.

scholarly journals The SPAtial EFficiency metric (SPAEF): multiple-component evaluation of spatial patterns for optimization of hydrological models

2018 ◽  
Vol 11 (5) ◽  
pp. 1873-1886 ◽  
Author(s):  
Julian Koch ◽  
Mehmet Cüneyd Demirel ◽  
Simon Stisen
Keyword(s):  
Spatial Pattern ◽  
Spatial Patterns ◽  
Model Calibration ◽  
Performance Metrics ◽  
Model Performance ◽  
Hydrologic Model ◽  
System Modelling ◽  
Spatial Efficiency ◽  
Spatial Performance ◽  
Multiple Component

Abstract. The process of model evaluation is not only an integral part of model development and calibration but also of paramount importance when communicating modelling results to the scientific community and stakeholders. The modelling community has a large and well-tested toolbox of metrics to evaluate temporal model performance. In contrast, spatial performance evaluation does not correspond to the grand availability of spatial observations readily available and to the sophisticate model codes simulating the spatial variability of complex hydrological processes. This study makes a contribution towards advancing spatial-pattern-oriented model calibration by rigorously testing a multiple-component performance metric. The promoted SPAtial EFficiency (SPAEF) metric reflects three equally weighted components: correlation, coefficient of variation and histogram overlap. This multiple-component approach is found to be advantageous in order to achieve the complex task of comparing spatial patterns. SPAEF, its three components individually and two alternative spatial performance metrics, i.e. connectivity analysis and fractions skill score, are applied in a spatial-pattern-oriented model calibration of a catchment model in Denmark. Results suggest the importance of multiple-component metrics because stand-alone metrics tend to fail to provide holistic pattern information. The three SPAEF components are found to be independent, which allows them to complement each other in a meaningful way. In order to optimally exploit spatial observations made available by remote sensing platforms, this study suggests applying bias insensitive metrics which further allow for a comparison of variables which are related but may differ in unit. This study applies SPAEF in the hydrological context using the mesoscale Hydrologic Model (mHM; version 5.8), but we see great potential across disciplines related to spatially distributed earth system modelling.

scholarly journals Combining satellite data and appropriate objective functions for improved spatial pattern performance of a distributed hydrologic model

2018 ◽  
Vol 22 (2) ◽  
pp. 1299-1315 ◽  
Author(s):  
Mehmet C. Demirel ◽  
Juliane Mai ◽  
Gorka Mendiguren ◽  
Julian Koch ◽  
Luis Samaniego ◽  
...  
Keyword(s):  
Water Balance ◽  
Spatial Pattern ◽  
Spatial Patterns ◽  
Spatial Model ◽  
Hydrologic Model ◽  
Actual Evapotranspiration ◽  
Parameter Distribution ◽  
Objective Functions ◽  
Performance Metric ◽  
Spatial Performance

Abstract. Satellite-based earth observations offer great opportunities to improve spatial model predictions by means of spatial-pattern-oriented model evaluations. In this study, observed spatial patterns of actual evapotranspiration (AET) are utilised for spatial model calibration tailored to target the pattern performance of the model. The proposed calibration framework combines temporally aggregated observed spatial patterns with a new spatial performance metric and a flexible spatial parameterisation scheme. The mesoscale hydrologic model (mHM) is used to simulate streamflow and AET and has been selected due to its soil parameter distribution approach based on pedo-transfer functions and the build in multi-scale parameter regionalisation. In addition two new spatial parameter distribution options have been incorporated in the model in order to increase the flexibility of root fraction coefficient and potential evapotranspiration correction parameterisations, based on soil type and vegetation density. These parameterisations are utilised as they are most relevant for simulated AET patterns from the hydrologic model. Due to the fundamental challenges encountered when evaluating spatial pattern performance using standard metrics, we developed a simple but highly discriminative spatial metric, i.e. one comprised of three easily interpretable components measuring co-location, variation and distribution of the spatial data. The study shows that with flexible spatial model parameterisation used in combination with the appropriate objective functions, the simulated spatial patterns of actual evapotranspiration become substantially more similar to the satellite-based estimates. Overall 26 parameters are identified for calibration through a sequential screening approach based on a combination of streamflow and spatial pattern metrics. The robustness of the calibrations is tested using an ensemble of nine calibrations based on different seed numbers using the shuffled complex evolution optimiser. The calibration results reveal a limited trade-off between streamflow dynamics and spatial patterns illustrating the benefit of combining separate observation types and objective functions. At the same time, the simulated spatial patterns of AET significantly improved when an objective function based on observed AET patterns and a novel spatial performance metric compared to traditional streamflow-only calibration were included. Since the overall water balance is usually a crucial goal in hydrologic modelling, spatial-pattern-oriented optimisation should always be accompanied by traditional discharge measurements. In such a multi-objective framework, the current study promotes the use of a novel bias-insensitive spatial pattern metric, which exploits the key information contained in the observed patterns while allowing the water balance to be informed by discharge observations.

scholarly journals Combining satellite data and appropriate objective functions for improved spatial pattern performance of a distributed hydrologic model

2017 ◽  
Author(s):  
Mehmet C. Demirel ◽  
Juliane Mai ◽  
Gorka Mendiguren ◽  
Julian Koch ◽  
Luis Samaniego ◽  
...  
Keyword(s):  
Water Balance ◽  
Spatial Pattern ◽  
Spatial Patterns ◽  
Spatial Model ◽  
Hydrologic Model ◽  
Actual Evapotranspiration ◽  
Parameter Distribution ◽  
Objective Functions ◽  
Performance Metric ◽  
Spatial Performance

Abstract. Satellite based earth observations offer great opportunities to improve spatial model predictions by means of spatial pattern oriented model evaluations. In this study, observed spatial patterns of actual evapotranspiration (AET) are utilized for spatial model calibration tailored to target the pattern performance of the model. The proposed calibration framework combines temporally aggregated observed spatial patterns with a new spatial performance metric and a flexible spatial parameterisation scheme. The mesoscale Hydrologic Model (mHM) is used to simulate streamflow and AET and has been selected due to its soil parameter distribution approach based on pedotransfer functions and the build in multiscale parameter regionalization. In addition two new domain specific spatial parameter distribution options have been incorporated in the model in order to increase the flexibility of root fraction coefficient and potential evapotranspiration correction parameterisations, based on soil type and vegetation density. These parametrisations are utilized as they are most relevant for simulated AET patterns from the hydrologic model. Due to the fundamental challenges encountered when evaluating spatial pattern performance using standard metrics, we developed a simple but highly discriminative spatial metric i.e. comprised of three easily interpretable components measuring co-location, variation and distribution of the spatial data. The study shows that with flexible spatial model parameterisation used in combination with the appropriate objective functions, the simulated spatial patterns of actual evapotranspiration become substantially more similar to the satellite based estimates. Overall 26 parameters are identified for calibration through a sequential screening approach based on a combination of streamflow and spatial pattern metrics. The robustness of the calibrations is tested using an ensemble of nine calibrations based on different seed numbers using the shuffled complex evolution optimizer. The calibration results reveal a limited trade-offs between streamflow dynamics and spatial patterns illustrating the benefit of combining separate observation types and objective functions. At the same time, the simulated spatial patterns of AET significantly improved when including an objective function based on observed AET patterns and a novel spatial performance metric compared to traditional streamflow only calibration. Since the overall water balance is usually a crucial goal in the hydrologic modelling, spatial pattern oriented optimization should always be accompanied by traditional discharge measurements. In such a multi-objective framework, the current study promotes the use of a novel bias-insensitive spatial pattern metric, which exploits the key information contained in the observed patterns while allowing the water balance to be informed by discharge observations.

scholarly journals Spatial Pattern Oriented Multi-Criteria Sensitivity Analysis of a Distributed Hydrologic Model

2018 ◽  
Author(s):  
Mehmet Cüneyd Demirel ◽  
Julian Koch ◽  
Gorka Mendiguren ◽  
Simon Stisen
Keyword(s):  
Sensitivity Analysis ◽  
Spatial Variability ◽  
Spatial Pattern ◽  
Spatial Patterns ◽  
Performance Measures ◽  
Sampling Strategy ◽  
Behavioral Model ◽  
Hydrologic Model ◽  
Actual Evapotranspiration ◽  
Model Parameters

Hydrologic models are conventionally constrained and evaluated using point measurements of streamflow, which represents an aggregated catchment measure. As a consequence of this single objective focus, model parametrization and model parameter sensitivity are typically not reflecting other aspects of catchment behavior. Specifically for distributed models, the spatial pattern aspect is often overlooked. Our paper examines the utility of multiple performance measures in a spatial sensitivity analysis framework to determine the key parameters governing the spatial variability of predicted actual evapotranspiration (AET). Latin hypercube one-at-a-time (LHS-OAT) sampling strategy with multiple initial parameter sets was applied using the mesoscale hydrologic model (mHM) and a total of 17 model parameters were identified as sensitive. The results indicate different parameter sensitivities for different performance measures focusing on temporal hydrograph dynamics and spatial variability of actual evapotranspiration. While spatial patterns were found to be sensitive to vegetation parameters, streamflow dynamics were sensitive to pedo-transfer function (PTF) parameters. Above all, our results show that behavioral model definition based only on streamflow metrics in the generalized likelihood uncertainty estimation (GLUE) type methods require reformulation by incorporating spatial patterns into the definition of threshold values to reveal robust hydrologic behavior in the analysis.

Model evaluation expectations of European ESM communities: results from a survey

2021 ◽  
Author(s):  
Jerome Servonnat ◽  
Eric Guilyardi ◽  
Zofia Stott ◽  
Kim Serradell ◽  
Axel Lauer ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Common Ground ◽  
Performance Metrics ◽  
Coupled Model ◽  
System Modelling ◽  
Earth System ◽  
Evaluation Tool ◽  
Technical Standards ◽  
Community Effort ◽  
And Performance

<p>Developing an Earth system model evaluation tool for a broad user community is a real challenge, as the potential users do not necessarily have the same needs or expectations. While many evaluation tasks across user communities include common steps, significant differences are also apparent, not least the investment by institutions and individuals in bespoke tools. A key question is whether there is sufficient common ground to pursue a community tool with broad appeal and application.</p><p>We present the main results of a survey carried out by Assimila for the H2020 IS-ENES3 project to review the model evaluation needs of European Earth System Modelling communities. Interviewing approximately 30 participants among several European institutions, the survey targeted a broad range of users, including model developers, model users, evaluation data providers, and infrastructure providers. The output of the study provides an analysis of  requirements focusing on key technical, standards, and governance aspects.</p><p>The study used ESMValTool as a  current benchmark in terms of European evaluation tools. It is a community diagnostics and performance metrics tool for the evaluation of Earth System Models that allows for comparison of single or multiple models, either against predecessor versions or against observations. The tool is being developed in such a way that additional analyses can be added. As a community effort open to both users and developers, it encourages open exchange of diagnostic source code and evaluation results. It is currently used in Coupled Model Intercomparison Projects as well as for the development and testing of “new” models.</p><p>A key result of the survey is the widespread support for ESMValTool amongst users, developers, and even those who have taken or promote other approaches. The results of the survey identify priorities and opportunities in the further development of the ESMValTool to ensure long-term adoption of the tool by a broad community.</p>

scholarly journals Spatial Pattern Oriented Multicriteria Sensitivity Analysis of a Distributed Hydrologic Model

Water ◽  
2018 ◽  
Vol 10 (9) ◽  
pp. 1188 ◽  
Author(s):  
Mehmet Demirel ◽  
Julian Koch ◽  
Gorka Mendiguren ◽  
Simon Stisen
Keyword(s):  
Sensitivity Analysis ◽  
Spatial Variability ◽  
Spatial Pattern ◽  
Spatial Patterns ◽  
Performance Measures ◽  
Sampling Strategy ◽  
Behavioral Model ◽  
Hydrologic Model ◽  
Actual Evapotranspiration ◽  
Model Parameters

Hydrologic models are conventionally constrained and evaluated using point measurements of streamflow, which represent an aggregated catchment measure. As a consequence of this single objective focus, model parametrization and model parameter sensitivity typically do not reflect other aspects of catchment behavior. Specifically for distributed models, the spatial pattern aspect is often overlooked. Our paper examines the utility of multiple performance measures in a spatial sensitivity analysis framework to determine the key parameters governing the spatial variability of predicted actual evapotranspiration (AET). The Latin hypercube one-at-a-time (LHS-OAT) sampling strategy with multiple initial parameter sets was applied using the mesoscale hydrologic model (mHM) and a total of 17 model parameters were identified as sensitive. The results indicate different parameter sensitivities for different performance measures focusing on temporal hydrograph dynamics and spatial variability of actual evapotranspiration. While spatial patterns were found to be sensitive to vegetation parameters, streamflow dynamics were sensitive to pedo-transfer function (PTF) parameters. Above all, our results show that behavioral model definitions based only on streamflow metrics in the generalized likelihood uncertainty estimation (GLUE) type methods require reformulation by incorporating spatial patterns into the definition of threshold values to reveal robust hydrologic behavior in the analysis.

scholarly journals Spatial Sensitivity Analysis of Simulated Land Surface Patterns in a Catchment Model Using a Set of Innovative Spatial Performance Metrics

2017 ◽  
Vol 18 (4) ◽  
pp. 1121-1142 ◽  
Author(s):  
Julian Koch ◽  
Gorka Mendiguren ◽  
Gregoire Mariethoz ◽  
Simon Stisen
Keyword(s):  
Sensitivity Analysis ◽  
Spatial Patterns ◽  
Land Surface ◽  
Performance Metrics ◽  
Surface Model ◽  
Model Parameters ◽  
Spatial Sensitivity ◽  
Spatial Performance ◽  
Surface Patterns ◽  
Spatial Sensitivity Analysis

Abstract Distributed hydrological models simulate states and fluxes of water and energy in the terrestrial hydrosphere at each cell. The predicted spatial patterns result from complex nonlinear relationships and feedbacks. Spatial patterns are often neglected during the modeling process, and therefore a spatial sensitivity analysis framework that highlights their importance is proposed. This study features a comprehensive analysis of spatial patterns of actual evapotranspiration (ET) and land surface temperature (LST), with the aim of quantifying the extent to which forcing data and model parameters drive these patterns. This framework is applied on a distributed model [MIKE Système Hydrologique Européen (MIKE SHE)] coupled to a land surface model [Shuttleworth and Wallace–Evapotranspiration (SW-ET)] of a catchment in Denmark. Twenty-two scenarios are defined, each having a simplified representation of a potential driver of spatial variability. A baseline model that incorporates full spatial detail is used to assess sensitivity. High sensitivity can be attested in scenarios where the simulated spatial patterns differ significantly from the baseline. The core novelty of this study is that the analysis is based on a set of innovative spatial performance metrics that enable a reliable spatial pattern comparison. Overall, LST is very sensitive to air temperature and wind speed whereas ET is rather driven by vegetation. Both are sensitive to groundwater coupling and precipitation. The conclusions may be limited to the selected catchment and to the applied modeling system, but the suggested framework is generically relevant for the modeling community. While the applied metrics focus on specific spatial information, they partly exhibit redundant information. Thus, a combination of metrics is the ideal approach to evaluate spatial patterns in models outputs.

scholarly journals Using dry and wet year hydroclimatic extremes to guide future hydrologic projections

2016 ◽  
Vol 20 (7) ◽  
pp. 2811-2825 ◽  
Author(s):  
Stephen Oni ◽  
Martyn Futter ◽  
Jose Ledesma ◽  
Claudia Teutschbein ◽  
Jim Buttle ◽  
...  
Keyword(s):  
Model Calibration ◽  
Climate Models ◽  
Performance Metrics ◽  
Climate Change Impacts ◽  
Climate Impact ◽  
Hydrologic Model ◽  
Climatic Conditions ◽  
Extreme Conditions ◽  
Landscape Processes ◽  
Behavioural Parameter

Abstract. There are growing numbers of studies on climate change impacts on forest hydrology, but limited attempts have been made to use current hydroclimatic variabilities to constrain projections of future climatic conditions. Here we used historical wet and dry years as a proxy for expected future extreme conditions in a boreal catchment. We showed that runoff could be underestimated by at least 35 % when dry year parameterizations were used for wet year conditions. Uncertainty analysis showed that behavioural parameter sets from wet and dry years separated mainly on precipitation-related parameters and to a lesser extent on parameters related to landscape processes, while uncertainties inherent in climate models (as opposed to differences in calibration or performance metrics) appeared to drive the overall uncertainty in runoff projections under dry and wet hydroclimatic conditions. Hydrologic model calibration for climate impact studies could be based on years that closely approximate anticipated conditions to better constrain uncertainty in projecting extreme conditions in boreal and temperate regions.

scholarly journals Climate Normalized Spatial Patterns of Evapotranspiration Enhance the Calibration of a Hydrological Model

2022 ◽  
Vol 14 (2) ◽  
pp. 315
Author(s):  
Julian Koch ◽  
Mehmet Cüneyd Demirel ◽  
Simon Stisen
Keyword(s):  
Spatial Pattern ◽  
River Basin ◽  
Spatial Patterns ◽  
Hydrological Model ◽  
Hydrological Cycle ◽  
Model Parameters ◽  
Climate Control ◽  
Spatial Efficiency ◽  
Senegal River ◽  
Senegal River Basin

Spatial pattern-oriented evaluations of distributed hydrological models have contributed towards an improved realism of hydrological simulations. This advancement has been supported by the broad range of readily available satellite-based datasets of key hydrological variables, such as evapotranspiration (ET). At larger scale, spatial patterns of ET are often driven by underlying climate gradients, and with this study, we argue that gradient dominated patterns may hamper the potential of spatial pattern-oriented evaluation frameworks. We hypothesize that the climate control of spatial patterns of ET overshadows the effect model parameters have on the simulated patterns. To address this, we propose a climate normalization strategy. This is demonstrated for the Senegal River basin as a modeling case study, where the dominant north-south precipitation gradient is the main driver of the observed hydrological variability. We apply the mesoscale Hydrological Model (mHM) to model the hydrological cycle of the Senegal River basin. Two multi-objective calibration experiments investigate the effect of climate normalization. Both calibrations utilize observed discharge (Q) in combination with remote sensing ET data, where one is based on the original ET pattern and the other utilizes the normalized ET pattern. As objective functions we applied the Kling-Gupta-Efficiency (KGE) for Q and the Spatial Efficiency (SPAEF) for ET. We identify parameter sets that balance the tradeoffs between the two independent observations and find that the calibration using the normalized ET pattern does not compromise the spatial pattern performance of the original pattern. However, vice versa, this is not necessarily the case, since the calibration using the original ET pattern showed a poorer performance for the normalized pattern, i.e., a 30% decrease in SPAEF. Both calibrations reached comparable performance of Q, i.e., KGE around 0.7. With this study, we identified a general shortcoming of spatial pattern-oriented model evaluations using ET in basins dominated by a climate gradient, but we argue that this also applies to other variables such as, soil moisture or land surface temperature.

Sign in / Sign up

Export Citation Format

Share Document