scholarly journals Global Gridded Crop Model evaluation: benchmarking, skills, deficiencies and implications

2016 ◽  
Author(s):  
Christoph Müller ◽  
Joshua Elliott ◽  
James Chryssanthacopoulos ◽  
Almut Arneth ◽  
Juraj Balkovic ◽  
...  
Keyword(s):  
Time Series ◽  
Model Evaluation ◽  
Reference Data ◽  
Model Performance ◽  
Global Scale ◽  
Crop Model ◽  
Yield Variability ◽  
Crop Modeling ◽  
Model Intercomparison ◽  
Simulation Results

Abstract. Crop models are increasingly used to simulate crop yields at the global scale, but there so far is no general framework on how to assess model performance. We here evaluate the simulation results of 14 global gridded crop modeling groups that have contributed historic crop yield simulations for maize, wheat, rice and soybean to the Global Gridded Crop Model Intercomparison (GGCMI) of the Agricultural Model Intercomparison and Improvement Project (AgMIP). Simulation results are compared to reference data at global, national and grid cell scales and we evaluate model performance with respect to time series correlation, spatial correlation and mean bias. We find that GGCMs show mixed skill in reproducing time-series correlations or spatial patterns at the different spatial scales. Generally, maize, wheat and soybean simulations of many GGCMs are capable of reproducing larger parts of observed temporal variability (time series correlation coefficients (r) of up to 0.888 for maize, 0.673 for wheat and 0.643 for soybean at the global scale) but rice yield variability cannot be well reproduced by most models. Yield variability can be well reproduced for most major producer countries by many GGCMS and for all countries by at least some. A comparison with gridded yield data and a statistical analysis of the effects of weather variability on yield variability shows that the ensemble of GGCMs can explain more of the yield variability than an ensemble of regression models for maize and soybean, but not for wheat and rice. We identify future research needs in global gridded crop modeling and for all individual crop modeling groups. In the absence of a purely observation-based benchmark for model evaluation, we propose that the best performing crop model per crop and region establishes the benchmark for all others, and modelers are encouraged to investigate how crop model performance can be increased. We make our evaluation system accessible to all crop modelers so that also other modeling groups can test their model performance against the reference data and the GGCMI benchmark.

scholarly journals Global gridded crop model evaluation: benchmarking, skills, deficiencies and implications

2017 ◽  
Vol 10 (4) ◽  
pp. 1403-1422 ◽  
Author(s):  
Christoph Müller ◽  
Joshua Elliott ◽  
James Chryssanthacopoulos ◽  
Almut Arneth ◽  
Juraj Balkovic ◽  
...  
Keyword(s):  
Time Series ◽  
Reference Data ◽  
Model Performance ◽  
Global Scale ◽  
Crop Model ◽  
Yield Variability ◽  
Crop Modeling ◽  
Model Intercomparison ◽  
Crop Models ◽  
Simulation Results

Abstract. Crop models are increasingly used to simulate crop yields at the global scale, but so far there is no general framework on how to assess model performance. Here we evaluate the simulation results of 14 global gridded crop modeling groups that have contributed historic crop yield simulations for maize, wheat, rice and soybean to the Global Gridded Crop Model Intercomparison (GGCMI) of the Agricultural Model Intercomparison and Improvement Project (AgMIP). Simulation results are compared to reference data at global, national and grid cell scales and we evaluate model performance with respect to time series correlation, spatial correlation and mean bias. We find that global gridded crop models (GGCMs) show mixed skill in reproducing time series correlations or spatial patterns at the different spatial scales. Generally, maize, wheat and soybean simulations of many GGCMs are capable of reproducing larger parts of observed temporal variability (time series correlation coefficients (r) of up to 0.888 for maize, 0.673 for wheat and 0.643 for soybean at the global scale) but rice yield variability cannot be well reproduced by most models. Yield variability can be well reproduced for most major producing countries by many GGCMs and for all countries by at least some. A comparison with gridded yield data and a statistical analysis of the effects of weather variability on yield variability shows that the ensemble of GGCMs can explain more of the yield variability than an ensemble of regression models for maize and soybean, but not for wheat and rice. We identify future research needs in global gridded crop modeling and for all individual crop modeling groups. In the absence of a purely observation-based benchmark for model evaluation, we propose that the best performing crop model per crop and region establishes the benchmark for all others, and modelers are encouraged to investigate how crop model performance can be increased. We make our evaluation system accessible to all crop modelers so that other modeling groups can also test their model performance against the reference data and the GGCMI benchmark.

scholarly journals The GRENE-TEA model intercomparison project (GTMIP): overview and experiment protocol for Stage 1

2015 ◽  
Vol 8 (9) ◽  
pp. 2841-2856 ◽  
Author(s):  
S. Miyazaki ◽  
K. Saito ◽  
J. Mori ◽  
T. Yamazaki ◽  
T. Ise ◽  
...  
Keyword(s):  
Gross Primary Production ◽  
Model Performance ◽  
Global Scale ◽  
The Arctic ◽  
Energy Budgets ◽  
Model Intercomparison ◽  
Climate Change Research ◽  
Intercomparison Project ◽  
Arctic Climate Change ◽  
Stage 1

Abstract. As part of the terrestrial branch of the Japan-funded Arctic Climate Change Research Project (GRENE-TEA), which aims to clarify the role and function of the terrestrial Arctic in the climate system and assess the influence of its changes on a global scale, this model intercomparison project (GTMIP) is designed to (1) enhance communication and understanding between the modelling and field scientists and (2) assess the uncertainty and variations stemming from variability in model implementation/design and in model outputs using climatic and historical conditions in the Arctic terrestrial regions. This paper provides an overview of all GTMIP activity, and the experiment protocol of Stage 1, which is site simulations driven by statistically fitted data created using the GRENE-TEA site observations for the last 3 decades. The target metrics for the model evaluation cover key processes in both physics and biogeochemistry, including energy budgets, snow, permafrost, phenology, and carbon budgets. Exemplary results for distributions of four metrics (annual mean latent heat flux, annual maximum snow depth, gross primary production, and net ecosystem production) and for seasonal transitions are provided to give an outlook of the planned analysis that will delineate the inter-dependence among the key processes and provide clues for improving model performance.

scholarly journals The Global Gridded Crop Model intercomparison: data and modeling protocols for Phase 1 (v1.0)

2014 ◽  
Vol 7 (4) ◽  
pp. 4383-4427 ◽  
Author(s):  
J. Elliott ◽  
C. Müller ◽  
D. Deryng ◽  
J. Chryssanthacopoulos ◽  
K. J. Boote ◽  
...  
Keyword(s):  
Land Surface ◽  
Input Data ◽  
Large Scale ◽  
Phase 1 ◽  
Climate Impact ◽  
Crop Model ◽  
Surface Fluxes ◽  
Historical Period ◽  
Crop Modeling ◽  
Model Intercomparison

Abstract. We present protocols and input data for Phase 1 of the Global Gridded Crop Model Intercomparison, a project of the Agricultural Model Intercomparison and Improvement Project's (AgMIP's) Gridded Crop Modeling Initiative (AgGRID). The project includes global simulations of yields, phenologies, and many land-surface fluxes by 12–15 modeling groups for many crops, climate forcing datasets, and scenarios over the historical period from 1948–2012. The primary outcomes of the project include (1) a detailed comparison of the major differences and similarities among global models commonly used for large-scale climate impact assessment, (2) an evaluation of model and ensemble hindcasting skill, (3) quantification of key uncertainties from climate input data, model choice, and other sources, and (4) a multi-model analysis of the impacts to agriculture of large-scale climate extremes from the historical record.

scholarly journals Global land-surface evaporation estimated from satellite-based observations

2010 ◽  
Vol 7 (5) ◽  
pp. 8479-8519 ◽  
Author(s):  
D. G. Miralles ◽  
T. R. H. Holmes ◽  
R. A. M. De Jeu ◽  
J. H. Gash ◽  
A. G. C. A. Meesters ◽  
...  
Keyword(s):  
Time Series ◽  
Land Surface ◽  
Model Performance ◽  
Global Scale ◽  
Surface Conductance ◽  
Vegetation Density ◽  
Climate Conditions ◽  
Eddy Covariance Measurements ◽  
One Year ◽  
Daily Evaporation

Abstract. This paper outlines a new methodology to derive evaporation from satellite observations. The approach uses a variety of satellite-sensor products to estimate daily evaporation at a global scale, with a 0.25 degree spatial resolution. Central to this approach is the use of the Priestley and Taylor (PT) evaporation model. Because the PT equation is driven by net radiation, this strategy avoids the need to specify surface fields of variables, such as the surface conductance, which cannot be detected directly from space. Key distinguishing features are the use of microwave-derived soil moisture, land surface temperature and vegetation density, as well as the use of a detailed rainfall interception module. The modelled evaporation is validated against one year of eddy covariance measurements from 43 stations. The estimated annual totals correlate well with the stations' annual cumulative evaporation (R = 0.84, N = 43) and show a negligible bias (−1.5%). The validation of the daily time series at each individual station shows good model performance in all vegetation types and climate conditions with an average correlation coefficient of R = 0.84, still lower than the R = 0.91 found in the validation of the monthly time series. The first global map of annual evaporation developed through this methodology is also presented.

scholarly journals Global land-surface evaporation estimated from satellite-based observations

2011 ◽  
Vol 15 (2) ◽  
pp. 453-469 ◽  
Author(s):  
D. G. Miralles ◽  
T. R. H. Holmes ◽  
R. A. M. De Jeu ◽  
J. H. Gash ◽  
A. G. C. A. Meesters ◽  
...  
Keyword(s):  
Time Series ◽  
Land Surface ◽  
Model Performance ◽  
Global Scale ◽  
Vegetation Density ◽  
Climate Conditions ◽  
New Strategy ◽  
Interception Loss ◽  
One Year ◽  
Daily Evaporation

Abstract. This paper outlines a new strategy to derive evaporation from satellite observations. The approach uses a variety of satellite-sensor products to estimate daily evaporation at a global scale and 0.25 degree spatial resolution. Central to this methodology is the use of the Priestley and Taylor (PT) evaporation model. The minimalistic PT equation combines a small number of inputs, the majority of which can be detected from space. This reduces the number of variables that need to be modelled. Key distinguishing features of the approach are the use of microwave-derived soil moisture, land surface temperature and vegetation density, as well as the detailed estimation of rainfall interception loss. The modelled evaporation is validated against one year of eddy covariance measurements from 43 stations. The estimated annual totals correlate well with the stations' annual cumulative evaporation (R=0.80, N=43) and present a low average bias (−5%). The validation of the daily time series at each individual station shows good model performance in all vegetation types and climate conditions with an average correlation coefficient of R=0.83, still lower than the R=0.90 found in the validation of the monthly time series. The first global map of annual evaporation developed through this methodology is also presented.

scholarly journals The GRENE-TEA Model Intercomparison Project (GTMIP): overview and experiment protocol for Stage 1

2015 ◽  
Vol 8 (4) ◽  
pp. 3443-3479 ◽  
Author(s):  
S. Miyazaki ◽  
K. Saito ◽  
J. Mori ◽  
T. Yamazaki ◽  
T. Ise ◽  
...  
Keyword(s):  
Gross Primary Production ◽  
Model Performance ◽  
Global Scale ◽  
The Arctic ◽  
Energy Budgets ◽  
Model Intercomparison ◽  
Climate Change Research ◽  
Intercomparison Project ◽  
Arctic Climate Change ◽  
Stage 1

Abstract. As part of the terrestrial branch of the Japan-funded Arctic Climate Change Research Project (GRENE-TEA), which aims to clarify the role and function of the Arctic terrestrial system in the climate system, and assess the influence of its changes on a global scale, this model intercomparison project (GTMIP) is planned and being conducted to (1) enhance communication and understanding between the "minds and hands" (i.e., between the modelling and field scientists) and (2) assess the uncertainty and variations stemming from variability in model implementation/design and in model outputs due to climatic and historical conditions in the Arctic terrestrial regions. This paper provides an overview and the experiment protocol of Stage 1 of the project, site simulations driven by statistically fitted data created using the GRENE-TEA site observations for the last three decades. The target metrics for the model evaluation cover key processes in both physics and biogeochemistry, including energy budgets, snow, permafrost, phenology, and carbon budgets. The preliminary results on four metrics (annual mean latent heat flux, annual maximum snow depth, gross primary production, and net ecosystem production) already demonstrate the range of variations in reproducibility among existing models and sites. Full analysis on annual as well as seasonal time scales, to be conducted upon completion of model outputs submission, will delineate inter-dependence among the key processes, and provide the clue for improving the model performance.

Constraining uncertainty in projected precipitation over land with Causal Discovery

2021 ◽  
Author(s):  
Kevin Debeire ◽  
Veronika Eyring ◽  
Peer Nowack ◽  
Jakob Runge
Keyword(s):  
Time Series ◽  
Model Evaluation ◽  
Climate Model ◽  
Coupled Model ◽  
Causal Discovery ◽  
Model Intercomparison ◽  
Simulated Precipitation ◽  
Intercomparison Project ◽  
Precipitation Changes ◽  
Global Precipitation

<p>The models participating in the Coupled Model Intercomparison Project Phase 6 (CMIP6) deliver insights on the evolution of the Earth's  climate. The global precipitation changes follow the magnitude of the warming according to a recent study (Tebaldi, Debeire, Eyring et al., 2020) of the CMIP6 ensemble-mean. However, Earth systems models exhibit a large range in simulated precipitation projection over land. In this study, we present a potential approach to constrain the precipitation changes over land globally and regionally. This approach performs a process-oriented model evaluation similar to Nowack et al. study. We evaluate the ability of models to represent atmospheric dynamical interactions by applying Causal Discovery algorithm. We find a relationship between the ability to represent dynamical interactions close to the observations and the projected precipitation changes over land of the model. We show how this relationship can be used to constrain projection of precipitation over land.</p><p> </p><p>References:</p><p>Nowack, P., Runge, J., Eyring, V. et al. Causal networks for climate model evaluation and constrained projections. Nat Commun 11, 1415. https://doi.org/10.1038/s41467-020-15195-y, 2020.</p><p>Runge, J., P. Nowack, M. Kretschmer, S. Flaxman, D. Sejdinovic, Detecting and quantifying causal associations in large nonlinear time series datasets. Sci. Adv. 5, eaau4996, 2019.</p><p>Runge, J, Causal Network Reconstruction from Time Series: From Theoretical Assumptions to Practical Estimation. Chaos: An Interdisciplinary Journal of Nonlinear Science 28 (7): 075310. https://aip.scitation.org/doi/10.1063/1.5025050, 2018.</p><p>Tebaldi, C., Debeire, K., Eyring, V., Fischer, E., Fyfe, J., Friedlingstein, P., Knutti, R., Lowe, J., O'Neill, B., Sanderson, B., van Vuuren, D., Riahi, K., Meinshausen, M., Nicholls, Z., Hurtt, G., Kriegler, E., Lamarque, J.-F., Meehl, G., Moss, R., Bauer, S. E., Boucher, O., Brovkin, V., Golaz, J.-C., Gualdi, S., Guo, H., John, J. G., Kharin, S., Koshiro, T., Ma, L., Olivié, D., Panickal, S., Qiao, F., Rosenbloom, N., Schupfner, M., Seferian, R., Song, Z., Steger, C., Sellar, A., Swart, N., Tachiiri, K., Tatebe, H., Voldoire, A., Volodin, E., Wyser, K., Xin, X., Xinyao, R., Yang, S., Yu, Y., and Ziehn, T.: Climate model projections from the Scenario Model Intercomparison Project (ScenarioMIP) of CMIP6, Earth Syst. Dynam. Discuss. [preprint], https://doi.org/10.5194/esd-2020-68, in review, 2020.</p>

scholarly journals Technical note: Inherent benchmark or not? Comparing Nash–Sutcliffe and Kling–Gupta efficiency scores

2019 ◽  
Vol 23 (10) ◽  
pp. 4323-4331 ◽  
Author(s):  
Wouter J. M. Knoben ◽  
Jim E. Freer ◽  
Ross A. Woods
Keyword(s):  
Time Series ◽  
Coefficient Of Variation ◽  
Ad Hoc ◽  
Model Performance ◽  
Technical Note ◽  
Mean Flow ◽  
Model Adequacy ◽  
Robust Model ◽  
The Mean ◽  
Adequacy Assessment

Abstract. A traditional metric used in hydrology to summarize model performance is the Nash–Sutcliffe efficiency (NSE). Increasingly an alternative metric, the Kling–Gupta efficiency (KGE), is used instead. When NSE is used, NSE = 0 corresponds to using the mean flow as a benchmark predictor. The same reasoning is applied in various studies that use KGE as a metric: negative KGE values are viewed as bad model performance, and only positive values are seen as good model performance. Here we show that using the mean flow as a predictor does not result in KGE = 0, but instead KGE =1-√2≈-0.41. Thus, KGE values greater than −0.41 indicate that a model improves upon the mean flow benchmark – even if the model's KGE value is negative. NSE and KGE values cannot be directly compared, because their relationship is non-unique and depends in part on the coefficient of variation of the observed time series. Therefore, modellers who use the KGE metric should not let their understanding of NSE values guide them in interpreting KGE values and instead develop new understanding based on the constitutive parts of the KGE metric and the explicit use of benchmark values to compare KGE scores against. More generally, a strong case can be made for moving away from ad hoc use of aggregated efficiency metrics and towards a framework based on purpose-dependent evaluation metrics and benchmarks that allows for more robust model adequacy assessment.

scholarly journals Global Analysis of Atmospheric Transmissivity Using Cloud Cover, Aridity and Flux Network Datasets

2021 ◽  
Vol 13 (9) ◽  
pp. 1716
Author(s):  
Ankur Srivastava ◽  
Jose F. Rodriguez ◽  
Patricia M. Saco ◽  
Nikul Kumari ◽  
Omer Yetemen
Keyword(s):  
Cloud Cover ◽  
Climatic Factors ◽  
Global Analysis ◽  
Empirical Relationship ◽  
Aridity Index ◽  
Global Scale ◽  
Empirical Models ◽  
Coefficient Of Determination ◽  
Crop Modeling ◽  
Atmospheric Transmissivity

Atmospheric transmissivity (τ) is a critical factor in climatology, which affects surface energy balance, measured at a limited number of meteorological stations worldwide. With the limited availability of meteorological datasets in remote areas across different climatic regions, estimation of τ is becoming a challenging task for adequate hydrological, climatic, and crop modeling studies. The availability of solar radiation data is comparatively less accessible on a global scale than the temperature and precipitation datasets, which makes it necessary to develop methods to estimate τ. Most of the previous studies provided region specific datasets of τ, which usually provide local assessments. Hence, there is a necessity to give the empirical models for τ estimation on a global scale that can be easily assessed. This study presents the analysis of the τ relationship with varying geographic features and climatic factors like latitude, aridity index, cloud cover, precipitation, temperature, diurnal temperature range, and elevation. In addition to these factors, the applicability of these relationships was evaluated for different climate types. Thus, empirical models have been proposed for each climate type to estimate τ by using the most effective factors such as cloud cover and aridity index. The cloud cover is an important yet often overlooked factor that can be used to determine the global atmospheric transmissivity. The empirical relationship and statistical indicator provided the best performance in equatorial climates as the coefficient of determination (r2) was 0.88 relatively higher than the warm temperate (r2 = 0.74) and arid regions (r2 = 0.46). According to the results, it is believed that the analysis presented in this work is applicable for estimating the τ in different ecosystems across the globe.

Sign in / Sign up

Export Citation Format

Share Document