ESMValCore: analyzing CMIP data made easy

The Earth System Model Evaluation Tool (ESMValTool) is a free and open-source community diagnostic and performance metrics tool for the evaluation of Earth system models participating in the Coupled Model Intercomparison Project (CMIP). Version 2 of the tool (Righi et al. 2019, www.esmvaltool.org) features a brand new design, consisting of ESMValCore (https://github.com/esmvalgroup/esmvalcore), a package for working with CMIP data and ESMValTool (https://github.com/esmvalgroup/esmvaltool), a package containing the scientific analysis scripts. This new version has been specifically developed to handle the increased data volume of CMIP Phase 6 (CMIP6) and the related challenges posed by the analysis and the evaluation of output from multiple high-resolution or complex Earth system models. The tool also supports CMIP5 and CMIP3 datasets, as well as a large number of re-analysis and observational datasets that can be formatted according to the same standards (CMOR) on-the-fly or through scripts currently included in the ESMValTool package.At the heart of this new version is the ESMValCore software package, which provides a configurable framework for finding CMIP files using a &#8220;data reference syntax&#8221;, applying commonly used pre-processing functions to them, running analysis scripts, and recording provenance. Numerous pre-processing functions, e.g. for data selection, regridding, and statistics are readily available and the modular design makes it easy to add more. The ESMValCore package is easy to install with relatively few dependencies, written in Python 3, based on state-of-the-art open-source libraries such as Iris and Dask, and widely used standards such as YAML, NetCDF, CF-Conventions, and W3C PROV. An extensive set of automated tests and code quality checks ensure the reliability of the package. Documentation is available at https://esmvaltool.readthedocs.io.The ESMValCore package uses human-readable recipes to define which variables and datasets to use, how to pre-process that data, and what scientific analysis scripts to run. The package provides convenient interfaces, based on the YAML and NetCDF/CF-convention file formats, for running diagnostic scripts written in any programming language. Because the ESMValCore framework takes care of running the workflow defined in the recipe in parallel, most analyses run much faster, with no additional programming effort required from the authors of the analysis scripts. For example, benchmarks show a factor of 30 speedup with respect to version 1 of the tool for a representative recipe on a 24 core machine. A large collection of standard recipes and associated analysis scripts is available in the ESMValTool package for reproducing selected peer-reviewed analyses. The ESMValCore package can also be used with any other script that implements it&#8217;s easy to use interface. All pre-processing functions of the ESMValCore can also be used directly from any Python program. These features allow for use by a wide community of scientific users and developers with different levels of programming skills and experience.Future plans involve extending the public Python API (application programming interface) from just preprocessor functions to include all functionality, including finding the data and running diagnostic scripts. This would make ESMValCore suitable for interactive data exploration from a Jupyter Notebook.

Download Full-text

ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth System Models in CMIP

Geoscientific Model Development Discussions ◽

10.5194/gmdd-8-7541-2015 ◽

2015 ◽

Vol 8 (9) ◽

pp. 7541-7661 ◽

Cited By ~ 8

Author(s):

V. Eyring ◽

M. Righi ◽

M. Evaldsson ◽

A. Lauer ◽

S. Wenzel ◽

...

Keyword(s):

Performance Metrics ◽

Stratospheric Ozone ◽

Coupled Model ◽

Earth System ◽

Evaluation Tool ◽

System Models ◽

The Earth ◽

Systematic Biases ◽

And Performance ◽

Earth System Models

Abstract. A community diagnostics and performance metrics tool for the evaluation of Earth System Models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected Essential Climate Variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases and soil hydrology-climate interactions, as well as atmospheric CO2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the CMIP ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within the Coupled Model Intercomparison Project (CMIP) and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

Download Full-text

ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP

Geoscientific Model Development ◽

10.5194/gmd-9-1747-2016 ◽

2016 ◽

Vol 9 (5) ◽

pp. 1747-1802 ◽

Cited By ~ 61

Author(s):

Veronika Eyring ◽

Mattia Righi ◽

Axel Lauer ◽

Martin Evaldsson ◽

Sabrina Wenzel ◽

...

Keyword(s):

Performance Metrics ◽

Stratospheric Ozone ◽

Coupled Model ◽

Earth System ◽

Evaluation Tool ◽

System Models ◽

The Earth ◽

Systematic Biases ◽

And Performance ◽

Earth System Models

Abstract. A community diagnostics and performance metrics tool for the evaluation of Earth system models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected essential climate variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases, and soil hydrology–climate interactions, as well as atmospheric CO2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the Coupled Model Intercomparison Project (CMIP) ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within CMIP and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

Download Full-text

Spatially resolved evaluation of Earth system models with satellite column-averaged CO2

10.5194/egusphere-egu21-11848 ◽

2021 ◽

Author(s):

Bettina K. Gier ◽

Michael Buchwitz ◽

Maximilian Reuter ◽

Peter M. Cox ◽

Pierre Friedlingstein ◽

...

Keyword(s):

Satellite Data ◽

Near Infrared ◽

Early Period ◽

Coupled Model ◽

Negative Trend ◽

Positive Trend ◽

Earth System ◽

Evaluation Tool ◽

System Models ◽

Earth System Models

Earth system models (ESMs) participating in the Coupled Model Intercomparison Project Phase 5 (CMIP5) showed large uncertainties in simulating atmospheric CO2 concentrations. We utilize the Earth System Model Evaluation Tool (ESMValTool) to evaluate emission-driven CMIP5 and CMIP6 simulations with satellite data of column-average CO2 mole fractions (XCO2). XCO2 time series show a large spread among the model ensembles both in CMIP5 and CMIP6. Using the satellite observations as reference, the CMIP6 models have a lower bias in the the multi-model mean than CMIP5, but the spread remains large. The satellite data are a combined data product covering the period 2003&#8211;2014 based on the Scanning Imaging Absorption Spectrometer for Atmospheric Chartography (SCIAMACHY)/Envisat (2003&#8211;2012) and Thermal And Near infrared Sensor for carbon Observation Fourier transform spectrometer/Greenhouse Gases Observing Satellite (TANSO-FTS/GOSAT) (2009&#8211;2014) instruments. While the combined satellite product shows a strong negative trend of decreasing seasonal cycle amplitude (SCA) with increasing XCO2 in the northern midlatitudes, both CMIP ensembles instead show a non-significant positive trend in the multi-model mean. The negative trend is reproduced by the models when sampling them as the observations, attributing it to sampling characteristics. Applying a mask of the mean data coverage of each satellite to the models, the SCA is higher for the SCIAMACHY/Envisat mask than when using the TANSO-FTS/GOSAT mask. This induces an artificial negative trend when using observational sampling over the full period, as SCIAMACHY/Envisat covers the early period until 2012, with TANSO-FTS/GOSAT measurements starting in 2009. Overall, the CMIP6 ensemble shows better agreement with the satellite data than the CMIP5 ensemble in all considered quantities (mean XCO2, growth rate, SCA and trend in SCA). This study shows that the availability of column-integral CO2 from satellite provides a promising new way to evaluate the performance of Earth system models on a global scale, complementing existing studies that are based on in situ measurements from single ground-based stations.

Download Full-text

ESMValTool v2.0 – Extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

10.5194/gmd-2019-291 ◽

2019 ◽

Cited By ~ 4

Author(s):

Veronika Eyring ◽

Lisa Bock ◽

Axel Lauer ◽

Mattia Righi ◽

Manuel Schlund ◽

...

Keyword(s):

Large Scale ◽

Performance Metrics ◽

Rapid Development ◽

Earth System ◽

Evaluation Tool ◽

Intergovernmental Panel ◽

System Models ◽

The Earth ◽

And Performance ◽

Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth System Models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of an easy-to-install, well documented Python package providing the core functionalities (ESMValCore) that performs common pre-processing operations and a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top-down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klima Rechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5.

Download Full-text

Earth System Model Evaluation Tool (ESMValTool) v2.0 – an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

Geoscientific Model Development ◽

10.5194/gmd-13-3383-2020 ◽

2020 ◽

Vol 13 (7) ◽

pp. 3383-3438 ◽

Cited By ~ 10

Author(s):

Veronika Eyring ◽

Lisa Bock ◽

Axel Lauer ◽

Mattia Righi ◽

Manuel Schlund ◽

...

Keyword(s):

Model Evaluation ◽

Large Scale ◽

Performance Metrics ◽

Earth System Model ◽

System Model ◽

Earth System ◽

Evaluation Tool ◽

The Earth ◽

And Performance ◽

Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth system models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of (1) an easy-to-install, well-documented Python package providing the core functionalities (ESMValCore) that performs common preprocessing operations and (2) a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability, the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top–down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klimarechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5.

Download Full-text

Disentangeling the shortwave surface energy balance in earth system models

10.5194/egusphere-egu21-15200 ◽

2021 ◽

Author(s):

Kine Onsum Moseid

Keyword(s):

Energy Balance ◽

Surface Energy ◽

Atmospheric Aerosols ◽

Surface Energy Balance ◽

Coupled Model ◽

Shortwave Radiation ◽

Earth System ◽

System Models ◽

Aerosol Forcing ◽

Earth System Models

The Earth&#8217;s surface energy balance is heavily affected by incoming solar radiation and how it propagates through our atmosphere. How the sunlight propagates towards the surface depends on the atmospheric presence of aerosols, gases, and clouds.&#160;Surface temperature evolution according to earth system models (ESMs) in the historical experiment from the coupled model intercomparison project phase 6 (CMIP6) suggests that models may be overly sensitive to aerosol forcing. Other studies have found that ESMs do not recreate observed decadal patterns in surface shortwave radiation - suggesting the models inaccurately underestimate the shortwave impact of atmospheric aerosols. These contradictory results act as a basis for our study. Our study decomposes what determines both all sky and clear sky downwelling shortwave radiation at the surface in ESMs, using different experiments of CMIP6. We try to determine the respective role of aerosols, clouds and gases in the shortwave energy balance at the surface, and assess the effect of seasonality and regional differences.

Download Full-text

Are Earth system models able to reproduce the soil heterotrophic respiration fluxes?

10.5194/ismc2021-23 ◽

2021 ◽

Author(s):

Bertrand Guenet ◽

Jérémie Orliac ◽

Lauric Cécillon ◽

Olivier Torres ◽

Laurent Bopp

Keyword(s):

Soil Respiration ◽

Coupled Model ◽

Global Scale ◽

Heterotrophic Respiration ◽

Earth System ◽

Mixed Effect ◽

Microbial Decomposition ◽

System Models ◽

Earth System Models ◽

Climate Dynamic

Earth system models (ESMs) are numerical representations of the Earth system aiming at representing the climate dynamic including feedbacks between climate and carbon cycle. CO2 flux due to soil respiration including heterotrophic respiration coming from the soil organic matter (SOM) microbial decomposition and autotrophic respiration coming from the roots respiration is one of the most important flux between the surface and the atmosphere. Thus, even small changes in this flux may impact drastically the climate dynamic. It is therefore essential that ESMs reliably reproduce soil respiration. Until recently, such an evaluation at global scale of the ESMs was not straightforward because of the absence of observation-derived product to evaluate heterotrophic respiration fluxes from ESMs at global scale. Recently, several gridded products were published opening a new research avenue on climate-carbon feedbacks. In this study, we used simulations from 13 ESMs performed within the sixth coupled model intercomparison project (CMIP6) and we evaluate their capacities to reproduce the heterotrophic respiration flux using three gridded observation-based products. We first evaluate the total heterotrophic respiration flux for each model as well as the spatial patterns. We observed that most of the models are able to reproduce the total heterotrophic respiration flux but the spatial analysis underlined that this was partially due to some bias compensation between regions overestimating the flux and regions underestimating the flux. To better identify the causes of the identified bias in predicting the total heterotrophic respiration flux, we analysed the residues of ESMs using linear mixed effect models and we observed that lithology and climate were the most important drivers of the ESMs residues. Our results suggest that the response of SOM microbial decomposition to soil moisture and temperature must be improved in the next ESMs generation and that the effect of lithology should be better taken into account.

Download Full-text

Tracking Improvement in Simulated Marine Biogeochemistry Between CMIP5 and CMIP6

Current Climate Change Reports ◽

10.1007/s40641-020-00160-0 ◽

2020 ◽

Cited By ~ 4

Author(s):

Roland Séférian ◽

Sarah Berthet ◽

Andrew Yool ◽

Julien Palmiéri ◽

Laurent Bopp ◽

...

Keyword(s):

Coupled Model ◽

Climate Feedback ◽

Full Potential ◽

Earth System ◽

Current Generation ◽

System Models ◽

Biogeochemical Models ◽

Marine Biogeochemistry ◽

Intercomparison Project ◽

Earth System Models

Abstract Purpose of Review The changes or updates in ocean biogeochemistry component have been mapped between CMIP5 and CMIP6 model versions, and an assessment made of how far these have led to improvements in the simulated mean state of marine biogeochemical models within the current generation of Earth system models (ESMs). Recent Findings The representation of marine biogeochemistry has progressed within the current generation of Earth system models. However, it remains difficult to identify which model updates are responsible for a given improvement. In addition, the full potential of marine biogeochemistry in terms of Earth system interactions and climate feedback remains poorly examined in the current generation of Earth system models. Summary Increasing availability of ocean biogeochemical data, as well as an improved understanding of the underlying processes, allows advances in the marine biogeochemical components of the current generation of ESMs. The present study scrutinizes the extent to which marine biogeochemistry components of ESMs have progressed between the 5th and the 6th phases of the Coupled Model Intercomparison Project (CMIP).

Download Full-text

Plant Physiology Increases the Magnitude and Spread of the Transient Climate Response to CO2 in CMIP6 Earth System Models

Journal of Climate ◽

10.1175/jcli-d-20-0078.1 ◽

2020 ◽

Vol 33 (19) ◽

pp. 8561-8578

Author(s):

Claire M. Zarakas ◽

Abigail L. S. Swann ◽

Marysa M. Laguë ◽

Kyle C. Armour ◽

James T. Randerson

Keyword(s):

Plant Physiology ◽

Coupled Model ◽

Surface Air Temperature ◽

Physiological Effect ◽

Climate Response ◽

Earth System ◽

Transient Climate Response ◽

Near Surface ◽

System Models ◽

Earth System Models

AbstractIncreasing concentrations of CO2 in the atmosphere influence climate both through CO2’s role as a greenhouse gas and through its impact on plants. Plants respond to atmospheric CO2 concentrations in several ways that can alter surface energy and water fluxes and thus surface climate, including changes in stomatal conductance, water use, and canopy leaf area. These plant physiological responses are already embedded in most Earth system models, and a robust literature demonstrates that they can affect global-scale temperature. However, the physiological contribution to transient warming has yet to be assessed systematically in Earth system models. Here this gap is addressed using carbon cycle simulations from phases 5 and 6 of the Coupled Model Intercomparison Project (CMIP) to isolate the radiative and physiological contributions to the transient climate response (TCR), which is defined as the change in globally averaged near-surface air temperature during the 20-yr window centered on the time of CO2 doubling relative to preindustrial CO2 concentrations. In CMIP6 models, the physiological effect contributes 0.12°C (σ: 0.09°C; range: 0.02°–0.29°C) of warming to the TCR, corresponding to 6.1% of the full TCR (σ: 3.8%; range: 1.4%–13.9%). Moreover, variation in the physiological contribution to the TCR across models contributes disproportionately more to the intermodel spread of TCR estimates than it does to the mean. The largest contribution of plant physiology to CO2-forced warming—and the intermodel spread in warming—occurs over land, especially in forested regions.

Download Full-text

Model evaluation expectations of European ESM communities: results from a survey

10.5194/egusphere-egu21-15681 ◽

2021 ◽

Author(s):

Jerome Servonnat ◽

Eric Guilyardi ◽

Zofia Stott ◽

Kim Serradell ◽

Axel Lauer ◽

...

Keyword(s):

Model Evaluation ◽

Common Ground ◽

Performance Metrics ◽

Coupled Model ◽

System Modelling ◽

Earth System ◽

Evaluation Tool ◽

Technical Standards ◽

Community Effort ◽

And Performance

Developing an Earth system model evaluation tool for a broad user community is a real challenge, as the potential users do not necessarily have the same needs or expectations. While many evaluation tasks across user communities include common steps, significant differences are also apparent, not least the investment by institutions and individuals in bespoke tools. A key question is whether there is sufficient common ground to pursue a community tool with broad appeal and application.We present the main results of a survey carried out by Assimila for the H2020 IS-ENES3 project to review the model evaluation needs of European Earth System Modelling communities. Interviewing approximately 30 participants among several European institutions, the survey targeted a broad range of users, including model developers, model users, evaluation data providers, and infrastructure providers. The output of the study provides an analysis of&#160; requirements focusing on key technical, standards, and governance aspects.The study used ESMValTool as a&#160; current benchmark in terms of European evaluation tools. It is a community diagnostics and performance metrics tool for the evaluation of Earth System Models that allows for comparison of single or multiple models, either against predecessor versions or against observations. The tool is being developed in such a way that additional analyses can be added. As a community effort open to both users and developers, it encourages open exchange of diagnostic source code and evaluation results. It is currently used in Coupled Model Intercomparison Projects as well as for the development and testing of &#8220;new&#8221; models.A key result of the survey is the widespread support for ESMValTool amongst users, developers, and even those who have taken or promote other approaches. The results of the survey identify priorities and opportunities in the further development of the ESMValTool to ensure long-term adoption of the tool by a broad community.

Download Full-text