ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth System Models in CMIP

Abstract. A community diagnostics and performance metrics tool for the evaluation of Earth System Models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected Essential Climate Variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases and soil hydrology-climate interactions, as well as atmospheric CO2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the CMIP ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within the Coupled Model Intercomparison Project (CMIP) and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

Download Full-text

ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP

Geoscientific Model Development ◽

10.5194/gmd-9-1747-2016 ◽

2016 ◽

Vol 9 (5) ◽

pp. 1747-1802 ◽

Cited By ~ 61

Author(s):

Veronika Eyring ◽

Mattia Righi ◽

Axel Lauer ◽

Martin Evaldsson ◽

Sabrina Wenzel ◽

...

Keyword(s):

Performance Metrics ◽

Stratospheric Ozone ◽

Coupled Model ◽

Earth System ◽

Evaluation Tool ◽

System Models ◽

The Earth ◽

Systematic Biases ◽

And Performance ◽

Earth System Models

Abstract. A community diagnostics and performance metrics tool for the evaluation of Earth system models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected essential climate variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases, and soil hydrology–climate interactions, as well as atmospheric CO2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the Coupled Model Intercomparison Project (CMIP) ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within CMIP and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

Download Full-text

ESMValTool v2.0 – Extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

10.5194/gmd-2019-291 ◽

2019 ◽

Cited By ~ 4

Author(s):

Veronika Eyring ◽

Lisa Bock ◽

Axel Lauer ◽

Mattia Righi ◽

Manuel Schlund ◽

...

Keyword(s):

Large Scale ◽

Performance Metrics ◽

Rapid Development ◽

Earth System ◽

Evaluation Tool ◽

Intergovernmental Panel ◽

System Models ◽

The Earth ◽

And Performance ◽

Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth System Models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of an easy-to-install, well documented Python package providing the core functionalities (ESMValCore) that performs common pre-processing operations and a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top-down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klima Rechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5.

Download Full-text

Earth System Model Evaluation Tool (ESMValTool) v2.0 – an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

Geoscientific Model Development ◽

10.5194/gmd-13-3383-2020 ◽

2020 ◽

Vol 13 (7) ◽

pp. 3383-3438 ◽

Cited By ~ 10

Author(s):

Veronika Eyring ◽

Lisa Bock ◽

Axel Lauer ◽

Mattia Righi ◽

Manuel Schlund ◽

...

Keyword(s):

Model Evaluation ◽

Large Scale ◽

Performance Metrics ◽

Earth System Model ◽

System Model ◽

Earth System ◽

Evaluation Tool ◽

The Earth ◽

And Performance ◽

Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth system models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of (1) an easy-to-install, well-documented Python package providing the core functionalities (ESMValCore) that performs common preprocessing operations and (2) a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability, the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top–down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klimarechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5.

Download Full-text

ESMValCore: analyzing CMIP data made easy

10.5194/egusphere-egu2020-17472 ◽

2020 ◽

Author(s):

Bouwe Andela ◽

Lisa Bock ◽

Björn Brötz ◽

Faruk Diblen ◽

Laura Dreyer ◽

...

Keyword(s):

Open Source ◽

Performance Metrics ◽

Modular Design ◽

Coupled Model ◽

Application Programming Interface ◽

Earth System ◽

Evaluation Tool ◽

System Models ◽

Scientific Analysis ◽

Earth System Models

The Earth System Model Evaluation Tool (ESMValTool) is a free and open-source community diagnostic and performance metrics tool for the evaluation of Earth system models participating in the Coupled Model Intercomparison Project (CMIP). Version 2 of the tool (Righi et al. 2019, www.esmvaltool.org) features a brand new design, consisting of ESMValCore (https://github.com/esmvalgroup/esmvalcore), a package for working with CMIP data and ESMValTool (https://github.com/esmvalgroup/esmvaltool), a package containing the scientific analysis scripts. This new version has been specifically developed to handle the increased data volume of CMIP Phase 6 (CMIP6) and the related challenges posed by the analysis and the evaluation of output from multiple high-resolution or complex Earth system models. The tool also supports CMIP5 and CMIP3 datasets, as well as a large number of re-analysis and observational datasets that can be formatted according to the same standards (CMOR) on-the-fly or through scripts currently included in the ESMValTool package.At the heart of this new version is the ESMValCore software package, which provides a configurable framework for finding CMIP files using a &#8220;data reference syntax&#8221;, applying commonly used pre-processing functions to them, running analysis scripts, and recording provenance. Numerous pre-processing functions, e.g. for data selection, regridding, and statistics are readily available and the modular design makes it easy to add more. The ESMValCore package is easy to install with relatively few dependencies, written in Python 3, based on state-of-the-art open-source libraries such as Iris and Dask, and widely used standards such as YAML, NetCDF, CF-Conventions, and W3C PROV. An extensive set of automated tests and code quality checks ensure the reliability of the package. Documentation is available at https://esmvaltool.readthedocs.io.The ESMValCore package uses human-readable recipes to define which variables and datasets to use, how to pre-process that data, and what scientific analysis scripts to run. The package provides convenient interfaces, based on the YAML and NetCDF/CF-convention file formats, for running diagnostic scripts written in any programming language. Because the ESMValCore framework takes care of running the workflow defined in the recipe in parallel, most analyses run much faster, with no additional programming effort required from the authors of the analysis scripts. For example, benchmarks show a factor of 30 speedup with respect to version 1 of the tool for a representative recipe on a 24 core machine. A large collection of standard recipes and associated analysis scripts is available in the ESMValTool package for reproducing selected peer-reviewed analyses. The ESMValCore package can also be used with any other script that implements it&#8217;s easy to use interface. All pre-processing functions of the ESMValCore can also be used directly from any Python program. These features allow for use by a wide community of scientific users and developers with different levels of programming skills and experience.Future plans involve extending the public Python API (application programming interface) from just preprocessor functions to include all functionality, including finding the data and running diagnostic scripts. This would make ESMValCore suitable for interactive data exploration from a Jupyter Notebook.

Download Full-text

ESMValTool - introducing a powerful model evaluation tool

10.5194/egusphere-egu2020-19181 ◽

2020 ◽

Author(s):

Valeriu Predoi ◽

Bouwe Andela ◽

Lee De Mora ◽

Axel Lauer

Keyword(s):

Model Evaluation ◽

Performance Metrics ◽

Stratospheric Ozone ◽

Scientific Development ◽

Multiple Model ◽

Earth System ◽

Evaluation Tool ◽

Climate Interactions ◽

Systematic Biases ◽

And Performance

The Earth System Model eValuation Tool (ESMValTool) is a powerful community-driven diagnostics and performance metrics tool. It is used for the evaluation of Earth System Models (ESMs) and allows for routine comparisons of either multiple model versions or observational datasets. ESMValTool's design is highly modular and flexible so that additional analyses can easily be added; in fact, this is essential to encourage the community-based approach to its scientific development. A set of standardized recipes for each scientific topic reproduces specific diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. Scientific themes include selected Essential Climate Variables, a range of known systematic biases common to ESMs such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases and soil hydrology-climate interactions, as well as atmospheric CO3 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. We will outline the main functional characteristics of ESMValTool Version 2; we will also introduce the reader to the current set of diagnostics and the methods they can use to contribute to its development.

Download Full-text

Model evaluation expectations of European ESM communities: results from a survey

10.5194/egusphere-egu21-15681 ◽

2021 ◽

Author(s):

Jerome Servonnat ◽

Eric Guilyardi ◽

Zofia Stott ◽

Kim Serradell ◽

Axel Lauer ◽

...

Keyword(s):

Model Evaluation ◽

Common Ground ◽

Performance Metrics ◽

Coupled Model ◽

System Modelling ◽

Earth System ◽

Evaluation Tool ◽

Technical Standards ◽

Community Effort ◽

And Performance

Developing an Earth system model evaluation tool for a broad user community is a real challenge, as the potential users do not necessarily have the same needs or expectations. While many evaluation tasks across user communities include common steps, significant differences are also apparent, not least the investment by institutions and individuals in bespoke tools. A key question is whether there is sufficient common ground to pursue a community tool with broad appeal and application.We present the main results of a survey carried out by Assimila for the H2020 IS-ENES3 project to review the model evaluation needs of European Earth System Modelling communities. Interviewing approximately 30 participants among several European institutions, the survey targeted a broad range of users, including model developers, model users, evaluation data providers, and infrastructure providers. The output of the study provides an analysis of&#160; requirements focusing on key technical, standards, and governance aspects.The study used ESMValTool as a&#160; current benchmark in terms of European evaluation tools. It is a community diagnostics and performance metrics tool for the evaluation of Earth System Models that allows for comparison of single or multiple models, either against predecessor versions or against observations. The tool is being developed in such a way that additional analyses can be added. As a community effort open to both users and developers, it encourages open exchange of diagnostic source code and evaluation results. It is currently used in Coupled Model Intercomparison Projects as well as for the development and testing of &#8220;new&#8221; models.A key result of the survey is the widespread support for ESMValTool amongst users, developers, and even those who have taken or promote other approaches. The results of the survey identify priorities and opportunities in the further development of the ESMValTool to ensure long-term adoption of the tool by a broad community.

Download Full-text

Spatially resolved evaluation of Earth system models with satellite column-averaged CO2

10.5194/egusphere-egu21-11848 ◽

2021 ◽

Author(s):

Bettina K. Gier ◽

Michael Buchwitz ◽

Maximilian Reuter ◽

Peter M. Cox ◽

Pierre Friedlingstein ◽

...

Keyword(s):

Satellite Data ◽

Near Infrared ◽

Early Period ◽

Coupled Model ◽

Negative Trend ◽

Positive Trend ◽

Earth System ◽

Evaluation Tool ◽

System Models ◽

Earth System Models

Earth system models (ESMs) participating in the Coupled Model Intercomparison Project Phase 5 (CMIP5) showed large uncertainties in simulating atmospheric CO2 concentrations. We utilize the Earth System Model Evaluation Tool (ESMValTool) to evaluate emission-driven CMIP5 and CMIP6 simulations with satellite data of column-average CO2 mole fractions (XCO2). XCO2 time series show a large spread among the model ensembles both in CMIP5 and CMIP6. Using the satellite observations as reference, the CMIP6 models have a lower bias in the the multi-model mean than CMIP5, but the spread remains large. The satellite data are a combined data product covering the period 2003&#8211;2014 based on the Scanning Imaging Absorption Spectrometer for Atmospheric Chartography (SCIAMACHY)/Envisat (2003&#8211;2012) and Thermal And Near infrared Sensor for carbon Observation Fourier transform spectrometer/Greenhouse Gases Observing Satellite (TANSO-FTS/GOSAT) (2009&#8211;2014) instruments. While the combined satellite product shows a strong negative trend of decreasing seasonal cycle amplitude (SCA) with increasing XCO2 in the northern midlatitudes, both CMIP ensembles instead show a non-significant positive trend in the multi-model mean. The negative trend is reproduced by the models when sampling them as the observations, attributing it to sampling characteristics. Applying a mask of the mean data coverage of each satellite to the models, the SCA is higher for the SCIAMACHY/Envisat mask than when using the TANSO-FTS/GOSAT mask. This induces an artificial negative trend when using observational sampling over the full period, as SCIAMACHY/Envisat covers the early period until 2012, with TANSO-FTS/GOSAT measurements starting in 2009. Overall, the CMIP6 ensemble shows better agreement with the satellite data than the CMIP5 ensemble in all considered quantities (mean XCO2, growth rate, SCA and trend in SCA). This study shows that the availability of column-integral CO2 from satellite provides a promising new way to evaluate the performance of Earth system models on a global scale, complementing existing studies that are based on in situ measurements from single ground-based stations.

Download Full-text

Earth System Model Evaluation Tool (ESMValTool) v2.0 – technical overview

Geoscientific Model Development ◽

10.5194/gmd-13-1179-2020 ◽

2020 ◽

Vol 13 (3) ◽

pp. 1179-1199 ◽

Cited By ~ 12

Author(s):

Mattia Righi ◽

Bouwe Andela ◽

Veronika Eyring ◽

Axel Lauer ◽

Valeriu Predoi ◽

...

Keyword(s):

Model Evaluation ◽

Performance Metrics ◽

Coupled Model ◽

Earth System Model ◽

System Model ◽

Earth System ◽

Evaluation Tool ◽

Data Volume ◽

And Performance ◽

User Friendly

Abstract. This paper describes the second major release of the Earth System Model Evaluation Tool (ESMValTool), a community diagnostic and performance metrics tool for the evaluation of Earth system models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). Compared to version 1.0, released in 2016, ESMValTool version 2.0 (v2.0) features a brand new design, with an improved interface and a revised preprocessor. It also features a significantly enhanced diagnostic part that is described in three companion papers. The new version of ESMValTool has been specifically developed to target the increased data volume of CMIP Phase 6 (CMIP6) and the related challenges posed by the analysis and the evaluation of output from multiple high-resolution or complex ESMs. The new version takes advantage of state-of-the-art computational libraries and methods to deploy an efficient and user-friendly data processing. Common operations on the input data (such as regridding or computation of multi-model statistics) are centralized in a highly optimized preprocessor, which allows applying a series of preprocessing functions before diagnostics scripts are applied for in-depth scientific analysis of the model output. Performance tests conducted on a set of standard diagnostics show that the new version is faster than its predecessor by about a factor of 3. The performance can be further improved, up to a factor of more than 30, when the newly introduced task-based parallelization options are used, which enable the efficient exploitation of much larger computing infrastructures. ESMValTool v2.0 also includes a revised and simplified installation procedure, the setting of user-configurable options based on modern language formats, and high code quality standards following the best practices for software development.

Download Full-text

ESMValTool v2.0 – Technical overview

10.5194/gmd-2019-226 ◽

2019 ◽

Author(s):

Mattia Righi ◽

Bouwe Andela ◽

Veronika Eyring ◽

Axel Lauer ◽

Valeriu Predoi ◽

...

Keyword(s):

Performance Metrics ◽

Coupled Model ◽

Earth System ◽

Evaluation Tool ◽

Modern Language ◽

Output Performance ◽

Data Volume ◽

Intercomparison Project ◽

And Performance ◽

User Friendly

Abstract. This paper describes the second major release of the Earth System Model Evaluation Tool (ESMValTool), a community diagnostic and performance metrics tool for the evaluation of Earth System Models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). Compared to version 1.0, released in 2016, ESMValTool version 2.0 (v2.0) features a brand new design, with an improved interface and a revised preprocessor. It also features a significantly enhanced diagnostic part that is described in three companion papers. The new version of the ESMValTool has been specifically developed to target the increased data volume of CMIP Phase 6 (CMIP6) and the related challenges posed by the analysis and the evaluation of output from multiple high-resolution or complex ESMs. The new version takes advantage of state-of-the-art computational libraries and methods to deploy an efficient and user-friendly data processing. Common operations on the input data (such as regridding or computation of multi-model statistics) are centralized in a highly optimized preprocessor, which allows applying a series of preprocessing functions before diagnostics scripts are applied for in-depth scientific analysis of the model output. Performance tests conducted on a set of standard diagnostics show that the new version is faster than its predecessor by about a factor of three. The performance can be further improved, up to a factor of more than 30, when the newly-introduced task-based parallelization options are used, which enable the efficient exploitation of much larger computing infrastructures. ESMValTool v2.0 also includes a revised and simplified installation procedure, setting of user configurable options based on modern language formats, and high code quality standards following the best practices for software development.

Download Full-text

Recent developments on the Earth System Model Evaluation Tool

10.5194/egusphere-egu21-3476 ◽

2021 ◽

Author(s):

Bouwe Andela ◽

Fakhereh Alidoost ◽

Lukas Brunner ◽

Jaro Camphuijsen ◽

Bas Crezee ◽

...

Keyword(s):

Model Evaluation ◽

Performance Metrics ◽

Climate Model ◽

Earth System Model ◽

System Model ◽

Earth System ◽

Evaluation Tool ◽

Scientific Review ◽

The Earth ◽

User Friendly

The Earth System Model Evaluation Tool (ESMValTool) is a free and open-source community diagnostic and performance metrics tool for the evaluation of Earth system models such as those participating in the Coupled Model Intercomparison Project (CMIP). Version 2 of the tool (Righi et al. 2020, www.esmvaltool.org) features a brand new design composed of a core that finds and processes data according to a &#8216;recipe&#8217; and an extensive collection of ready-to-use recipes and associated diagnostic codes for reproducing results from published papers. Development and discussion of the tool (mostly) takes place in public on https://github.com/esmvalgroup and anyone with an interest in climate model evaluation is welcome to join there.&#160;Since the initial release of version 2 in the summer of 2020, many improvements have been made to the tool. It is now more user friendly with extensive documentation available on docs.esmvaltool.org and a step by step online tutorial. Regular releases, currently planned three times a year, ensure that recent contributions become available quickly while still ensuring a high level of quality control. The tool can be installed from conda, but portable docker and singularity containers are also available.&#160;Recent new features include a more user-friendly command-line interface, citation information per figure including CMIP6 data citation using ES-DOC, more and faster preprocessor functions that require less memory, automatic corrections for a larger number of CMIP6 datasets, support for more observational and reanalysis datasets, and more recipes and diagnostics.&#160;The tool is now also more reliable, with improved automated testing through more unit tests for the core, as well as a recipe testing service running at DKRZ for testing the scientific recipes and diagnostics that are bundled into the tool. The community maintaining and developing the tool is growing, making the project less dependent on individual contributors. There are now technical and scientific review teams that review new contributions for technical quality and scientific correctness and relevance respectively, two new principal investigators for generating a larger support base in the community, and a newly created user engagement team that is taking care of improving the overall user experience.

Download Full-text