scholarly journals Earth System Model Evaluation Tool (ESMValTool) v2.0 – an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

2020 ◽  
Vol 13 (7) ◽  
pp. 3383-3438 ◽  
Author(s):  
Veronika Eyring ◽  
Lisa Bock ◽  
Axel Lauer ◽  
Mattia Righi ◽  
Manuel Schlund ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Large Scale ◽  
Performance Metrics ◽  
Earth System Model ◽  
System Model ◽  
Earth System ◽  
Evaluation Tool ◽  
The Earth ◽  
And Performance ◽  
Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth system models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of (1) an easy-to-install, well-documented Python package providing the core functionalities (ESMValCore) that performs common preprocessing operations and (2) a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability, the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top–down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klimarechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5.

Recent developments on the Earth System Model Evaluation Tool

2021 ◽  
Author(s):  
Bouwe Andela ◽  
Fakhereh Alidoost ◽  
Lukas Brunner ◽  
Jaro Camphuijsen ◽  
Bas Crezee ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Performance Metrics ◽  
Climate Model ◽  
Earth System Model ◽  
System Model ◽  
Earth System ◽  
Evaluation Tool ◽  
Scientific Review ◽  
The Earth ◽  
User Friendly

<p>The Earth System Model Evaluation Tool (ESMValTool) is a free and open-source community diagnostic and performance metrics tool for the evaluation of Earth system models such as those participating in the Coupled Model Intercomparison Project (CMIP). Version 2 of the tool (Righi et al. 2020, www.esmvaltool.org) features a brand new design composed of a core that finds and processes data according to a ‘recipe’ and an extensive collection of ready-to-use recipes and associated diagnostic codes for reproducing results from published papers. Development and discussion of the tool (mostly) takes place in public on https://github.com/esmvalgroup and anyone with an interest in climate model evaluation is welcome to join there.</p><p> </p><p>Since the initial release of version 2 in the summer of 2020, many improvements have been made to the tool. It is now more user friendly with extensive documentation available on docs.esmvaltool.org and a step by step online tutorial. Regular releases, currently planned three times a year, ensure that recent contributions become available quickly while still ensuring a high level of quality control. The tool can be installed from conda, but portable docker and singularity containers are also available.</p><p> </p><p>Recent new features include a more user-friendly command-line interface, citation information per figure including CMIP6 data citation using ES-DOC, more and faster preprocessor functions that require less memory, automatic corrections for a larger number of CMIP6 datasets, support for more observational and reanalysis datasets, and more recipes and diagnostics.</p><p> </p><p>The tool is now also more reliable, with improved automated testing through more unit tests for the core, as well as a recipe testing service running at DKRZ for testing the scientific recipes and diagnostics that are bundled into the tool. The community maintaining and developing the tool is growing, making the project less dependent on individual contributors. There are now technical and scientific review teams that review new contributions for technical quality and scientific correctness and relevance respectively, two new principal investigators for generating a larger support base in the community, and a newly created user engagement team that is taking care of improving the overall user experience.</p>

scholarly journals Earth System Model Evaluation Tool (ESMValTool) v2.0 – technical overview

2020 ◽  
Vol 13 (3) ◽  
pp. 1179-1199 ◽  
Author(s):  
Mattia Righi ◽  
Bouwe Andela ◽  
Veronika Eyring ◽  
Axel Lauer ◽  
Valeriu Predoi ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Performance Metrics ◽  
Coupled Model ◽  
Earth System Model ◽  
System Model ◽  
Earth System ◽  
Evaluation Tool ◽  
Data Volume ◽  
And Performance ◽  
User Friendly

Abstract. This paper describes the second major release of the Earth System Model Evaluation Tool (ESMValTool), a community diagnostic and performance metrics tool for the evaluation of Earth system models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). Compared to version 1.0, released in 2016, ESMValTool version 2.0 (v2.0) features a brand new design, with an improved interface and a revised preprocessor. It also features a significantly enhanced diagnostic part that is described in three companion papers. The new version of ESMValTool has been specifically developed to target the increased data volume of CMIP Phase 6 (CMIP6) and the related challenges posed by the analysis and the evaluation of output from multiple high-resolution or complex ESMs. The new version takes advantage of state-of-the-art computational libraries and methods to deploy an efficient and user-friendly data processing. Common operations on the input data (such as regridding or computation of multi-model statistics) are centralized in a highly optimized preprocessor, which allows applying a series of preprocessing functions before diagnostics scripts are applied for in-depth scientific analysis of the model output. Performance tests conducted on a set of standard diagnostics show that the new version is faster than its predecessor by about a factor of 3. The performance can be further improved, up to a factor of more than 30, when the newly introduced task-based parallelization options are used, which enable the efficient exploitation of much larger computing infrastructures. ESMValTool v2.0 also includes a revised and simplified installation procedure, the setting of user-configurable options based on modern language formats, and high code quality standards following the best practices for software development.

scholarly journals ESMValTool v2.0 – Extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

2019 ◽  
Author(s):  
Veronika Eyring ◽  
Lisa Bock ◽  
Axel Lauer ◽  
Mattia Righi ◽  
Manuel Schlund ◽  
...  
Keyword(s):  
Large Scale ◽  
Performance Metrics ◽  
Rapid Development ◽  
Earth System ◽  
Evaluation Tool ◽  
Intergovernmental Panel ◽  
System Models ◽  
The Earth ◽  
And Performance ◽  
Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth System Models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of an easy-to-install, well documented Python package providing the core functionalities (ESMValCore) that performs common pre-processing operations and a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top-down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klima Rechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5.

New scientific diagnostics in the ESMValTool – an overview

2021 ◽  
Author(s):  
Lisa Bock ◽  
Birgit Hassler ◽  
Axel Lauer ◽  
Keyword(s):  
Model Evaluation ◽  
Performance Metrics ◽  
System Model ◽  
Data Sets ◽  
Model Data ◽  
Evaluation Tool ◽  
The Earth ◽  
Community Effort ◽  
The Mean ◽  
And Performance

<p>The Earth System Model Evaluation Tool (ESMValTool) has been developed with the aim of taking model evaluation to the next level by facilitating analysis of many different ESM components, providing well-documented source code and scientific background of implemented diagnostics and metrics and allowing for traceability and reproducibility of results (provenance). This has been made possible by a lively and growing development community continuously improving the tool supported by multiple national and European projects. The latest major release (v2.0) of the ESMValTool has been officially introduced in August 2020 as a large community effort, and since then several additional smaller releases have followed.</p><p>The diagnostic part of the ESMValTool includes a large collection of standard “recipes” for reproducing peer-reviewed analyses of many variables across ESM compartments including atmosphere, ocean, and land domains, with diagnostics and performance metrics focusing on the mean-state, trends, variability and important processes, phenomena, as well as emergent constraints. While most of the diagnostics use observational data sets (in particular satellite and ground-based observations) or reanalysis products for model evaluation some are also based on model-to-model comparisons. This presentation gives an overview on the latest scientific diagnostics and metrics added during the last year including examples of applications of these diagnostics to CMIP6 model data.</p>

scholarly journals ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth System Models in CMIP

2015 ◽  
Vol 8 (9) ◽  
pp. 7541-7661 ◽  
Author(s):  
V. Eyring ◽  
M. Righi ◽  
M. Evaldsson ◽  
A. Lauer ◽  
S. Wenzel ◽  
...  
Keyword(s):  
Performance Metrics ◽  
Stratospheric Ozone ◽  
Coupled Model ◽  
Earth System ◽  
Evaluation Tool ◽  
System Models ◽  
The Earth ◽  
Systematic Biases ◽  
And Performance ◽  
Earth System Models

Abstract. A community diagnostics and performance metrics tool for the evaluation of Earth System Models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected Essential Climate Variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases and soil hydrology-climate interactions, as well as atmospheric CO2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the CMIP ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within the Coupled Model Intercomparison Project (CMIP) and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

scholarly journals Earth System Model Evaluation Tool (ESMValTool) v2.0 – diagnostics for emergent constraints and future projections from Earth system models in CMIP

2020 ◽  
Vol 13 (9) ◽  
pp. 4205-4228 ◽  
Author(s):  
Axel Lauer ◽  
Veronika Eyring ◽  
Omar Bellprat ◽  
Lisa Bock ◽  
Bettina K. Gier ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Arctic Sea Ice ◽  
Earth System Model ◽  
System Model ◽  
Earth System ◽  
Evaluation Tool ◽  
Future Projections ◽  
System Models ◽  
Emergent Constraints ◽  
Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool), a community diagnostics and performance metrics tool for evaluation and analysis of Earth system models (ESMs), is designed to facilitate a more comprehensive and rapid comparison of single or multiple models participating in the Coupled Model Intercomparison Project (CMIP). The ESM results can be compared against observations or reanalysis data as well as against other models including predecessor versions of the same model. The updated and extended version (v2.0) of the ESMValTool includes several new analysis scripts such as large-scale diagnostics for evaluation of ESMs as well as diagnostics for extreme events, regional model and impact evaluation. In this paper, the newly implemented climate metrics such as effective climate sensitivity (ECS) and transient climate response (TCR) as well as emergent constraints for various climate-relevant feedbacks and diagnostics for future projections from ESMs are described and illustrated with examples using results from the well-established model ensemble CMIP5. The emergent constraints implemented include constraints on ECS, snow-albedo effect, climate–carbon cycle feedback, hydrologic cycle intensification, future Indian summer monsoon precipitation and year of disappearance of summer Arctic sea ice. The diagnostics included in ESMValTool v2.0 to analyze future climate projections from ESMs further include analysis scripts to reproduce selected figures of chapter 12 of the Intergovernmental Panel on Climate Change's (IPCC) Fifth Assessment Report (AR5) and various multi-model statistics.

scholarly journals Earth System Model Evaluation Tool (ESMValTool) v2.0 – diagnostics for emergent constraints and future projections from Earth system models in CMIP

2020 ◽  
Author(s):  
Axel Lauer ◽  
Veronika Eyring ◽  
Omar Bellprat ◽  
Lisa Bock ◽  
Bettina K. Gier ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Arctic Sea Ice ◽  
Earth System Model ◽  
System Model ◽  
Earth System ◽  
Evaluation Tool ◽  
Future Projections ◽  
System Models ◽  
Emergent Constraints ◽  
Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool), a community diagnostics and performance metrics tool for evaluation and analysis of Earth system models (ESMs) is designed to facilitate a more comprehensive and rapid comparison of single or multiple models participating in the coupled model intercomparison project (CMIP). The ESM results can be compared against observations or reanalysis data as well as against other models including predecessor versions of the same model. The updated and extended version 2.0 of the ESMValTool includes several new analysis scripts such as large-scale diagnostics for evaluation of ESMs as well as diagnostics for extreme events, regional model and impact evaluation. In this paper, the newly implemented climate metrics such as effective climate sensitivity (ECS) and transient climate response (TCR) as well as emergent constraints for various climate-relevant feedbacks and diagnostics for future projections from ESMs are described and illustrated with examples using results from the well-established model ensemble CMIP5. The emergent constraints implemented include ECS, snow-albedo effect, climate-carbon cycle feedback, hydrologic cycle intensification, future Indian summer monsoon precipitation, and year of disappearance of summer Arctic sea ice. The diagnostics included in ESMValTool v2.0 to analyze future climate projections from ESMs include analysis scripts to reproduce selected figures of chapter 12 of the Intergovernmental Panel on Climate Change’s (IPCC) Fifth Assessment report (AR5) and various multi-model statistics.

scholarly journals Earth System Model Evaluation Tool (ESMValTool) v2.0 – diagnostics for extreme events, regional and impact evaluation and analysis of Earth system models in CMIP

2020 ◽  
Author(s):  
Katja Weigel ◽  
Lisa Bock ◽  
Bettina K. Gier ◽  
Axel Lauer ◽  
Mattia Righi ◽  
...  
Keyword(s):  
Climate Change ◽  
Impact Assessment ◽  
Extreme Events ◽  
Model Evaluation ◽  
Hydrological Cycle ◽  
Earth System Model ◽  
System Model ◽  
Earth System ◽  
Evaluation Tool ◽  
Earth System Models

Abstract. This paper complements a series of now four publications that document the release of the Earth System Model Evaluation Tool (ESMValTool) v2.0. It describes new diagnostics on the hydrological cycle, extreme events, impact assessment, regional evaluations, and ensemble member selection. The diagnostics are developed by a large community of scientists aiming to facilitate the evaluation and comparison of Earth System Models (ESMs) which are participating in the Coupled Model Intercomparison Project (CMIP). The second release of this tool aims to support the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). Furthermore, data sets from other models and observations can be analysed. The diagnostics for the hydrological cycle include several precipitation and drought indices, as well as hydroclimatic intensity and indices from the Expert Team on Climate Change Detection and Indices (ETCCDI). The latter are also used for identification of extreme events and for impact assessment, and to project and characterize the risks and impacts of climate change for natural and socio-economic systems. Further impact assessment diagnostics are included to compute daily temperature ranges and capacity factors for wind and solar energy generation. Regional scales can be analysed with new diagnostics implemented for selected regions and stochastic downscaling. ESMValTool v2.0 also includes diagnostics to analyse large multi-model ensembles including grouping and selecting ensemble members by user specified criteria. Here, we present examples for their capabilities based on the well-established CMIP Phase 5 (CMIP5) data set.

scholarly journals Earth System Model Evaluation Tool (ESMValTool) v2.0 – diagnostics for extreme events, regional and impact evaluation, and analysis of Earth system models in CMIP

2021 ◽  
Vol 14 (6) ◽  
pp. 3159-3184
Author(s):  
Katja Weigel ◽  
Lisa Bock ◽  
Bettina K. Gier ◽  
Axel Lauer ◽  
Mattia Righi ◽  
...  
Keyword(s):  
Climate Change ◽  
Impact Assessment ◽  
Extreme Events ◽  
Model Evaluation ◽  
Hydrological Cycle ◽  
Earth System Model ◽  
System Model ◽  
Earth System ◽  
Evaluation Tool ◽  
Earth System Models

Abstract. This paper complements a series of now four publications that document the release of the Earth System Model Evaluation Tool (ESMValTool) v2.0. It describes new diagnostics on the hydrological cycle, extreme events, impact assessment, regional evaluations, and ensemble member selection. The diagnostics are developed by a large community of scientists aiming to facilitate the evaluation and comparison of Earth system models (ESMs) which are participating in the Coupled Model Intercomparison Project (CMIP). The second release of this tool aims to support the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). Furthermore, datasets from other models and observations can be analysed. The diagnostics for the hydrological cycle include several precipitation and drought indices, as well as hydroclimatic intensity and indices from the Expert Team on Climate Change Detection and Indices (ETCCDI). The latter are also used for identification of extreme events, for impact assessment, and to project and characterize the risks and impacts of climate change for natural and socio-economic systems. Further impact assessment diagnostics are included to compute daily temperature ranges and capacity factors for wind and solar energy generation. Regional scales can be analysed with new diagnostics implemented for selected regions and stochastic downscaling. ESMValTool v2.0 also includes diagnostics to analyse large multi-model ensembles including grouping and selecting ensemble members by user-specified criteria. Here, we present examples for their capabilities based on the well-established CMIP Phase 5 (CMIP5) dataset.

Sign in / Sign up

Export Citation Format

Share Document