scholarly journals ESMValTool v2.0 – Extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

2019 ◽  
Author(s):  
Veronika Eyring ◽  
Lisa Bock ◽  
Axel Lauer ◽  
Mattia Righi ◽  
Manuel Schlund ◽  
...  
Keyword(s):  
Large Scale ◽  
Performance Metrics ◽  
Rapid Development ◽  
Earth System ◽  
Evaluation Tool ◽  
Intergovernmental Panel ◽  
System Models ◽  
The Earth ◽  
And Performance ◽  
Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth System Models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of an easy-to-install, well documented Python package providing the core functionalities (ESMValCore) that performs common pre-processing operations and a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top-down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klima Rechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5.

scholarly journals Earth System Model Evaluation Tool (ESMValTool) v2.0 – an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

2020 ◽  
Vol 13 (7) ◽  
pp. 3383-3438 ◽  
Author(s):  
Veronika Eyring ◽  
Lisa Bock ◽  
Axel Lauer ◽  
Mattia Righi ◽  
Manuel Schlund ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Large Scale ◽  
Performance Metrics ◽  
Earth System Model ◽  
System Model ◽  
Earth System ◽  
Evaluation Tool ◽  
The Earth ◽  
And Performance ◽  
Earth System Models

Abstract. The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth system models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of (1) an easy-to-install, well-documented Python package providing the core functionalities (ESMValCore) that performs common preprocessing operations and (2) a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability, the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top–down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klimarechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5.

scholarly journals ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth System Models in CMIP

2015 ◽  
Vol 8 (9) ◽  
pp. 7541-7661 ◽  
Author(s):  
V. Eyring ◽  
M. Righi ◽  
M. Evaldsson ◽  
A. Lauer ◽  
S. Wenzel ◽  
...  
Keyword(s):  
Performance Metrics ◽  
Stratospheric Ozone ◽  
Coupled Model ◽  
Earth System ◽  
Evaluation Tool ◽  
System Models ◽  
The Earth ◽  
Systematic Biases ◽  
And Performance ◽  
Earth System Models

Abstract. A community diagnostics and performance metrics tool for the evaluation of Earth System Models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected Essential Climate Variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases and soil hydrology-climate interactions, as well as atmospheric CO2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the CMIP ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within the Coupled Model Intercomparison Project (CMIP) and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

scholarly journals ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP

2016 ◽  
Vol 9 (5) ◽  
pp. 1747-1802 ◽  
Author(s):  
Veronika Eyring ◽  
Mattia Righi ◽  
Axel Lauer ◽  
Martin Evaldsson ◽  
Sabrina Wenzel ◽  
...  
Keyword(s):  
Performance Metrics ◽  
Stratospheric Ozone ◽  
Coupled Model ◽  
Earth System ◽  
Evaluation Tool ◽  
System Models ◽  
The Earth ◽  
Systematic Biases ◽  
And Performance ◽  
Earth System Models

Abstract. A community diagnostics and performance metrics tool for the evaluation of Earth system models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected essential climate variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases, and soil hydrology–climate interactions, as well as atmospheric CO2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the Coupled Model Intercomparison Project (CMIP) ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within CMIP and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

ESMValCore: analyzing CMIP data made easy

2020 ◽  
Author(s):  
Bouwe Andela ◽  
Lisa Bock ◽  
Björn Brötz ◽  
Faruk Diblen ◽  
Laura Dreyer ◽  
...  
Keyword(s):  
Open Source ◽  
Performance Metrics ◽  
Modular Design ◽  
Coupled Model ◽  
Application Programming Interface ◽  
Earth System ◽  
Evaluation Tool ◽  
System Models ◽  
Scientific Analysis ◽  
Earth System Models

<p>The Earth System Model Evaluation Tool (ESMValTool) is a free and open-source community diagnostic and performance metrics tool for the evaluation of Earth system models participating in the Coupled Model Intercomparison Project (CMIP). Version 2 of the tool (Righi et al. 2019, www.esmvaltool.org) features a brand new design, consisting of ESMValCore (https://github.com/esmvalgroup/esmvalcore), a package for working with CMIP data and ESMValTool (https://github.com/esmvalgroup/esmvaltool), a package containing the scientific analysis scripts. This new version has been specifically developed to handle the increased data volume of CMIP Phase 6 (CMIP6) and the related challenges posed by the analysis and the evaluation of output from multiple high-resolution or complex Earth system models. The tool also supports CMIP5 and CMIP3 datasets, as well as a large number of re-analysis and observational datasets that can be formatted according to the same standards (CMOR) on-the-fly or through scripts currently included in the ESMValTool package.</p><p>At the heart of this new version is the ESMValCore software package, which provides a configurable framework for finding CMIP files using a “data reference syntax”, applying commonly used pre-processing functions to them, running analysis scripts, and recording provenance. Numerous pre-processing functions, e.g. for data selection, regridding, and statistics are readily available and the modular design makes it easy to add more. The ESMValCore package is easy to install with relatively few dependencies, written in Python 3, based on state-of-the-art open-source libraries such as Iris and Dask, and widely used standards such as YAML, NetCDF, CF-Conventions, and W3C PROV. An extensive set of automated tests and code quality checks ensure the reliability of the package. Documentation is available at https://esmvaltool.readthedocs.io.</p><p>The ESMValCore package uses human-readable recipes to define which variables and datasets to use, how to pre-process that data, and what scientific analysis scripts to run. The package provides convenient interfaces, based on the YAML and NetCDF/CF-convention file formats, for running diagnostic scripts written in any programming language. Because the ESMValCore framework takes care of running the workflow defined in the recipe in parallel, most analyses run much faster, with no additional programming effort required from the authors of the analysis scripts. For example, benchmarks show a factor of 30 speedup with respect to version 1 of the tool for a representative recipe on a 24 core machine. A large collection of standard recipes and associated analysis scripts is available in the ESMValTool package for reproducing selected peer-reviewed analyses. The ESMValCore package can also be used with any other script that implements it’s easy to use interface. All pre-processing functions of the ESMValCore can also be used directly from any Python program. These features allow for use by a wide community of scientific users and developers with different levels of programming skills and experience.</p><p>Future plans involve extending the public Python API (application programming interface) from just preprocessor functions to include all functionality, including finding the data and running diagnostic scripts. This would make ESMValCore suitable for interactive data exploration from a Jupyter Notebook.</p>

scholarly journals On an improved sub-regional water resources management representation for integration into earth system models

2013 ◽  
Vol 17 (9) ◽  
pp. 3605-3622 ◽  
Author(s):  
N. Voisin ◽  
H. Li ◽  
D. Ward ◽  
M. Huang ◽  
M. Wigmosta ◽  
...  
Keyword(s):  
Water Resources ◽  
Flood Control ◽  
Large Scale ◽  
Regulation System ◽  
Earth System ◽  
Mean Flow ◽  
System Models ◽  
Operating Rules ◽  
Reservoir Models ◽  
Earth System Models

Abstract. Human influence on the hydrologic cycle includes regulation and storage, consumptive use and overall redistribution of water resources in space and time. Representing these processes is essential for applications of earth system models in hydrologic and climate predictions, as well as impact studies at regional to global scales. Emerging large-scale research reservoir models use generic operating rules that are flexible for coupling with earth system models. Those generic operating rules have been successful in reproducing the overall regulated flow at large basin scales. This study investigates the uncertainties of the reservoir models from different implementations of the generic operating rules using the complex multi-objective Columbia River Regulation System in northwestern United States as an example to understand their effects on not only regulated flow but also reservoir storage and fraction of the demand that is met. Numerical experiments are designed to test new generic operating rules that combine storage and releases targets for multi-purpose reservoirs and to compare the use of reservoir usage priorities and predictors (withdrawals vs. consumptive demands, as well as natural vs. regulated mean flow) for configuring operating rules. Overall the best performing implementation is with combined priorities rules (flood control storage targets and irrigation release targets) set up with mean annual natural flow and mean monthly withdrawals. The options of not accounting for groundwater withdrawals, or on the contrary, of assuming that all remaining demand is met through groundwater extractions, are discussed.

Recent developments on the Earth System Model Evaluation Tool

2021 ◽  
Author(s):  
Bouwe Andela ◽  
Fakhereh Alidoost ◽  
Lukas Brunner ◽  
Jaro Camphuijsen ◽  
Bas Crezee ◽  
...  
Keyword(s):  
Model Evaluation ◽  
Performance Metrics ◽  
Climate Model ◽  
Earth System Model ◽  
System Model ◽  
Earth System ◽  
Evaluation Tool ◽  
Scientific Review ◽  
The Earth ◽  
User Friendly

<p>The Earth System Model Evaluation Tool (ESMValTool) is a free and open-source community diagnostic and performance metrics tool for the evaluation of Earth system models such as those participating in the Coupled Model Intercomparison Project (CMIP). Version 2 of the tool (Righi et al. 2020, www.esmvaltool.org) features a brand new design composed of a core that finds and processes data according to a ‘recipe’ and an extensive collection of ready-to-use recipes and associated diagnostic codes for reproducing results from published papers. Development and discussion of the tool (mostly) takes place in public on https://github.com/esmvalgroup and anyone with an interest in climate model evaluation is welcome to join there.</p><p> </p><p>Since the initial release of version 2 in the summer of 2020, many improvements have been made to the tool. It is now more user friendly with extensive documentation available on docs.esmvaltool.org and a step by step online tutorial. Regular releases, currently planned three times a year, ensure that recent contributions become available quickly while still ensuring a high level of quality control. The tool can be installed from conda, but portable docker and singularity containers are also available.</p><p> </p><p>Recent new features include a more user-friendly command-line interface, citation information per figure including CMIP6 data citation using ES-DOC, more and faster preprocessor functions that require less memory, automatic corrections for a larger number of CMIP6 datasets, support for more observational and reanalysis datasets, and more recipes and diagnostics.</p><p> </p><p>The tool is now also more reliable, with improved automated testing through more unit tests for the core, as well as a recipe testing service running at DKRZ for testing the scientific recipes and diagnostics that are bundled into the tool. The community maintaining and developing the tool is growing, making the project less dependent on individual contributors. There are now technical and scientific review teams that review new contributions for technical quality and scientific correctness and relevance respectively, two new principal investigators for generating a larger support base in the community, and a newly created user engagement team that is taking care of improving the overall user experience.</p>

scholarly journals An Overview of CMIP5 and the Experiment Design

2012 ◽  
Vol 93 (4) ◽  
pp. 485-498 ◽  
Author(s):  
Karl E. Taylor ◽  
Ronald J. Stouffer ◽  
Gerald A. Meehl
Keyword(s):  
Climate Change ◽  
Climate Models ◽  
Global Climate ◽  
Experiment Design ◽  
Global Climate Models ◽  
Model Output ◽  
Earth System ◽  
Intergovernmental Panel ◽  
System Models ◽  
Earth System Models

The fifth phase of the Coupled Model Intercomparison Project (CMIP5) will produce a state-of-the- art multimodel dataset designed to advance our knowledge of climate variability and climate change. Researchers worldwide are analyzing the model output and will produce results likely to underlie the forthcoming Fifth Assessment Report by the Intergovernmental Panel on Climate Change. Unprecedented in scale and attracting interest from all major climate modeling groups, CMIP5 includes “long term” simulations of twentieth-century climate and projections for the twenty-first century and beyond. Conventional atmosphere–ocean global climate models and Earth system models of intermediate complexity are for the first time being joined by more recently developed Earth system models under an experiment design that allows both types of models to be compared to observations on an equal footing. Besides the longterm experiments, CMIP5 calls for an entirely new suite of “near term” simulations focusing on recent decades and the future to year 2035. These “decadal predictions” are initialized based on observations and will be used to explore the predictability of climate and to assess the forecast system's predictive skill. The CMIP5 experiment design also allows for participation of stand-alone atmospheric models and includes a variety of idealized experiments that will improve understanding of the range of model responses found in the more complex and realistic simulations. An exceptionally comprehensive set of model output is being collected and made freely available to researchers through an integrated but distributed data archive. For researchers unfamiliar with climate models, the limitations of the models and experiment design are described.

CMIP model evaluation with the ESMValTool v2.0

2020 ◽  
Author(s):  
Axel Lauer ◽  
Fernando Iglesias-Suarez ◽  
Veronika Eyring ◽  
the ESMValTool development team
Keyword(s):  
Model Evaluation ◽  
Large Scale ◽  
Performance Metrics ◽  
Comprehensive Evaluation ◽  
Coupled Model ◽  
Data Sets ◽  
Evaluation Tool ◽  
Emergent Constraints ◽  
Data Volume ◽  
And Performance

<p>The Earth System Model Evaluation Tool (ESMValTool) has been developed with the aim of taking model evaluation to the next level by facilitating analysis of many different ESM components, providing well-documented source code and scientific background of implemented diagnostics and metrics and allowing for traceability and reproducibility of results (provenance). This has been made possible by a lively and growing development community continuously improving the tool supported by multiple national and European projects. The latest version (2.0) of the ESMValTool has been developed as a large community effort to specifically target the increased data volume of the Coupled Model Intercomparison Project Phase 6 (CMIP6) and the related challenges posed by analysis and evaluation of output from multiple high-resolution and complex ESMs. For this, the core functionalities have been completely rewritten in order to take advantage of state-of-the-art computational libraries and methods to allow for efficient and user-friendly data processing. Common operations on the input data such as regridding or computation of multi-model statistics are now centralized in a highly optimized preprocessor written in Python. The diagnostic part of the ESMValTool includes a large collection of standard recipes for reproducing peer-reviewed analyses of many variables across atmosphere, ocean, and land domains, with diagnostics and performance metrics focusing on the mean-state, trends, variability and important processes, phenomena, as well as emergent constraints. While most of the diagnostics use observational data sets (in particular satellite and ground-based observations) or reanalysis products for model evaluation some are also based on model-to-model comparisons. This presentation introduces the diagnostics newly implemented into ESMValTool v2.0 including an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of ESMs, new diagnostics for extreme events, regional model and impact evaluation and analysis of ESMs, as well as diagnostics for emergent constraints and analysis of future projections from ESMs. The new diagnostics are illustrated with examples using results from the well-established CMIP5 and the newly available CMIP6 data sets.</p>

scholarly journals Performance Analysis of Large Scale HPC Workflows for Earth System Models

2017 ◽  
Author(s):  
Joseph H. Kennedy ◽  
Benjamin W. Mayer ◽  
Katherine J. Evans ◽  
Jeff Duracha
Keyword(s):  
Performance Analysis ◽  
Large Scale ◽  
Earth System ◽  
System Models ◽  
Earth System Models
Sign in / Sign up

Export Citation Format

Share Document