The ARM Data-Oriented Metrics and Diagnostics Package for Climate Models: A New Tool for Evaluating Climate Models with Field Data

AbstractThe U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) program User Facility produces ground-based long-term continuous unique measurements for atmospheric state, precipitation, turbulent fluxes, radiation, aerosol, cloud, and the land surface, which are collected at multiple sites. These comprehensive datasets have been widely used to calibrate climate models and are proven to be invaluable for climate model development and improvement. This article introduces an evaluation package to facilitate the use of ground-based ARM measurements in climate model evaluation. The ARM data-oriented metrics and diagnostics package (ARM-DIAGS) includes both ARM observational datasets and a Python-based analysis toolkit for computation and visualization. The observational datasets are compiled from multiple ARM data products and specifically tailored for use in climate model evaluation. In addition, ARM-DIAGS also includes simulation data from models participating the Coupled Model Intercomparison Project (CMIP), which will allow climate-modeling groups to compare a new, candidate version of their model to existing CMIP models. The analysis toolkit is designed to make the metrics and diagnostics quickly available to the model developers.

Download Full-text

A More Powerful Reality Test for Climate Models

Eos ◽

10.1029/2016eo051663 ◽

2016 ◽

Vol 97 ◽

Cited By ~ 32

Author(s):

Peter Gleckler ◽

Charles Doutriaux ◽

Paul Durack ◽

Karl Taylor ◽

Yuying Zhang ◽

...

Keyword(s):

Community Engagement ◽

Model Evaluation ◽

Climate Models ◽

Climate Model ◽

Model Development ◽

Climate Model Evaluation

A new climate model evaluation package will deliver objective comparisons between models and observations for research and model development and provide a framework for community engagement.

Download Full-text

Climate Model Evaluation in the Presence of Observational Uncertainty: Precipitation Indices over the Contiguous United States

Journal of Hydrometeorology ◽

10.1175/jhm-d-18-0230.1 ◽

2019 ◽

Vol 20 (7) ◽

pp. 1339-1357 ◽

Cited By ~ 9

Author(s):

Peter B. Gibson ◽

Duane E. Waliser ◽

Huikyo Lee ◽

Baijun Tian ◽

Elias Massoud

Keyword(s):

United States ◽

Model Evaluation ◽

Climate Models ◽

Climate Model ◽

Dynamical Downscaling ◽

Daily Precipitation ◽

Regional Climate ◽

Climate Model Evaluation ◽

Observational Uncertainty ◽

Precipitation Indices

Abstract Climate model evaluation is complicated by the presence of observational uncertainty. In this study we analyze daily precipitation indices and compare multiple gridded observational and reanalysis products with regional climate models (RCMs) from the North American component of the Coordinated Regional Climate Downscaling Experiment (NA-CORDEX) multimodel ensemble. In the context of model evaluation, observational product differences across the contiguous United States (CONUS) are also deemed nontrivial for some indices, especially for annual counts of consecutive wet days and for heavy precipitation indices. Multidimensional scaling (MDS) is used to directly include this observational spread into the model evaluation procedure, enabling visualization and interpretation of model differences relative to a “cloud” of observational uncertainty. Applying MDS to the evaluation of NA-CORDEX RCMs reveals situations of added value from dynamical downscaling, situations of degraded performance from dynamical downscaling, and the sensitivity of model performance to model resolution. On precipitation days, higher-resolution RCMs typically simulate higher mean and extreme precipitation rates than their lower-resolution pairs, sometimes improving model fidelity with observations. These results document the model spread and biases in daily precipitation extremes across the full NA-CORDEX model ensemble. The often-large divergence between in situ observations, satellite data, and reanalysis, shown here for CONUS, is especially relevant for data-sparse regions of the globe where satellite and reanalysis products are extensively relied upon. This highlights the need to carefully consider multiple observational products when evaluating climate models.

Download Full-text

Regional Climate Model Evaluation System powered by Apache Open Climate Workbench v1.3.0: an enabling tool for facilitating regional climate studies

10.5194/gmd-2018-113 ◽

2018 ◽

Cited By ~ 1

Author(s):

Huikyo Lee ◽

Alexander Goodman ◽

Lewis McGibbney ◽

Duane Waliser ◽

Jinwon Kim ◽

...

Keyword(s):

Regional Climate Model ◽

Model Evaluation ◽

Evaluation System ◽

Climate Models ◽

Performance Metrics ◽

Climate Model ◽

Regional Climate ◽

The United States ◽

Climate Data ◽

Climate Model Evaluation

Abstract. The Regional Climate Model Evaluation System (RCMES) is an enabling tool of the National Aeronautics and Space Administration to support the United States National Climate Assessment. As a comprehensive system for evaluating climate models on regional and continental scales using observational datasets from a variety of sources, RCMES is designed to yield information on the performance of climate models and guide their improvement. Here we present a user-oriented document describing the latest version of RCMES, its development process and future plans for improvements. The main objective of RCMES is to facilitate the climate model evaluation process at regional scales. RCMES provides a framework for performing systematic evaluations of climate simulations, such as those from the Coordinated Regional Climate Downscaling Experiment (CORDEX), using in-situ observations as well as satellite and reanalysis data products. The main components of RCMES are: 1) a database of observations widely used for climate model evaluation, 2) various data loaders to import climate models and observations in different formats, 3) a versatile processor to subset and regrid the loaded datasets, 4) performance metrics designed to assess and quantify model skill, 5) plotting routines to visualize the performance metrics, 6) a toolkit for statistically downscaling climate model simulations, and 7) two installation packages to maximize convenience of users without Python skills. RCMES website is maintained up to date with brief explanation of these components. Although there are other open-source software (OSS) toolkits that facilitate analysis and evaluation of climate models, there is a need for climate scientists to participate in the development and customization of OSS to study regional climate change. To establish infrastructure and to ensure software sustainability, development of RCMES is an open, publicly accessible process enabled by leveraging the Apache Software Foundation's OSS library, Apache Open Climate Workbench (OCW). The OCW software that powers RCMES includes a Python OSS library for common climate model evaluation tasks as well as a set of user-friendly interfaces for quickly configuring a model evaluation task. OCW also allows users to build their own climate data analysis tools, such as the statistical downscaling toolkit provided as a part of RCMES.

Download Full-text

Systematic Climate Model Rainfall Biases over Southern Africa: Links to Moisture Circulation and Topography

Journal of Climate ◽

10.1175/jcli-d-18-0008.1 ◽

2018 ◽

Vol 31 (18) ◽

pp. 7533-7548 ◽

Cited By ~ 15

Author(s):

C. Munday ◽

R. Washington

Keyword(s):

Southern Africa ◽

Climate Models ◽

Climate Model ◽

Regional Scale ◽

Model Development ◽

Coupled Model ◽

Climate Science ◽

Summer Rainfall ◽

Southwest Indian Ocean ◽

Low Level

An important challenge for climate science is to understand the regional circulation and rainfall response to global warming. Unfortunately, the climate models used to project future changes struggle to represent present-day rainfall and circulation, especially at a regional scale. This is the case in southern Africa, where models from phase 5 of the Coupled Model Intercomparison Project (CMIP5) overestimate summer rainfall by as much as 300% compared to observations and tend to underestimate rainfall in Madagascar and the southwest Indian Ocean. In this paper, we explore the climate processes associated with the rainfall bias, with the aim of assessing the reliability of the CMIP5 ensemble and highlighting important areas for model development. We find that the high precipitation rates in models that are wet over southern Africa are associated with an anomalous northeasterly moisture transport (~10–30 g kg−1 s−1) that penetrates across the high topography of Tanzania and Malawi and into subtropical southern Africa. This transport occurs in preference to a southeasterly recurvature toward Madagascar that is seen in drier models and reanalysis data. We demonstrate that topographically related model biases in low-level flow are important for explaining the intermodel spread in rainfall; wetter models have a reduced tendency to block the oncoming northeasterly flow compared to dry models. The differences in low-level flow among models are related to upstream wind speed and model representation of topography, both of which should be foci for model development.

Download Full-text

A community diagnostic tool for Chemistry Climate Model Validation

Geoscientific Model Development Discussions ◽

10.5194/gmdd-5-1229-2012 ◽

2012 ◽

Vol 5 (2) ◽

pp. 1229-1261

Author(s):

A. Gettelman ◽

V. Eyring ◽

C. Fischer ◽

H. Shiona ◽

I. Cionni ◽

...

Keyword(s):

Model Validation ◽

Climate Models ◽

Performance Metrics ◽

Climate Model ◽

Model Development ◽

Coupled Model ◽

Technical Note ◽

Complex Evaluation ◽

System User ◽

Intercomparison Project

Abstract. This technical note presents an overview of the Chemistry-Climate Model Validation Diagnostic (CCMVal-Diag) tool for model evaluation. The CCMVal-Diag tool is a flexible and extensible open source package that facilitates the complex evaluation of global models. Models can be compared to other models, ensemble members (simulations with the same model), and/or many types of observations. The tool can also compute quantitative performance metrics. The initial construction and application is to coupled Chemistry-Climate Models (CCMs) participating in CCMVal, but the evaluation of climate models that submitted output to the Coupled Model Intercomparison Project (CMIP) is also possible. The package has been used to assist with analysis of simulations for the 2010 WMO/UNEP Scientific Ozone Assessment and the SPARC Report on the Evaluation of CCMs. The CCMVal-Diag tool is described and examples of how it functions are presented, along with links to detailed descriptions, instructions and source code. The CCMVal-Diag tool is supporting model development as well as quantifying model improvements, both for different versions of individual models and for different generations of community-wide collections of models used in international assessments. The code allows further extensions by different users for different applications and types, e.g. to other components of the Earth System. User modifications are encouraged and easy to perform with a minimum of coding.

Download Full-text

Global exposure to flooding from the new CMIP6 climate model projections

Scientific Reports ◽

10.1038/s41598-021-83279-w ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Yukiko Hirabayashi ◽

Masahiro Tanoue ◽

Orie Sasaki ◽

Xudong Zhou ◽

Dai Yamazaki

Keyword(s):

Spatial Patterns ◽

Flood Risk ◽

Climate Models ◽

Climate Model ◽

Model Development ◽

Coupled Model ◽

Model Intercomparison ◽

Internal Climate Variability ◽

Intercomparison Project ◽

New Generation

AbstractEstimates of future flood risk rely on projections from climate models. The relatively few climate models used to analyze future flood risk cannot easily quantify of their associated uncertainties. In this study, we demonstrated that the projected fluvial flood changes estimated by a new generation of climate models, the collectively known as Coupled Model Intercomparison Project Phase 6 (CMIP6), are similar to those estimated by CMIP5. The spatial patterns of the multi-model median signs of change (+ or −) were also very consistent, implying greater confidence in the projections. The model spread changed little over the course of model development, suggesting irreducibility of the model spread due to internal climate variability, and the consistent projections of models from the same institute suggest the potential to reduce uncertainties caused by model differences. Potential global exposure to flooding is projected to be proportional to the degree of warming, and a greater threat is anticipated as populations increase, demonstrating the need for immediate decisions.

Download Full-text

A Bayesian approach to climate model evaluation and multi-model averaging with an application to global mean surface temperatures from IPCC AR4 coupled climate models

Geophysical Research Letters ◽

10.1029/2006gl025779 ◽

2006 ◽

Vol 33 (8) ◽

Cited By ~ 64

Author(s):

Seung-Ki Min ◽

Andreas Hense

Keyword(s):

Bayesian Approach ◽

Model Evaluation ◽

Climate Models ◽

Climate Model ◽

Model Averaging ◽

Surface Temperatures ◽

Coupled Climate Models ◽

Climate Model Evaluation ◽

Ipcc Ar4 ◽

Global Mean

Download Full-text

A community diagnostic tool for chemistry climate model validation

Geoscientific Model Development ◽

10.5194/gmd-5-1061-2012 ◽

2012 ◽

Vol 5 (5) ◽

pp. 1061-1073 ◽

Cited By ~ 9

Author(s):

A. Gettelman ◽

V. Eyring ◽

C. Fischer ◽

H. Shiona ◽

I. Cionni ◽

...

Keyword(s):

Model Validation ◽

Climate Models ◽

Climate Model ◽

Model Development ◽

Coupled Model ◽

Technical Note ◽

The Earth ◽

Complex Evaluation ◽

System User ◽

Intercomparison Project

Abstract. This technical note presents an overview of the Chemistry-Climate Model Validation Diagnostic (CCMVal-Diag) tool for model evaluation. The CCMVal-Diag tool is a flexible and extensible open source package that facilitates the complex evaluation of global models. Models can be compared to other models, ensemble members (simulations with the same model), and/or many types of observations. The initial construction and application is to coupled chemistry-climate models (CCMs) participating in CCMVal, but the evaluation of climate models that submitted output to the Coupled Model Intercomparison Project (CMIP) is also possible. The package has been used to assist with analysis of simulations for the 2010 WMO/UNEP Scientific Ozone Assessment and the SPARC Report on the Evaluation of CCMs. The CCMVal-Diag tool is described and examples of how it functions are presented, along with links to detailed descriptions, instructions and source code. The CCMVal-Diag tool supports model development as well as quantifies model changes, both for different versions of individual models and for different generations of community-wide collections of models used in international assessments. The code allows further extensions by different users for different applications and types, e.g. to other components of the Earth system. User modifications are encouraged and easy to perform with minimum coding.

Download Full-text

Regional Climate Model Evaluation System powered by Apache Open Climate Workbench v1.3.0: an enabling tool for facilitating regional climate studies

Geoscientific Model Development ◽

10.5194/gmd-11-4435-2018 ◽

2018 ◽

Vol 11 (11) ◽

pp. 4435-4449 ◽

Cited By ~ 6

Author(s):

Huikyo Lee ◽

Alexander Goodman ◽

Lewis McGibbney ◽

Duane E. Waliser ◽

Jinwon Kim ◽

...

Keyword(s):

Regional Climate Model ◽

Model Evaluation ◽

Evaluation System ◽

Climate Models ◽

Performance Metrics ◽

Climate Model ◽

Regional Climate ◽

The United States ◽

Climate Data ◽

Climate Model Evaluation

Abstract. The Regional Climate Model Evaluation System (RCMES) is an enabling tool of the National Aeronautics and Space Administration to support the United States National Climate Assessment. As a comprehensive system for evaluating climate models on regional and continental scales using observational datasets from a variety of sources, RCMES is designed to yield information on the performance of climate models and guide their improvement. Here, we present a user-oriented document describing the latest version of RCMES, its development process, and future plans for improvements. The main objective of RCMES is to facilitate the climate model evaluation process at regional scales. RCMES provides a framework for performing systematic evaluations of climate simulations, such as those from the Coordinated Regional Climate Downscaling Experiment (CORDEX), using in situ observations, as well as satellite and reanalysis data products. The main components of RCMES are (1) a database of observations widely used for climate model evaluation, (2) various data loaders to import climate models and observations on local file systems and Earth System Grid Federation (ESGF) nodes, (3) a versatile processor to subset and regrid the loaded datasets, (4) performance metrics designed to assess and quantify model skill, (5) plotting routines to visualize the performance metrics, (6) a toolkit for statistically downscaling climate model simulations, and (7) two installation packages to maximize convenience of users without Python skills. RCMES website is maintained up to date with a brief explanation of these components. Although there are other open-source software (OSS) toolkits that facilitate analysis and evaluation of climate models, there is a need for climate scientists to participate in the development and customization of OSS to study regional climate change. To establish infrastructure and to ensure software sustainability, development of RCMES is an open, publicly accessible process enabled by leveraging the Apache Software Foundation's OSS library, Apache Open Climate Workbench (OCW). The OCW software that powers RCMES includes a Python OSS library for common climate model evaluation tasks as well as a set of user-friendly interfaces for quickly configuring a model evaluation task. OCW also allows users to build their own climate data analysis tools, such as the statistical downscaling toolkit provided as a part of RCMES.

Download Full-text

Documenting Climate Models and Their Simulations

Bulletin of the American Meteorological Society ◽

10.1175/bams-d-11-00035.1 ◽

2013 ◽

Vol 94 (5) ◽

pp. 623-627 ◽

Cited By ~ 16

Author(s):

Eric Guilyardi ◽

V. Balaji ◽

Bryan Lawrence ◽

Sarah Callaghan ◽

Cecelia Deluca ◽

...

Keyword(s):

Climate Models ◽

Climate Model ◽

Climate Modeling ◽

Coupled Model ◽

Climate Change Impacts ◽

Model Output ◽

Initial Application ◽

Wide Range ◽

Community Initiative ◽

Climate Model Output

The results of climate models are of increasing and widespread importance. No longer is climate model output of sole interest to climate scientists and researchers in the climate change impacts and adaptation fields. Now nonspecialists such as government officials, policy makers, and the general public all have an increasing need to access climate model output and understand its implications. For this host of users, accurate and complete metadata (i.e., information about how and why the data were produced) is required to document the climate modeling results. Here we describe a pilot community initiative to collect and make available documentation of climate models and their simulations. In an initial application, a metadata repository is being established to provide information of this kind for a major internationally coordinated modeling activity known as CMIP5 (Coupled Model Intercomparison Project, Phase 5). It is expected that for a wide range of stakeholders, this and similar community-managed metadata repositories will spur development of analysis tools that facilitate discovery and exploitation of Earth system simulations.

Download Full-text