scholarly journals A Framework for Evaluating Climate Model Performance Metrics

2016 ◽  
Vol 29 (5) ◽  
pp. 1773-1782 ◽  
Author(s):  
Noel C. Baker ◽  
Patrick C. Taylor
Keyword(s):  
Performance Metrics ◽  
Climate Model ◽  
Probability Distributions ◽  
Radiant Energy ◽  
Model Performance ◽  
Coupled Model ◽  
Energy System ◽  
Cmip5 Models ◽  
Model Intercomparison ◽  
Model Quality

Abstract Given the large amount of climate model output generated from the series of simulations from phase 5 of the Coupled Model Intercomparison Project (CMIP5), a standard set of performance metrics would facilitate model intercomparison and tracking performance improvements. However, no framework exists for the evaluation of performance metrics. The proposed framework systematically integrates observations into metric assessment to quantitatively evaluate metrics. An optimal metric is defined in this framework as one that measures a behavior that is strongly linked to model quality in representing mean-state present-day climate. The goal of the framework is to objectively and quantitatively evaluate the ability of a performance metric to represent overall model quality. The framework is demonstrated, and the design principles are discussed using a novel set of performance metrics, which assess the simulation of top-of-atmosphere (TOA) and surface radiative flux variance and probability distributions within 34 CMIP5 models against Clouds and the Earth’s Radiant Energy System (CERES) observations and GISS Surface Temperature Analysis (GISTEMP). Of the 44 tested metrics, the optimal metrics are found to be those that evaluate global-mean TOA radiation flux variance.

Evaluation of net shortwave radiation over China with a regional climate model

2020 ◽  
Vol 80 (2) ◽  
pp. 147-163
Author(s):  
X Liu ◽  
Y Kang ◽  
Q Liu ◽  
Z Guo ◽  
Y Chen ◽  
...  
Keyword(s):  
Regional Climate Model ◽  
Climate Model ◽  
Regional Climate ◽  
Radiant Energy ◽  
Energy System ◽  
Radiation Budget ◽  
Shortwave Radiation ◽  
Monsoon Region ◽  
Clear Sky ◽  
Sky Conditions

The regional climate model RegCM version 4.6, developed by the European Centre for Medium-Range Weather Forecasts Reanalysis, was used to simulate the radiation budget over China. Clouds and the Earth’s Radiant Energy System (CERES) satellite data were utilized to evaluate the simulation results based on 4 radiative components: net shortwave (NSW) radiation at the surface of the earth and top of the atmosphere (TOA) under all-sky and clear-sky conditions. The performance of the model for low-value areas of NSW was superior to that for high-value areas. NSW at the surface and TOA under all-sky conditions was significantly underestimated; the spatial distribution of the bias was negative in the north and positive in the south, bounded by 25°N for the annual and seasonal averaged difference maps. Compared with the all-sky condition, the simulation effect under clear-sky conditions was significantly better, which indicates that the cloud fraction is the key factor affecting the accuracy of the simulation. In particular, the bias of the TOA NSW under the clear-sky condition was <±10 W m-2 in the eastern areas. The performance of the model was better over the eastern monsoon region in winter and autumn for surface NSW under clear-sky conditions, which may be related to different levels of air pollution during each season. Among the 3 areas, the regional average biases overall were largest (negative) over the Qinghai-Tibet alpine region and smallest over the eastern monsoon region.

Observed and Projected Frontal Activities in East Asia

2021 ◽  
pp. 1-46
Author(s):  
Chia-Chi Wang ◽  
Huang-Hsiung Hsu ◽  
Ying-Ting Chen
Keyword(s):  
East Asia ◽  
Climate Model ◽  
Southern China ◽  
Model Performance ◽  
Moisture Flux ◽  
Central China ◽  
Cmip5 Models ◽  
The South ◽  
Front Detection ◽  
Local Moisture

AbstractAn objective front detection method is applied to ERA5, CMIP5 historical, and RCP8.5 simulations to evaluate climate model performance in simulating front frequency and understand future projections of seasonal front activities. The study area is East Asia for two natural seasons, defined as winter (December 2nd –February 14th) and spring (February 15th –May 15th), in accordance with regional circulation and precipitation patterns. Seasonal means of atmospheric circulation and thermal structures are analyzed to understand possible factors responsible for future front changes.The front location and frequency in CMIP5 historical simulations are captured reasonably. Frontal precipitation accounts for more than 30% of total precipitation over subtropical regions. Projections suggest that winter fronts will decrease over East Asia, especially over southern China. Frontal precipitation is projected to decrease for 10-30%. Front frequency increases in the South China Sea and tropical western Pacific because of more tropical moisture supply, which enhances local moisture contrasts. During spring, southern China and Taiwan will experience fewer fronts and less frontal precipitation while central China, Korea, and Japan may experience more fronts and more frontal precipitation due to moisture flux from the south that enhances 𝜽𝒘 gradients.Consensus among CMIP5 models in front frequency tendency is evaluated. The models exhibit relatively high consensus in the decreasing trend over polar and subtropical frontal zone in winter and over southern China and Taiwan in spring that may prolong the dry season. Spring front activities are crucial for water resource and risk management in the southern China and Taiwan.

scholarly journals Inconsistent strategies to spin up models in CMIP5: implications for ocean biogeochemical model performance assessment

2016 ◽  
Vol 9 (5) ◽  
pp. 1827-1851 ◽  
Author(s):  
Roland Séférian ◽  
Marion Gehlen ◽  
Laurent Bopp ◽  
Laure Resplandy ◽  
James C. Orr ◽  
...  
Keyword(s):  
Performance Assessment ◽  
Initial Conditions ◽  
Model Performance ◽  
Cmip5 Models ◽  
Biogeochemical Model ◽  
Earth System ◽  
Model Intercomparison ◽  
System Models ◽  
Marine Biogeochemistry ◽  
Earth System Models

Abstract. During the fifth phase of the Coupled Model Intercomparison Project (CMIP5) substantial efforts were made to systematically assess the skill of Earth system models. One goal was to check how realistically representative marine biogeochemical tracer distributions could be reproduced by models. In routine assessments model historical hindcasts were compared with available modern biogeochemical observations. However, these assessments considered neither how close modeled biogeochemical reservoirs were to equilibrium nor the sensitivity of model performance to initial conditions or to the spin-up protocols. Here, we explore how the large diversity in spin-up protocols used for marine biogeochemistry in CMIP5 Earth system models (ESMs) contributes to model-to-model differences in the simulated fields. We take advantage of a 500-year spin-up simulation of IPSL-CM5A-LR to quantify the influence of the spin-up protocol on model ability to reproduce relevant data fields. Amplification of biases in selected biogeochemical fields (O2, NO3, Alk-DIC) is assessed as a function of spin-up duration. We demonstrate that a relationship between spin-up duration and assessment metrics emerges from our model results and holds when confronted with a larger ensemble of CMIP5 models. This shows that drift has implications for performance assessment in addition to possibly aliasing estimates of climate change impact. Our study suggests that differences in spin-up protocols could explain a substantial part of model disparities, constituting a source of model-to-model uncertainty. This requires more attention in future model intercomparison exercises in order to provide quantitatively more correct ESM results on marine biogeochemistry and carbon cycle feedbacks.

scholarly journals Evaluation of Surface Upward Longwave Radiation in the CMIP6 Models with Ground and Satellite Observations

2021 ◽  
Vol 13 (21) ◽  
pp. 4464
Author(s):  
Jiawen Xu ◽  
Xiaotong Zhang ◽  
Chunjie Feng ◽  
Shuyue Yang ◽  
Shikang Guan ◽  
...  
Keyword(s):  
General Circulation ◽  
Bayesian Model Averaging ◽  
Mean Squared Error ◽  
Radiant Energy ◽  
Coupled Model ◽  
Model Averaging ◽  
Longwave Radiation ◽  
Energy System ◽  
General Circulation Models ◽  
The Individual

Surface upward longwave radiation (SULR) is an indicator of thermal conditions over the Earth’s surface. In this study, we validated the simulated SULR from 51 Coupled Model Intercomparison Project (CMIP6) general circulation models (GCMs) through a comparison with ground measurements and satellite-retrieved SULR from the Clouds and the Earth’s Radiant Energy System, Energy Balanced and Filled (CERES EBAF). Moreover, we improved the SULR estimations by a fusion of multiple CMIP6 GCMs using multimodel ensemble (MME) methods. Large variations were found in the monthly mean SULR among the 51 CMIP6 GCMs; the bias and root mean squared error (RMSE) of the individual CMIP6 GCMs at 133 sites ranged from −3 to 24 W m−2 and 22 to 38 W m−2, respectively, which were higher than those found between the CERES EBAF and GCMs. The CMIP6 GCMs did not improve the overestimation of SULR compared to the CMIP5 GCMs. The Bayesian model averaging (BMA) method showed better performance in simulating SULR than the individual GCMs and simple model averaging (SMA) method, with a bias of 0 W m−2 and an RMSE of 19.29 W m−2 for the 133 sites. In terms of the global annual mean SULR, our best estimation for the CMIP6 GCMs using the BMA method was 392 W m−2 during 2000–2014. We found that the SULR varied between 386 and 393 W m−2 from 1850 to 2014, exhibiting an increasing tendency of 0.2 W m−2 per decade (p < 0.05).

scholarly journals Evaluation of soil moisture in CMIP5 simulations over contiguous United States using in situ and satellite observations

2016 ◽  
Author(s):  
Shanshui Yuan ◽  
Steven M. Quiring
Keyword(s):  
United States ◽  
Soil Moisture ◽  
Soil Column ◽  
Regional Analysis ◽  
Soil Layer ◽  
Model Performance ◽  
Coupled Model ◽  
Cmip5 Models ◽  
Near Surface

Abstract. This study provides a comprehensive evaluation of soil moisture simulations in the Coupled Model Intercomparison Project Phase 5 (CMIP5) extended historical experiment (2003 to 2012). Soil moisture from in situ and satellite sources are used to evaluate CMIP5 simulations in the contiguous United States (CONUS). Both near-surface (0–10 cm) and soil column (0–100 cm) simulations from more than 14 CMIP5 models are evaluated during the warm season (April–September). Multi-model ensemble means and the performance of individual models are assessed at a monthly time scale. Our results indicate that CMIP5 models can reproduce the seasonal variability in soil moisture over CONUS. However, the models tend to overestimate the magnitude of both near-surface and soil-column soil moisture in the western U.S. and underestimate it in the eastern U.S. There are large variations in model performance, especially in the near-surface. There are significant regional and inter-model variations in performance. Results of a regional analysis show that in deeper soil layer, the CMIP5 soil moisture simulations tend to be most skillful in the southern U.S. Based on both the satellite-derived and in situ soil moisture, CESM1, CCSM4 and GFDL-ESM2M perform best in the 0–10 cm soil layer and CESM1, CCSM4, GFDL-ESM2M and HadGEM2-ES perform best in the 0–100 cm soil layer.

Evaluating the ability of NA-CORDEX to simulate the seasonal modes of precipitation variability across the Western United States: Does resolution matter?

2021 ◽  
Author(s):  
Jonathan Meyer ◽  
Shih-Yu (Simon) Wang ◽  
Robert Gillies ◽  
Jin-Ho Yoon
Keyword(s):  
Performance Metrics ◽  
Climate Model ◽  
Regional Climate ◽  
Storm Track ◽  
Model Performance ◽  
Precipitation Variability ◽  
Future Trend ◽  
Spring Precipitation ◽  
Precipitation Patterns ◽  
The North

&lt;p&gt;The western U.S. precipitation climatology simulated by the NA-CORDEX regional climate model ensembles are examined to evaluate the capability of the 0.44&lt;sup&gt;&amp;#176; &lt;/sup&gt;and 0.22&lt;sup&gt;&amp;#176; &lt;/sup&gt;resolution&lt;sup&gt;&lt;/sup&gt;ensembles to reproduce 1) the annual and semi-annual precipitation cycle of several hydrologically important western U.S. regions and 2) localized seasonality in the amount and timing of precipitation. Collectively, when compared against observation-based gridded precipitation, NA-CORDEX RCMs driven by ERA-Interim reanalysis at the higher resolution 0.22&lt;sup&gt;&amp;#176; &lt;/sup&gt;domain resolution dramatically outperformed the 0.44&lt;sup&gt;&amp;#176;&lt;/sup&gt; ensemble over the 1950-2005 historical periods. Furthermore, the ability to capture the annual and semi-annual modes of variability was starkly improved in the higher resolution 0.22&amp;#176; ensemble. The higher resolution members reproduced more consistent spatial patterns of variance featuring lower errors in magnitude&amp;#8212;especially with respect to the winter-summer and spring-fall seasonality. A great deal of spread in model performance was found for the semi-annual cycles, although the higher-resolution ensemble exhibited a more coherent clustering of performance metrics. In general, model performance was a function of which RCM was used, while future trend scenarios seem to cluster around which GCM was downscaled.&lt;/p&gt;&lt;p&gt;&lt;br&gt;Future projections of precipitation patterns from the 0.22&amp;#176; NA-CORDEX RCMs driven by the RCP4.5 &amp;#8220;stabilization scenario&amp;#8221; and the RCP8.5 &amp;#8220;high emission&amp;#8221; scenario were analyzed to examine trends to the &amp;#8220;end of century&amp;#8221; (i.e. 2050-2099) precipitation patterns. Except for the Desert Southwest&amp;#8217;s spring season, the RCP4.5 and RCP8.5 scenarios show a consensus change towards an increase in winter and spring precipitation throughout all regions of interest with the RCP8.5 scenario containing a greater number of ensemble members simulating greater wetting trends. The future winter-summer mode of variability exhibited a general consensus towards increasing variability with greatest change found over the region&amp;#8217;s terrain suggesting a greater year-to-year variability of the region&amp;#8217;s orographic response to the strength and location of the mid-latitude jet streams and storm track. Increasing spring-fall precipitation variability suggests an expanding influence of tropical moisture advection associated with the North American Monsoon, although we note that like many future monsoon projections, a spring &amp;#8220;convective barrier&amp;#8221; was also apparent in the NA-CORDEX ensembles.&lt;/p&gt;

scholarly journals Evaluation of CMIP5 Model Precipitation Using PERSIANN-CDR

2017 ◽  
Vol 18 (9) ◽  
pp. 2313-2330 ◽  
Author(s):  
Phu Nguyen ◽  
Andrea Thorstensen ◽  
Soroosh Sorooshian ◽  
Qian Zhu ◽  
Hoang Tran ◽  
...  
Keyword(s):  
Extreme Precipitation ◽  
Climate Model ◽  
Model Performance ◽  
Model Averaging ◽  
Cmip5 Models ◽  
Global Evaluation ◽  
Climate Data ◽  
Climate Zone ◽  
Precipitation Estimates ◽  
Annual Total

Abstract The purpose of this study is to use the PERSIANN–Climate Data Record (PERSIANN-CDR) dataset to evaluate the ability of 32 CMIP5 models in capturing the behavior of daily extreme precipitation estimates globally. The daily long-term historical global PERSIANN-CDR allows for a global investigation of eight precipitation indices that is unattainable with other datasets. Quantitative comparisons against CPC daily gauge; GPCP One-Degree Daily (GPCP1DD); and TRMM 3B42, version 7 (3B42V7), datasets show the credibility of PERSIANN-CDR to be used as the reference data for global evaluation of CMIP5 models. This work uniquely defines different study regions by partitioning global land areas into 25 groups based on continent and climate zone type. Results show that model performance in warm temperate and equatorial regions in capturing daily extreme precipitation behavior is largely mixed in terms of index RMSE and correlation, suggesting that these regions may benefit from weighted model averaging schemes or model selection as opposed to simple model averaging. The three driest climate regions (snow, polar, and arid) exhibit high correlations and low RMSE values when compared against PERSIANN-CDR estimates, with the exceptions of the cold regions showing an inability to capture the 95th and 99th percentile annual total precipitation characteristics. A comprehensive assessment of each model’s performance in each continent–climate zone defined group is provided as a guide for both model developers to target regions and processes that are not yet fully captured in certain climate types, and for climate model output users to be able to select the models and/or the study areas that may best fit their applications of interest.

scholarly journals Climate Extremes over the Arabian Peninsula Using RegCM4 for Present Conditions Forced by Several CMIP5 Models

Atmosphere ◽  
2019 ◽  
Vol 10 (11) ◽  
pp. 675 ◽  
Author(s):  
Almazroui
Keyword(s):  
Land Surface ◽  
Arabian Peninsula ◽  
Climate Model ◽  
Regional Climate ◽  
Coupled Model ◽  
Climate Extremes ◽  
Maximum Temperature ◽  
Cmip5 Models ◽  
Study Region ◽  
Temperature And Precipitation

This paper investigates the temperature and precipitation extremes over the Arabian Peninsula using data from the regional climate model RegCM4 forced by three Coupled Model Intercomparison Project Phase 5 (CMIP5) models and ERA–Interim reanalysis data. Indices of extremes are calculated using daily temperature and precipitation data at 27 meteorological stations located across Saudi Arabia in line with the suggested procedure from the Expert Team on Climate Change Detection and Indices (ETCCDI) for the present climate (1986–2005) using 1981–2000 as the reference period. The results show that RegCM4 accurately captures the main features of temperature extremes found in surface observations. The results also show that RegCM4 with the CLM land–surface scheme performs better in the simulation of precipitation and minimum temperature, while the BATS scheme is better than CLM in simulating maximum temperature. Among the three CMIP5 models, the two best performing models are found to accurately reproduce the observations in calculating the extreme indices, while the other is not so successful. The reason for the good performance by these two models is that they successfully capture the circulation patterns and the humidity fields, which in turn influence the temperature and precipitation patterns that determine the extremes over the study region.

scholarly journals Intercomparison of temperature trends in IPCC CMIP5 simulations with observations, reanalyses and CMIP3 models

2013 ◽  
Vol 6 (5) ◽  
pp. 1705-1714 ◽  
Author(s):  
J. Xu ◽  
L. Zhao ◽  
Keyword(s):  
Coupled Model ◽  
Temperature Trend ◽  
The Arctic ◽  
Cmip5 Models ◽  
Model Intercomparison ◽  
Model Simulations ◽  
Temperature Trends ◽  
Intercomparison Project ◽  
Stratospheric Cooling ◽  
The Tropics

Abstract. On the basis of the fifth Coupled Model Intercomparison Project (CMIP5) and the climate model simulations covering 1979 through 2005, the temperature trends and their uncertainties have been examined to note the similarities or differences compared to the radiosonde observations, reanalyses and the third Coupled Model Intercomparison Project (CMIP3) simulations. The results show noticeable discrepancies for the estimated temperature trends in the four data groups (radiosonde, reanalysis, CMIP3 and CMIP5), although similarities can be observed. Compared to the CMIP3 model simulations, the simulations in some of the CMIP5 models were improved. The CMIP5 models displayed a negative temperature trend in the stratosphere closer to the strong negative trend seen in the observations. However, the positive tropospheric trend in the tropics is overestimated by the CMIP5 models relative to CMIP3 models. While some of the models produce temperature trend patterns more highly correlated with the observed patterns in CMIP5, the other models (such as CCSM4 and IPSL_CM5A-LR) exhibit the reverse tendency. The CMIP5 temperature trend uncertainty was significantly reduced in most areas, especially in the Arctic and Antarctic stratosphere, compared to the CMIP3 simulations. Similar to the CMIP3, the CMIP5 simulations overestimated the tropospheric warming in the tropics and Southern Hemisphere and underestimated the stratospheric cooling. The crossover point where tropospheric warming changes into stratospheric cooling occurred near 100 hPa in the tropics, which is higher than in the radiosonde and reanalysis data. The result is likely related to the overestimation of convective activity over the tropical areas in both the CMIP3 and CMIP5 models. Generally, for the temperature trend estimates associated with the numerical models including the reanalyses and global climate models, the uncertainty in the stratosphere is much larger than that in the troposphere, and the uncertainty in the Antarctic is the largest. In addition, note that the reanalyses show the largest uncertainty in the lower tropical stratosphere, and the CMIP3 simulations show the largest uncertainty in both the south and north polar regions.

scholarly journals Comparison of Monthly Temperature Extremes Simulated by CMIP3 and CMIP5 Models

2013 ◽  
Vol 26 (19) ◽  
pp. 7692-7707 ◽  
Author(s):  
Yao Yao ◽  
Yong Luo ◽  
Jianbin Huang ◽  
Zongci Zhao
Keyword(s):  
Coupled Model ◽  
Cmip5 Models ◽  
Temperature Extremes ◽  
Extreme Temperatures ◽  
Model Intercomparison ◽  
Annual Cycles ◽  
Monthly Temperature ◽  
Intercomparison Project ◽  
Cmip3 Ensemble ◽  
Cold Extremes

Abstract The extreme monthly-mean temperatures simulated by 28 models in the fifth phase of the Coupled Model Intercomparison Project (CMIP5) are evaluated and compared with those from 24 models in the third phase of the Coupled Model Intercomparison Project (CMIP3). Comparisons with observations and reanalyses indicate that the models from both CMIP3 and CMIP5 perform well in simulating temperature extremes, which are expressed as 20-yr return values. When the climatological annual cycle is removed, the ensemble spread in CMIP5 is smaller than that in CMIP3. Benefitting from a higher resolution, the CMIP5 models perform better at simulating extreme temperatures on the local gridcell scale. The CMIP5 representative concentration pathway (RCP4.5) and CMIP3 B1 experiments project a similar change pattern in the near future for both warm and cold extremes, and the pattern is in agreement with that of the seasonal extremes. By the late twenty-first century, the changes in monthly temperature extremes projected under the three CMIP3 (B1, A1B, and A2) and two CMIP5 (RCP4.5 and RCP8.5) scenarios generally follow the changes in climatological annual cycles, which is consistent with previous studies on daily extremes. Compared with the CMIP3 ensemble, the CMIP5 ensemble shows a larger intermodel uncertainty with regard to the change in cold extremes in snow-covered regions. Enhanced changes in extreme temperatures that exceed the global mean warming are found in regions where the retreat of snow (or the soil moisture feedback effect) plays an important role, confirming the findings for daily temperature extremes.

Sign in / Sign up

Export Citation Format

Share Document