scholarly journals The Generalized Discrimination Score for Ensemble Forecasts

2011 ◽  
Vol 139 (9) ◽  
pp. 3069-3074 ◽  
Author(s):  
Andreas P. Weigel ◽  
Simon J. Mason
Keyword(s):  
Binary Outcomes ◽  
Continuous Variables ◽  
Discrimination Score ◽  
Ensemble Forecasts ◽  
Probabilistic Forecasts ◽  
Forecast Quality

This article refers to the study of Mason and Weigel, where the generalized discrimination score D has been introduced. This score quantifies whether a set of observed outcomes can be correctly discriminated by the corresponding forecasts (i.e., it is a measure of the skill attribute of discrimination). Because of its generic definition, D can be adapted to essentially all relevant verification contexts, ranging from simple yes–no forecasts of binary outcomes to probabilistic forecasts of continuous variables. For most of these cases, Mason and Weigel have derived expressions for D, many of which have turned out to be equivalent to scores that are already known under different names. However, no guidance was provided on how to calculate D for ensemble forecasts. This gap is aggravated by the fact that there are currently very few measures of forecast quality that could be directly applied to ensemble forecasts without requiring that probabilities be derived from the ensemble members prior to verification. This study seeks to close this gap. A definition is proposed of how ensemble forecasts can be ranked; the ranks of the ensemble forecasts can then be used as a basis for attempting to discriminate between corresponding observations. Given this definition, formulations of D are derived that are directly applicable to ensemble forecasts.

scholarly journals Short-Range Ensemble Forecasts of Precipitation Type

2005 ◽  
Vol 20 (4) ◽  
pp. 609-626 ◽  
Author(s):  
Matthew S. Wandishin ◽  
Michael E. Baldwin ◽  
Steven L. Mullen ◽  
John V. Cortinas
Keyword(s):  
Short Range ◽  
Ensemble Forecast ◽  
Ensemble Forecasting ◽  
Precipitation Forecast ◽  
Initial Condition ◽  
Forecast System ◽  
Ensemble Forecasts ◽  
Probabilistic Forecasts ◽  
Precipitation Type ◽  
Forecast Quality

Abstract Short-range ensemble forecasting is extended to a critical winter weather problem: forecasting precipitation type. Forecast soundings from the operational NCEP Short-Range Ensemble Forecast system are combined with five precipitation-type algorithms to produce probabilistic forecasts from January through March 2002. Thus the ensemble combines model diversity, initial condition diversity, and postprocessing algorithm diversity. All verification numbers are conditioned on both the ensemble and observations recording some form of precipitation. This separates the forecast of type from the yes–no precipitation forecast. The ensemble is very skillful in forecasting rain and snow but it is only moderately skillful for freezing rain and unskillful for ice pellets. However, even for the unskillful forecasts the ensemble shows some ability to discriminate between the different precipitation types and thus provides some positive value to forecast users. Algorithm diversity is shown to be as important as initial condition diversity in terms of forecast quality, although neither has as big an impact as model diversity. The algorithms have their individual strengths and weaknesses, but no algorithm is clearly better or worse than the others overall.

scholarly journals A Generic Forecast Verification Framework for Administrative Purposes

2009 ◽  
Vol 137 (1) ◽  
pp. 331-349 ◽  
Author(s):  
Simon J. Mason ◽  
Andreas P. Weigel
Keyword(s):  
General Public ◽  
Forced Choice ◽  
Continuous Variables ◽  
Forecast Verification ◽  
Complex Information ◽  
Intuitive Appeal ◽  
Probabilistic Forecasts ◽  
Dichotomous Outcomes ◽  
Forecast Quality ◽  
Random Guessing

Abstract There are numerous reasons for calculating forecast verification scores, and considerable attention has been given to designing and analyzing the properties of scores that can be used for scientific purposes. Much less attention has been given to scores that may be useful for administrative reasons, such as communicating changes in forecast quality to bureaucrats and providing indications of forecast quality to the general public. The two-alternative forced choice (2AFC) test is proposed as a scoring procedure that is sufficiently generic to be usable on forecasts ranging from simple yes–no forecasts of dichotomous outcomes to forecasts of continuous variables, and can be used with deterministic or probabilistic forecasts without seriously reducing the more complex information when available. Although, as with any single verification score, the proposed test has limitations, it does have broad intuitive appeal in that the expected score of an unskilled set of forecasts (random guessing or perpetually identical forecasts) is 50%, and is interpretable as an indication of how often the forecasts are correct, even when the forecasts are expressed probabilistically and/or the observations are not discrete.

Ensemble Probabilistic Prediction of a Mesoscale Convective System and Associated Polarimetric Radar Variables Using Single-Moment and Double-Moment Microphysics Schemes and EnKF Radar Data Assimilation

2017 ◽  
Vol 145 (6) ◽  
pp. 2257-2279 ◽  
Author(s):  
Bryan J. Putnam ◽  
Ming Xue ◽  
Youngsun Jung ◽  
Nathan A. Snook ◽  
Guifu Zhang
Keyword(s):  
Mesoscale Convective System ◽  
Radar Data ◽  
Convective Precipitation ◽  
Convective System ◽  
Ensemble Forecasts ◽  
Verification Methods ◽  
Probabilistic Forecasts ◽  
Mesoscale Convective ◽  
Single Moment ◽  
Double Moment

Abstract Ensemble-based probabilistic forecasts are performed for a mesoscale convective system (MCS) that occurred over Oklahoma on 8–9 May 2007, initialized from ensemble Kalman filter analyses using multinetwork radar data and different microphysics schemes. Two experiments are conducted, using either a single-moment or double-moment microphysics scheme during the 1-h-long assimilation period and in subsequent 3-h ensemble forecasts. Qualitative and quantitative verifications are performed on the ensemble forecasts, including probabilistic skill scores. The predicted dual-polarization (dual-pol) radar variables and their probabilistic forecasts are also evaluated against available dual-pol radar observations, and discussed in relation to predicted microphysical states and structures. Evaluation of predicted reflectivity (Z) fields shows that the double-moment ensemble predicts the precipitation coverage of the leading convective line and stratiform precipitation regions of the MCS with higher probabilities throughout the forecast period compared to the single-moment ensemble. In terms of the simulated differential reflectivity (ZDR) and specific differential phase (KDP) fields, the double-moment ensemble compares more realistically to the observations and better distinguishes the stratiform and convective precipitation regions. The ZDR from individual ensemble members indicates better raindrop size sorting along the leading convective line in the double-moment ensemble. Various commonly used ensemble forecast verification methods are examined for the prediction of dual-pol variables. The results demonstrate the challenges associated with verifying predicted dual-pol fields that can vary significantly in value over small distances. Several microphysics biases are noted with the help of simulated dual-pol variables, such as substantial overprediction of KDP values in the single-moment ensemble.

scholarly journals Trade-offs between individual and ensemble forecasts of an emerging infectious disease

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Rachel J. Oidtman ◽  
Elisa Omodei ◽  
Moritz U. G. Kraemer ◽  
Carlos A. Castañeda-Orjuela ◽  
Erica Cruz-Rivera ◽  
...  
Keyword(s):  
Human Mobility ◽  
Emerging Infectious Diseases ◽  
Spatiotemporal Variation ◽  
Emerging Pathogens ◽  
Ensemble Forecasts ◽  
Transmission Potential ◽  
Trade Offs ◽  
Probabilistic Forecasts ◽  
Model Structures ◽  
Answering Questions

AbstractProbabilistic forecasts play an indispensable role in answering questions about the spread of newly emerged pathogens. However, uncertainties about the epidemiology of emerging pathogens can make it difficult to choose among alternative model structures and assumptions. To assess the potential for uncertainties about emerging pathogens to affect forecasts of their spread, we evaluated the performance 16 forecasting models in the context of the 2015-2016 Zika epidemic in Colombia. Each model featured a different combination of assumptions about human mobility, spatiotemporal variation in transmission potential, and the number of virus introductions. We found that which model assumptions had the most ensemble weight changed through time. We additionally identified a trade-off whereby some individual models outperformed ensemble models early in the epidemic, but on average the ensembles outperformed all individual models. Our results suggest that multiple models spanning uncertainty across alternative assumptions are necessary to obtain robust forecasts for emerging infectious diseases.

scholarly journals Storm-Scale Data Assimilation and Ensemble Forecasts for the 27 April 2011 Severe Weather Outbreak in Alabama

2015 ◽  
Vol 143 (8) ◽  
pp. 3044-3066 ◽  
Author(s):  
Nusrat Yussouf ◽  
David C. Dowell ◽  
Louis J. Wicker ◽  
Kent H. Knopfmeier ◽  
Dustan M. Wheatley
Keyword(s):  
Short Range ◽  
Type System ◽  
Severe Weather ◽  
Forecast Errors ◽  
Model Errors ◽  
Ensemble Forecasts ◽  
Low Level ◽  
Design Storm ◽  
Probabilistic Forecasts ◽  
Arw Model

Abstract As part of NOAA’s Warn-on-Forecast (WoF) initiative, a multiscale ensemble-based assimilation and prediction system is developed using the WRF-ARW model and DART assimilation software. To evaluate the capabilities of the system, retrospective short-range probabilistic storm-scale (convection allowing) ensemble analyses and forecasts are produced for the 27 April 2011 Alabama severe weather outbreak. Results indicate that the storm-scale ensembles are able to analyze the observed storms with strong low-level rotation at approximately the correct locations and to retain the supercell structures during the 0–1-h forecasts with reasonable accuracy. The system predicts the low-level mesocyclones of significant isolated tornadic supercells that align well with the locations of radar-derived rotation. For cases with multiple interacting storms in close proximity, the system tends to produce more variability in mesocyclone forecasts from one initialization time to the next until the observations show the dominance of one of the cells. The short-range ensemble probabilistic forecasts obtained from this continuous 5-min storm-scale 6-h-long update system demonstrate the potential of a frequently updated, high-resolution NWP system that could be used to extend severe weather warning lead times. This study also demonstrates the challenges associated with developing a WoF-type system. The results motivate future work to reduce model errors associated with storm motion and spurious cells, and to design storm-scale ensembles that better represent typical 1-h forecast errors.

scholarly journals Impact of a Stochastic Kinetic Energy Backscatter Scheme on Warm Season Convection-Allowing Ensemble Forecasts

2016 ◽  
Vol 144 (5) ◽  
pp. 1887-1908 ◽  
Author(s):  
Jeffrey D. Duda ◽  
Xuguang Wang ◽  
Fanyou Kong ◽  
Ming Xue ◽  
Judith Berner
Keyword(s):  
Kinetic Energy ◽  
Wrf Model ◽  
Warm Season ◽  
Weather Research And Forecasting ◽  
Potential Utility ◽  
Ensemble Forecasts ◽  
Probabilistic Forecasts ◽  
Ensemble Mean ◽  
Central United States ◽  
Surface Observations

The efficacy of a stochastic kinetic energy backscatter (SKEB) scheme to improve convection-allowing probabilistic forecasts was studied. While SKEB has been explored for coarse, convection-parameterizing models, studies of SKEB for convective scales are limited. Three ensembles were compared. The SKMP ensemble used mixed physics with the SKEB scheme, whereas the MP ensemble was configured identically but without using the SKEB scheme. The SK ensemble used the SKEB scheme with no physics diversity. The experiment covered May 2013 over the central United States on a 4-km Weather Research and Forecasting (WRF) Model domain. The SKEB scheme was successful in increasing the spread in all fields verified, especially mid- and upper-tropospheric fields. Additionally, the rmse of the ensemble mean was maintained or reduced, in some cases significantly. Rank histograms in the SKMP ensemble were flatter than those in the MP ensemble, indicating the SKEB scheme produces a less underdispersive forecast distribution. Some improvement was seen in probabilistic precipitation forecasts, particularly when examining Brier scores. Verification against surface observations agree with verification against Rapid Refresh (RAP) model analyses, showing that probabilistic forecasts for 2-m temperature, 2-m dewpoint, and 10-m winds were also improved using the SKEB scheme. The SK ensemble gave competitive forecasts for some fields. The SK ensemble had reduced spread compared to the MP ensemble at the surface due to the lack of physics diversity. These results suggest the potential utility of mixed physics plus the SKEB scheme in the design of convection-allowing ensemble forecasts.

scholarly journals Toward 1-km Ensemble Forecasts over Large Domains

2017 ◽  
Vol 145 (8) ◽  
pp. 2943-2969 ◽  
Author(s):  
Craig S. Schwartz ◽  
Glen S. Romine ◽  
Kathryn R. Fossell ◽  
Ryan A. Sobash ◽  
Morris L. Weisman
Keyword(s):  
The United States ◽  
Grid Spacing ◽  
Ensemble Forecasts ◽  
Convective Systems ◽  
Object Based ◽  
Probabilistic Forecasts ◽  
Mesoscale Convective ◽  
Computational Resources ◽  
Horizontal Grid ◽  
Better Than

Precipitation forecasts from convection-allowing ensembles with 3- and 1-km horizontal grid spacing were evaluated between 15 May and 15 June 2013 over central and eastern portions of the United States. Probabilistic forecasts produced from 10- and 30-member, 3-km ensembles were consistently better than forecasts from individual 1-km ensemble members. However, 10-member, 1-km probabilistic forecasts usually were best, especially over the first 12 h and at rainfall rates ≥ 5.0 mm h−1 at later times. Further object-based investigation revealed that better 1-km forecasts at heavier rainfall rates were associated with more accurate placement of mesoscale convective systems compared to 3-km forecasts. The collective results indicate promise for 1-km ensembles once computational resources can support their operational implementation.

scholarly journals Statistical Postprocessing of Ensemble Precipitation Forecasts by Fitting Censored, Shifted Gamma Distributions*

2015 ◽  
Vol 143 (11) ◽  
pp. 4578-4596 ◽  
Author(s):  
Michael Scheuerer ◽  
Thomas M. Hamill
Keyword(s):  
Probability Distributions ◽  
Precipitable Water ◽  
Training Dataset ◽  
Ensemble Forecasts ◽  
Gamma Distributions ◽  
Predictive Probability ◽  
Skill Scores ◽  
Probabilistic Forecasts ◽  
The Mean ◽  
Set Up

Abstract A parametric statistical postprocessing method is presented that transforms raw (and frequently biased) ensemble forecasts from the Global Ensemble Forecast System (GEFS) into reliable predictive probability distributions for precipitation accumulations. Exploratory analysis based on 12 years of reforecast data and ⅛° climatology-calibrated precipitation analyses shows that censored, shifted gamma distributions can well approximate the conditional distribution of observed precipitation accumulations given the ensemble forecasts. A nonhomogeneous regression model is set up to link the parameters of this distribution to ensemble statistics that summarize the mean and spread of predicted precipitation amounts within a certain neighborhood of the location of interest, and in addition the predicted mean of precipitable water. The proposed method is demonstrated with precipitation reforecasts over the conterminous United States using common metrics such as Brier skill scores and reliability diagrams. It yields probabilistic forecasts that are reliable, highly skillful, and sharper than the previously demonstrated analog procedure. In situations with limited predictability, increasing the size of the neighborhood within which ensemble forecasts are considered as predictors can further improve forecast skill. It is found, however, that even a parametric postprocessing approach crucially relies on the availability of a sufficiently large training dataset.

scholarly journals Hydrometeorological Short-Range Ensemble Forecasts in Complex Terrain. Part I: Meteorological Evaluation

2008 ◽  
Vol 23 (4) ◽  
pp. 533-556 ◽  
Author(s):  
Doug McCollor ◽  
Roland Stull
Keyword(s):  
Complex Terrain ◽  
Short Range ◽  
Economic Value ◽  
Companion Paper ◽  
High Mountains ◽  
Ensemble Forecasts ◽  
Probabilistic Forecasts ◽  
Frontal Systems ◽  
Precipitation Gauge ◽  
Horizontal Grid

Abstract This paper addresses the question of whether it is better to include lower-resolution members of a nested suite of numerical precipitation forecasts to increase ensemble size, or to utilize high-resolution members only to maximize forecast details in regions of complex terrain. A short-range ensemble forecast (SREF) system is formed from three models running in nested configurations at 108-, 36-, 12-, and 4-km horizontal grid spacings. The forecasts are sampled at 27 precipitation-gauge locations, representing 15 pluvial watersheds in southwestern British Columbia, Canada. This is a region of complex topography characterized by high mountains, glaciers, fjords, and land–ocean boundaries. Matching forecast–observation pairs are analyzed for two consecutive wet seasons: October 2003–March 2004 and October 2004–March 2005. The northwest coast of North America is typically subject to intense landfalling Pacific cyclones and frontal systems during these months. Using forecast analysis tools that are well designed for SREF systems, it is found that utilizing the full suite of ensemble members, including the lowest-resolution members, produced the highest quality probabilistic forecasts of precipitation. A companion paper assesses the economic value of SREF probabilistic forecasts for hydroelectric operations.

Sign in / Sign up

Export Citation Format

Share Document