scholarly journals On the predictability of outliers in ensemble forecasts

2012 ◽  
Vol 8 (1) ◽  
pp. 53-57
Author(s):  
S. Siegert ◽  
J. Bröcker ◽  
H. Kantz
Keyword(s):  
Logistic Regression ◽  
Weather Prediction ◽  
Weather Conditions ◽  
Skill Score ◽  
Ensemble Forecast ◽  
Ensemble Member ◽  
Base Rate ◽  
Ensemble Forecasts ◽  
Probabilistic Forecasts ◽  
Ensemble Interpretation

Abstract. In numerical weather prediction, ensembles are used to retrieve probabilistic forecasts of future weather conditions. We consider events where the verification is smaller than the smallest, or larger than the largest ensemble member of a scalar ensemble forecast. These events are called outliers. In a statistically consistent K-member ensemble, outliers should occur with a base rate of 2/(K+1). In operational ensembles this base rate tends to be higher. We study the predictability of outlier events in terms of the Brier Skill Score and find that forecast probabilities can be calculated which are more skillful than the unconditional base rate. This is shown analytically for statistically consistent ensembles. Using logistic regression, forecast probabilities for outlier events in an operational ensemble are calculated. These probabilities exhibit positive skill which is quantitatively similar to the analytical results. Possible causes of these results as well as their consequences for ensemble interpretation are discussed.

scholarly journals Beyond univariate calibration: verifying spatial structure in ensembles of forecast fields

2020 ◽  
Vol 27 (3) ◽  
pp. 411-427
Author(s):  
Josh Jacobson ◽  
William Kleiber ◽  
Michael Scheuerer ◽  
Joseph Bellier
Keyword(s):  
Spatial Structure ◽  
Weather Conditions ◽  
Heavy Precipitation ◽  
Ensemble Forecast ◽  
Ensemble Member ◽  
Precipitation Forecast ◽  
Ensemble Forecasts ◽  
Adequate Representation ◽  
Grid Points ◽  
High Winds

Abstract. Most available verification metrics for ensemble forecasts focus on univariate quantities. That is, they assess whether the ensemble provides an adequate representation of the forecast uncertainty about the quantity of interest at a particular location and time. For spatially indexed ensemble forecasts, however, it is also important that forecast fields reproduce the spatial structure of the observed field and represent the uncertainty about spatial properties such as the size of the area for which heavy precipitation, high winds, critical fire weather conditions, etc., are expected. In this article we study the properties of the fraction of threshold exceedance (FTE) histogram, a new diagnostic tool designed for spatially indexed ensemble forecast fields. Defined as the fraction of grid points where a prescribed threshold is exceeded, the FTE is calculated for the verification field and separately for each ensemble member. It yields a projection of a – possibly high-dimensional – multivariate quantity onto a univariate quantity that can be studied with standard tools like verification rank histograms. This projection is appealing since it reflects a spatial property that is intuitive and directly relevant in applications, though it is not obvious whether the FTE is sufficiently sensitive to misrepresentation of spatial structure in the ensemble. In a comprehensive simulation study we find that departures from uniformity of the FTE histograms can indeed be related to forecast ensembles with biased spatial variability and that these histograms detect shortcomings in the spatial structure of ensemble forecast fields that are not obvious by eye. For demonstration, FTE histograms are applied in the context of spatially downscaled ensemble precipitation forecast fields from NOAA's Global Ensemble Forecast System.

scholarly journals Beyond Univariate Calibration: Verifying Spatial Structure in Ensembles of Forecast Fields

2020 ◽  
Author(s):  
Joshuah Jacobson ◽  
William Kleiber ◽  
Michael Scheuerer ◽  
Joseph Bellier
Keyword(s):  
Spatial Structure ◽  
Weather Conditions ◽  
Heavy Precipitation ◽  
Ensemble Forecast ◽  
Ensemble Member ◽  
Precipitation Forecast ◽  
Ensemble Forecasts ◽  
Spatial Properties ◽  
Adequate Representation ◽  
High Winds

Abstract. Most available verification metrics for ensemble forecasts focus on univariate quantities. That is, they assess whether the ensemble provides an adequate representation of the forecast uncertainty about the quantity of interest at a particular location and time. For spatially-indexed ensemble forecasts, however, it is also important that forecast fields reproduce the spatial structure of the observed field, and represent the uncertainty about spatial properties such as the size of the area for which heavy precipitation, high winds, critical fire weather conditions, etc. are expected. In this article we study the properties of a new diagnostic tool designed for spatially-indexed ensemble forecast fields. The metric is based on a level-crossing statistic that we term the fraction of threshold exceedance (FTE), and is calculated for the verification field, and separately for each ensemble member. The FTE yields a projection of a – possibly high-dimensional – multivariate quantity onto a univariate quantity that can be studied with standard tools like verification rank histograms. This projection is appealing since it reflects a spatial property that is intuitive and directly relevant in applications, though it is not obvious whether the FTE is sufficiently sensitive to misrepresentation of spatial structure in the ensemble. In a comprehensive simulation study we find that departures from uniformity of these so called FTE histograms can be indeed be related to forecast ensembles with biased spatial variability, and that these histograms detect shortcomings in the spatial structure of ensemble forecast fields that are not obvious by eye. For demonstration, FTE histograms are applied in the context of spatially downscaled ensemble precipitation forecast fields from NOAA's Global Ensemble Forecast System.

scholarly journals A Transformed Lagged Ensemble Forecasting Technique for Increasing Ensemble Size

2007 ◽  
Vol 135 (4) ◽  
pp. 1424-1438 ◽  
Author(s):  
Andrew R. Lawrence ◽  
James A. Hansen
Keyword(s):  
Prediction Models ◽  
Weather Prediction ◽  
Ensemble Forecast ◽  
Ensemble Forecasting ◽  
Limited Area ◽  
Ensemble Size ◽  
Probabilistic Forecasting ◽  
Ensemble Forecasts ◽  
Chaotic Model ◽  
The Impact

Abstract An ensemble-based data assimilation approach is used to transform old ensemble forecast perturbations with more recent observations for the purpose of inexpensively increasing ensemble size. The impact of the transformations are propagated forward in time over the ensemble’s forecast period without rerunning any models, and these transformed ensemble forecast perturbations can be combined with the most recent ensemble forecast to sensibly increase forecast ensemble sizes. Because the transform takes place in perturbation space, the transformed perturbations must be centered on the ensemble mean from the most recent forecasts. Thus, the benefit of the approach is in terms of improved ensemble statistics rather than improvements in the mean. Larger ensemble forecasts can be used for numerous purposes, including probabilistic forecasting, targeted observations, and to provide boundary conditions to limited-area models. This transformed lagged ensemble forecasting approach is explored and is shown to give positive results in the context of a simple chaotic model. By incorporating a suitable perturbation inflation factor, the technique was found to generate forecast ensembles whose skill were statistically comparable to those produced by adding nonlinear model integrations. Implications for ensemble forecasts generated by numerical weather prediction models are briefly discussed, including multimodel ensemble forecasting.

scholarly journals Post-Processing and Evaluation of Precipitation Ensemble Forecast under Multiple Schemes in Beijiang River Basin

Water ◽  
2020 ◽  
Vol 12 (9) ◽  
pp. 2631
Author(s):  
Xinchi Chen ◽  
Xiaohong Chen ◽  
Dong Huang ◽  
Huamei Liu
Keyword(s):  
Time Scales ◽  
Weather Prediction ◽  
Forecast Accuracy ◽  
Skill Score ◽  
Ensemble Forecast ◽  
Efficiency Coefficient ◽  
Forecast System ◽  
Factors Affecting ◽  
Hydrological Forecasting ◽  
Better Than

Precipitation is one of the most important factors affecting the accuracy and uncertainty of hydrological forecasting. Considerable progress has been made in numerical weather prediction after decades of development, but the forecast products still cannot be used directly for hydrological forecasting. This study used ensemble pro-processor (EPP) to post-process the Global Ensemble Forecast System (GEFS) and Climate Forecast System version 2 (CFSv2) with four designed schemes, and then integrated them to investigate the forecast accuracy in longer time scales based on the best scheme. Many indices such as correlation coefficient, Nash efficiency coefficient, rank histogram, and continuous ranked probability skill score were used to evaluate the results in different aspects. The results show that EPP can improve the accuracy of raw forecast significantly, and the scheme considering cumulative forecast precipitation is better than that only considers single-day forecast. Moreover, the scheme that considers some observed precipitation would help to improve the accuracy and reduce the uncertainty. In terms of medium- and long-term forecasts, the integrated forecast based on GEFS and CFSv2 after post-processed would be better than CFSv2 significantly. The results of this study would be a very important demonstration to remove the deviation of ensemble forecast and improve the accuracy of hydrological forecasting in different time scales.

scholarly journals Why do some probabilistic forecasts lack reliability?

2019 ◽  
Vol 9 ◽  
pp. A17
Author(s):  
Yûki Kubo
Keyword(s):  
Solar Flare ◽  
Skill Score ◽  
Base Rate ◽  
Probabilistic Forecast ◽  
Partial Answer ◽  
Sufficient Condition ◽  
Forecast System ◽  
Threshold Probability ◽  
Probabilistic Forecasts ◽  
Forecast Models

In this work, we investigate the reliability of the probabilistic binary forecast. We mathematically prove that a necessary, but not sufficient, condition for achieving a reliable probabilistic forecast is maximizing the Peirce Skill Score (PSS) at the threshold probability of the climatological base rate. The condition is confirmed by using artificially synthesized forecast–outcome pair data and previously published probabilistic solar flare forecast models. The condition gives a partial answer as to why some probabilistic forecast system lack reliability, because the system, which does not satisfy the proved condition, can never be reliable. Therefore, the proved condition is very important for the developers of a probabilistic forecast system. The result implies that those who want to develop a reliable probabilistic forecast system must adjust or train the system so as to maximize PSS near the threshold probability of the climatological base rate.

scholarly journals Short-Range Ensemble Forecasts of Precipitation Type

2005 ◽  
Vol 20 (4) ◽  
pp. 609-626 ◽  
Author(s):  
Matthew S. Wandishin ◽  
Michael E. Baldwin ◽  
Steven L. Mullen ◽  
John V. Cortinas
Keyword(s):  
Short Range ◽  
Ensemble Forecast ◽  
Ensemble Forecasting ◽  
Precipitation Forecast ◽  
Initial Condition ◽  
Forecast System ◽  
Ensemble Forecasts ◽  
Probabilistic Forecasts ◽  
Precipitation Type ◽  
Forecast Quality

Abstract Short-range ensemble forecasting is extended to a critical winter weather problem: forecasting precipitation type. Forecast soundings from the operational NCEP Short-Range Ensemble Forecast system are combined with five precipitation-type algorithms to produce probabilistic forecasts from January through March 2002. Thus the ensemble combines model diversity, initial condition diversity, and postprocessing algorithm diversity. All verification numbers are conditioned on both the ensemble and observations recording some form of precipitation. This separates the forecast of type from the yes–no precipitation forecast. The ensemble is very skillful in forecasting rain and snow but it is only moderately skillful for freezing rain and unskillful for ice pellets. However, even for the unskillful forecasts the ensemble shows some ability to discriminate between the different precipitation types and thus provides some positive value to forecast users. Algorithm diversity is shown to be as important as initial condition diversity in terms of forecast quality, although neither has as big an impact as model diversity. The algorithms have their individual strengths and weaknesses, but no algorithm is clearly better or worse than the others overall.

Regime-dependent statistical post-processing of ensemble forecasts

2020 ◽  
Author(s):  
Sam Allen ◽  
Christopher Ferro ◽  
Frank Kwasniok
Keyword(s):  
Initial Conditions ◽  
Weather Prediction ◽  
Systematic Errors ◽  
Ensemble Forecast ◽  
State Variables ◽  
Post Processing ◽  
Ensemble Forecasts ◽  
Nwp Model ◽  
Processing Techniques ◽  
Geographical Locations

<p>A number of realizations of one or more numerical weather prediction (NWP) models, initialised at a variety of initial conditions, compose an ensemble forecast. These forecasts exhibit systematic errors and biases that can be corrected by statistical post-processing. Post-processing yields calibrated forecasts by analysing the statistical relationship between historical forecasts and their corresponding observations. This article aims to extend post processing methodology to incorporate atmospheric circulation. The circulation, or flow, is largely responsible for the weather that we experience and it is hypothesized here that relationships between the NWP model and the atmosphere depend upon the prevailing flow. Numerous studies have focussed on the tendency of this flow to reduce to a set of recognisable arrangements, known as regimes, which recur and persist at fixed geographical locations. This dynamical phenomenon allows the circulation to be categorized into a small number of regime states. In a highly idealized model of the atmosphere, the Lorenz ‘96 system, ensemble forecasts are subjected to well-known post-processing techniques conditional on the system's underlying regime. Two different variables, one of the state variables and one related to the energy of the system, are forecasted and considerable improvements in forecast skill upon standard post-processing are seen when the distribution of the predictand varies depending on the regime. Advantages of this approach and its inherent challenges are discussed, along with potential extensions for operational forecasters.</p>

scholarly journals Dispersion of aerosol particles in the free atmosphere using ensemble forecasts

2013 ◽  
Vol 20 (5) ◽  
pp. 759-770 ◽  
Author(s):  
T. Haszpra ◽  
I. Lagzi ◽  
T. Tél
Keyword(s):  
Aerosol Particles ◽  
Geographical Area ◽  
Ensemble Forecast ◽  
Ensemble Member ◽  
Threshold Concentration ◽  
Free Atmosphere ◽  
Ensemble Forecasts ◽  
Centers Of Mass ◽  
Mean Square Distance ◽  
Observation Period

Abstract. The dispersion of aerosol particle pollutants is studied using 50 members of an ensemble forecast in the example of a hypothetical free atmospheric emission above Fukushima over a period of 2.5 days. Considerable differences are found among the dispersion predictions of the different ensemble members, as well as between the ensemble mean and the deterministic result at the end of the observation period. The variance is found to decrease with the particle size. The geographical area where a threshold concentration is exceeded in at least one ensemble member expands to a 5–10 times larger region than the area from the deterministic forecast, both for air column "concentration" and in the "deposition" field. We demonstrate that the root-mean-square distance of any particle from its own clones in the ensemble members can reach values on the order of one thousand kilometers. Even the centers of mass of the particle cloud of the ensemble members deviate considerably from that obtained by the deterministic forecast. All these indicate that an investigation of the dispersion of aerosol particles in the spirit of ensemble forecast contains useful hints for the improvement of risk assessment.

scholarly journals Summary Verification Measures and Their Interpretation for Ensemble Forecasts

2011 ◽  
Vol 139 (9) ◽  
pp. 3075-3089 ◽  
Author(s):  
A. Allen Bradley ◽  
Stuart S. Schwartz
Keyword(s):  
Weighted Average ◽  
Threshold Value ◽  
Skill Score ◽  
Ensemble Forecast ◽  
Ensemble Prediction ◽  
Threshold Values ◽  
Ensemble Forecasts ◽  
Ensemble Prediction System ◽  
Prediction Systems ◽  
Forecast Quality

Ensemble prediction systems produce forecasts that represent the probability distribution of a continuous forecast variable. Most often, the verification problem is simplified by transforming the ensemble forecast into probability forecasts for discrete events, where the events are defined by one or more threshold values. Then, skill is evaluated using the mean-square error (MSE; i.e., Brier) skill score for binary events, or the ranked probability skill score (RPSS) for multicategory events. A framework is introduced that generalizes this approach, by describing the forecast quality of ensemble forecasts as a continuous function of the threshold value. Viewing ensemble forecast quality this way leads to the interpretation of the RPSS and the continuous ranked probability skill score (CRPSS) as measures of the weighted-average skill over the threshold values. It also motivates additional measures, derived to summarize other features of a continuous forecast quality function, which can be interpreted as descriptions of the function’s geometric shape. The measures can be computed not only for skill, but also for skill score decompositions, which characterize the resolution, reliability, discrimination, and other aspects of forecast quality. Collectively, they provide convenient metrics for comparing the performance of an ensemble prediction system at different locations, lead times, or issuance times, or for comparing alternative forecasting systems.

Probabilistic Wind Vector Forecasting Using Ensembles and Bayesian Model Averaging

2013 ◽  
Vol 141 (6) ◽  
pp. 2107-2119 ◽  
Author(s):  
J. McLean Sloughter ◽  
Tilmann Gneiting ◽  
Adrian E. Raftery
Keyword(s):  
Bayesian Model ◽  
Bayesian Model Averaging ◽  
Prediction Models ◽  
Weather Prediction ◽  
Model Averaging ◽  
Wind Vector ◽  
Bivariate Distributions ◽  
Ensemble Forecasts ◽  
Probabilistic Forecasts ◽  
Wide Range

Abstract Probabilistic forecasts of wind vectors are becoming critical as interest grows in wind as a clean and renewable source of energy, in addition to a wide range of other uses, from aviation to recreational boating. Unlike other common forecasting problems, which deal with univariate quantities, statistical approaches to wind vector forecasting must be based on bivariate distributions. The prevailing paradigm in weather forecasting is to issue deterministic forecasts based on numerical weather prediction models. Uncertainty can then be assessed through ensemble forecasts, where multiple estimates of the current state of the atmosphere are used to generate a collection of deterministic predictions. Ensemble forecasts are often uncalibrated, however, and Bayesian model averaging (BMA) is a statistical way of postprocessing these forecast ensembles to create calibrated predictive probability density functions (PDFs). It represents the predictive PDF as a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights reflect the forecasts’ relative contributions to predictive skill over a training period. In this paper the authors extend the BMA methodology to use bivariate distributions, enabling them to provide probabilistic forecasts of wind vectors. The BMA method is applied to 48-h-ahead forecasts of wind vectors over the North American Pacific Northwest in 2003 using the University of Washington mesoscale ensemble and is shown to provide better-calibrated probabilistic forecasts than the raw ensemble, which are also sharper than probabilistic forecasts derived from climatology.

Sign in / Sign up

Export Citation Format

Share Document