Learning uncertainty models from weather forecast performance databases using quantile regression

Abstract Using quantile regression, we evaluate the forecasting performance of futures prices in the soybean complex. The procedure provides a more complete picture of the distribution of forecasts than mainstream methods that only focus on central tendency measures. Forecast performance differs by location in the futures price distribution. Futures forecast perform well in the centre of the distribution. However, futures prices tend to over-forecast when futures prices are high and under-forecast when futures prices are low, suggesting that futures prices tend to under-estimate price reversion towards the centre of the distribution. Forecast errors are larger when futures prices are high. The findings are related to theories in the literature used to explain pricing bias, and their implications for market participants are discussed.

Download Full-text

Exploring the potential of vegetation information for improving weather forecast performance

10.5194/egusphere-egu21-3105 ◽

2021 ◽

Author(s):

Melissa Ruiz ◽

Sungmin Oh ◽

Rene Orth ◽

Gianpaolo Balsamo

Keyword(s):

Land Surface ◽

Vegetation Index ◽

Weather Forecasting ◽

Forecast Accuracy ◽

Weather Forecast ◽

Spatial And Temporal Variation ◽

Forecast Errors ◽

Forecast Performance ◽

Forecast Models ◽

Moderate Resolution Imaging Spectroradiometer

<p>The quality of weather forecasts is continuously improving for decades. However, increases in forecast skills have slowed down in recent years. This highlights the importance of exploring new avenues towards future forecast system improvements. Until now, (near) real-time information on vegetation anomalies is not used in most forecasting models. Addressing this gap, we explore the potential of the vegetation state for explaining the spatial and temporal variation in forecast accuracy globally across climate regions, seasons, and vegetation types. For this purpose, we employ re-forecasts from the European Centre of Medium-Range Weather Forecasting (ECMWF) and infer the vegetation status through the Enhanced Vegetation Index derived from the Moderate-Resolution Imaging Spectroradiometer (MODIS) satellite observations during the 2000-2019 period. In particular, we focus on land surface variables such as evaporation and temperature to study the relationship between forecast errors and vegetation anomalies.</p><p>The results show a stronger correlation between forecast errors and vegetation anomalies in semi-arid and sub-humid regions during the growing season, which highlights that vegetation information has the potential to help advance weather forecast performance. To put these results into perspective, we will further perform a multivariate analysis to determine the relative roles of vegetation, hydrology and climate in explaining weather forecast errors. Thereby, our results can inform the future development of weather forecast models and underlying data assimilation schemes.</p>

Download Full-text

Performance and robustness of probabilistic river forecasts computed with quantile regression based on multiple independent variables

Hydrology and Earth System Sciences ◽

10.5194/hess-19-3969-2015 ◽

2015 ◽

Vol 19 (9) ◽

pp. 3969-3990 ◽

Cited By ~ 10

Author(s):

F. Hoss ◽

P. S. Fischbeck

Keyword(s):

Quantile Regression ◽

High Reliability ◽

Forecast Error ◽

Skill Score ◽

Water Levels ◽

Training Data ◽

Forecast Errors ◽

Data Set ◽

Forecast Performance ◽

Independent Variables

Abstract. This study applies quantile regression (QR) to predict exceedance probabilities of various water levels, including flood stages, with combinations of deterministic forecasts, past forecast errors and rates of water level rise as independent variables. A computationally cheap technique to estimate forecast uncertainty is valuable, because many national flood forecasting services, such as the National Weather Service (NWS), only publish deterministic single-valued forecasts. The study uses data from the 82 river gauges, for which the NWS' North Central River Forecast Center issues forecasts daily. Archived forecasts for lead times of up to 6 days from 2001 to 2013 were analyzed. Besides the forecast itself, this study uses the rate of rise of the river stage in the last 24 and 48 h and the forecast error 24 and 48 h ago as predictors in QR configurations. When compared to just using the forecast as an independent variable, adding the latter four predictors significantly improved the forecasts, as measured by the Brier skill score and the continuous ranked probability score. Mainly, the resolution increases, as the forecast-only QR configuration already delivered high reliability. Combining the forecast with the other four predictors results in a much less favorable performance. Lastly, the forecast performance does not strongly depend on the size of the training data set but on the year, the river gauge, lead time and event threshold that are being forecast. We find that each event threshold requires a separate configuration or at least calibration.

Download Full-text

Performance and robustness of probabilistic river forecasts computed with quantile regression based on multiple independent variables in the North Central USA

Hydrology and Earth System Sciences Discussions ◽

10.5194/hessd-11-11281-2014 ◽

2014 ◽

Vol 11 (10) ◽

pp. 11281-11333

Author(s):

F. Hoss ◽

P. S. Fischbeck

Keyword(s):

Quantile Regression ◽

High Reliability ◽

Forecast Error ◽

Skill Score ◽

Training Dataset ◽

Forecast Performance ◽

North Central ◽

The North ◽

Rise Rate ◽

Using Data

Abstract. This study further develops the method of quantile regression (QR) to predict exceedance probabilities of flood stages by post-processing forecasts. Using data from the 82 river gages, for which the National Weather Service's North Central River Forecast Center issues forecasts daily, this is the first QR application to US American river gages. Archived forecasts for lead times up to six days from 2001–2013 were analyzed. Earlier implementations of QR used the forecast itself as the only independent variable (Weerts et al., 2011; López López et al., 2014). This study adds the rise rate of the river stage in the last 24 and 48 h and the forecast error 24 and 48 h ago to the QR model. Including those four variables significantly improved the forecasts, as measured by the Brier Skill Score (BSS). Mainly, the resolution increases, as the original QR implementation already delivered high reliability. Combining the forecast with the other four variables results in much less favorable BSSs. Lastly, the forecast performance does not depend on the size of the training dataset, but on the year, the river gage, lead time and event threshold that are being forecast. We find that each event threshold requires a separate model configuration or at least calibration.

Download Full-text

Skewed and Mixture of Gaussian Distributions for Ensemble Postprocessing

Atmosphere ◽

10.3390/atmos12080966 ◽

2021 ◽

Vol 12 (8) ◽

pp. 966

Author(s):

Maxime Taillardat

Keyword(s):

Computational Cost ◽

Weather Forecast ◽

Estimation Methods ◽

Ensemble Forecasts ◽

Forecast Performance ◽

Medium Range Weather Forecast ◽

Analytical Formulas ◽

Ensemble Model Output Statistics ◽

Model Output Statistics ◽

Mixture Of Gaussian

The implementation of statistical postprocessing of ensemble forecasts is increasingly developed among national weather services. The so-called Ensemble Model Output Statistics (EMOS) method, which consists of generating a given distribution whose parameters depend on the raw ensemble, leads to significant improvements in forecast performance for a low computational cost, and so is particularly appealing for reduced performance computing architectures. However, the choice of a parametric distribution has to be sufficiently consistent so as not to lose information on predictability such as multimodalities or asymmetries. Different distributions are applied to the postprocessing of the European Centre for Medium-range Weather Forecast (ECMWF) ensemble forecast of surface temperature. More precisely, a mixture of Gaussian and skewed normal distributions are tried from 3- up to 360-h lead time forecasts, with different estimation methods. For this work, analytical formulas of the continuous ranked probability score have been derived and appropriate link functions are used to prevent overfitting. The mixture models outperform single parametric distributions, especially for the longest lead times. This statement is valid judging both overall performance and tolerance to misspecification.

Download Full-text

Probabilistic Contingency Tables: An Improvement to Verify Probability Forecasts

Weather and Forecasting ◽

10.1175/waf-d-19-0116.1 ◽

2019 ◽

Vol 35 (2) ◽

pp. 609-621 ◽

Cited By ~ 1

Author(s):

Sarah Gold ◽

Edward White ◽

William Roeder ◽

Mike McAleenan ◽

Christine Schubert Kabban ◽

...

Keyword(s):

Contingency Table ◽

Performance Metrics ◽

Weather Forecast ◽

Forecast Verification ◽

Forecast Performance ◽

Binary Event ◽

Probabilistic Forecasts ◽

Verification Process ◽

Statistical Metrics

Abstract The 45th Weather Squadron (45 WS) records daily rain and lightning probabilistic forecasts and the associated binary event outcomes. Subsequently, they evaluate forecast performance and determine necessary adjustments with an established verification process. For deterministic outcomes, weather forecast analysis typically utilizes a traditional contingency table (TCT) for verification; however, the 45 WS uses an alternative tool, the probabilistic contingency table (PCT). Using the TCT for verification requires a threshold, typically at 50%, to dichotomize probabilistic forecasts. The PCT maintains the valuable information in probabilities and verifies the true forecasts being reported. Simulated forecasts and outcomes as well as 2015–18 45 WS data are utilized to compare forecast performance metrics produced from the TCT and PCT to determine which verification tool better reflects the quality of forecasts. Comparisons of frequency bias and other statistical metrics computed from both dichotomized and continuous forecasts reveal misrepresentative performance metrics from the TCT as well as a loss of information necessary for verification. PCT bias better reflects forecast verification in contrast to that of TCT bias, which suggests suboptimal forecasts when in fact the forecasts are accurate.

Download Full-text

Calibrated Probabilistic Hub-Height Wind Forecasts in Complex Terrain

Weather and Forecasting ◽

10.1175/waf-d-16-0137.1 ◽

2017 ◽

Vol 32 (2) ◽

pp. 555-577 ◽

Cited By ~ 4

Author(s):

David Siuta ◽

Gregory West ◽

Roland Stull ◽

Thomas Nipen

Keyword(s):

Wind Speed ◽

Complex Terrain ◽

Forecast Error ◽

Wind Farms ◽

Probabilistic Forecast ◽

Forecast Performance ◽

Uncertainty Model ◽

Ensemble Mean ◽

Exact Shape ◽

Uncertainty Models

Abstract This work evaluates the use of a WRF ensemble for short-term, probabilistic, hub-height wind speed forecasts in complex terrain. Testing for probabilistic-forecast improvements is conducted by increasing the number of planetary boundary layer schemes used in the ensemble. Additionally, several prescribed uncertainty models used to derive forecast probabilities based on knowledge of the error within a past training period are evaluated. A Gaussian uncertainty model provided calibrated wind speed forecasts at all wind farms tested. Attempts to scale the Gaussian distribution based on the ensemble mean or variance values did not result in further improvement of the probabilistic forecast performance. When using the Gaussian uncertainty model, a small-sized six-member ensemble showed equal skill to that of the full 48-member ensemble. A new uncertainty model called the pq distribution that better fits the ensemble wind forecast error distribution is introduced. Results indicate that the gross attributes (central tendency, spread, and symmetry) of the prescribed uncertainty model are more important than its exact shape.

Download Full-text

The Impact of Writing Area Forecast Discussions on Student Forecaster Performance

Weather and Forecasting ◽

10.1175/waf905.1 ◽

2006 ◽

Vol 21 (1) ◽

pp. 104-108 ◽

Cited By ~ 3

Author(s):

Patrick S. Market

Keyword(s):

Student Performance ◽

Weather Forecast ◽

Chi Square ◽

University Of Missouri ◽

Forecast Performance ◽

Significant Difference ◽

Chi Square Test ◽

The University ◽

The Impact ◽

Local Weather

Abstract A brief study is provided on the forecast performance of students who write a mock area forecast discussion (AFD) on a weekly basis. Student performance was tracked for one semester (11 weeks) during the University of Missouri—Columbia's local weather forecast game. The hypothesis posed is that student performance is no better on days when they compose an AFD. A nonparametric Mann–Whitney test cannot reject that hypothesis. However, the same test employed on precipitation forecasts (for days when precipitation actually fell) shows that there is a statistically significant difference (p = 0.02) between the scores of those students writing an AFD and those who do not. Similar results are found with a chi-square test. Thus, AFD writers improve their precipitation score on days when significant weather occurred. Forecaster confidence is also enhanced by AFD composition.

Download Full-text

Comparison of Hybrid Ensemble/4DVar and 4DVar within the NAVDAS-AR Data Assimilation Framework

Monthly Weather Review ◽

10.1175/mwr-d-12-00182.1 ◽

2013 ◽

Vol 141 (8) ◽

pp. 2740-2758 ◽

Cited By ~ 86

Author(s):

David D. Kuhl ◽

Thomas E. Rosmond ◽

Craig H. Bishop ◽

Justin McLay ◽

Nancy L. Baker

Keyword(s):

Data Assimilation ◽

Forecast Error ◽

Statistical Significance ◽

Three Dimensional ◽

Weather Forecast ◽

Variational Data Assimilation ◽

Naval Research ◽

Background Error ◽

Forecast Performance ◽

Wide Range

Abstract The effect on weather forecast performance of incorporating ensemble covariances into the initial covariance model of the four-dimensional variational data assimilation (4D-Var) Naval Research Laboratory Atmospheric Variational Data Assimilation System-Accelerated Representer (NAVDAS-AR) is investigated. This NAVDAS-AR-hybrid scheme linearly combines the static NAVDAS-AR initial background error covariance with a covariance derived from an 80-member flow-dependent ensemble. The ensemble members are generated using the ensemble transform technique with a (three-dimensional variational data assimilation) 3D-Var-based estimate of analysis error variance. The ensemble covariances are localized using an efficient algorithm enabled via a separable formulation of the localization matrix. The authors describe the development and testing of this scheme, which allows for assimilation experiments using differing linear combinations of the static and flow-dependent background error covariances. The tests are performed for two months of summer and two months of winter using operational model resolution and the operational observational dataset, which is dominated by satellite observations. Results show that the hybrid mode data assimilation scheme significantly reduces the forecast error across a wide range of variables and regions. The improvements were particularly pronounced for tropical winds. The verification against radiosondes showed a greater than 0.5% reduction in vector wind RMS differences in areas of statistical significance. The verification against self-analysis showed a greater than 1% reduction from verifying against analyses between 2- and 5-day lead time at all eight vertical levels examined in areas of statistical significance. Using the Navy's summary of verification results, the Navy Operational Global Atmospheric Prediction System (NOGAPS) scorecard, the improvements resulted in a score (+1) that justifies a major system upgrade.

Download Full-text