Sensitivity of ensemble forecast verification to model bias

2020 ◽  
Author(s):  
Jingzhuo Wang ◽  
Jing Chen ◽  
Jun Du
Keyword(s):  
Bias Correction ◽  
Ensemble Prediction ◽  
Prediction System ◽  
Model Bias ◽  
Ensemble Spread ◽  
Ensemble Prediction System ◽  
Forecast Bias ◽  
Probabilistic Forecasts ◽  
Before And After

<p>        This study demonstrates how model bias can adversely affect the quality assessment of an ensemble prediction system (EPS) by verification metrics. A regional EPS [Global and Regional Assimilation and Prediction Enhanced System-Regional Ensemble Prediction System (GRAPES-REPS)] was verified over a period of one month over China. Three variables (500-hPa and 2-m temperatures, and 250-hPa wind) are selected to represent "strong" and "weak" bias situations. Ensemble spread and probabilistic forecasts are compared before and after a bias correction. The results show that the conclusions drawn from ensemble verification about the EPS are dramatically different with or without model bias. This is true for both ensemble spread and probabilistic forecasts. The GRAPES-REPS is severely underdispersive before the bias correction but becomes calibrated afterward, although the improvement in the spread' spatial structure is much less; the spread-skill relation is also improved. The probabilities become much sharper and almost perfectly reliable after the bias is removed. Therefore, it is necessary to remove forecast biases before an EPS can be accurately evaluated since an EPS deals only with random error but not systematic error. Only when an EPS has no or little forecast bias, can ensemble verification metrics reliably reveal the true quality of an EPS without removing forecast bias first. An implication is that EPS developers should not be expected to introduce methods to dramatically increase ensemble spread (either by perturbation method or statistical calibration) to achieve reliability. Instead, the preferred solution is to reduce model bias through prediction system developments and to focus on the quality of spread (not the quantity of spread). Forecast products should also be produced from the debiased but not the raw ensemble.</p>

scholarly journals Sensitivity of Ensemble Forecast Verification to Model Bias

2018 ◽  
Vol 146 (3) ◽  
pp. 781-796 ◽  
Author(s):  
Jingzhuo Wang ◽  
Jing Chen ◽  
Jun Du ◽  
Yutao Zhang ◽  
Yu Xia ◽  
...  
Keyword(s):  
Bias Correction ◽  
Ensemble Prediction ◽  
Prediction System ◽  
Model Bias ◽  
Ensemble Spread ◽  
Ensemble Prediction System ◽  
Forecast Bias ◽  
Probabilistic Forecasts ◽  
Before And After

This study demonstrates how model bias can adversely affect the quality assessment of an ensemble prediction system (EPS) by verification metrics. A regional EPS [Global and Regional Assimilation and Prediction Enhanced System-Regional Ensemble Prediction System (GRAPES-REPS)] was verified over a period of one month over China. Three variables (500-hPa and 2-m temperatures, and 250-hPa wind) are selected to represent “strong” and “weak” bias situations. Ensemble spread and probabilistic forecasts are compared before and after a bias correction. The results show that the conclusions drawn from ensemble verification about the EPS are dramatically different with or without model bias. This is true for both ensemble spread and probabilistic forecasts. The GRAPES-REPS is severely underdispersive before the bias correction but becomes calibrated afterward, although the improvement in the spread’s spatial structure is much less; the spread–skill relation is also improved. The probabilities become much sharper and almost perfectly reliable after the bias is removed. Therefore, it is necessary to remove forecast biases before an EPS can be accurately evaluated since an EPS deals only with random error but not systematic error. Only when an EPS has no or little forecast bias, can ensemble verification metrics reliably reveal the true quality of an EPS without removing forecast bias first. An implication is that EPS developers should not be expected to introduce methods to dramatically increase ensemble spread (either by perturbation method or statistical calibration) to achieve reliability. Instead, the preferred solution is to reduce model bias through prediction system developments and to focus on the quality of spread (not the quantity of spread). Forecast products should also be produced from the debiased but not the raw ensemble.

A Unified Scheme of Stochastic Physics and Bias Correction in an Ensemble Model to Reduce Both Random and Systematic Errors

2019 ◽  
Vol 34 (6) ◽  
pp. 1675-1691 ◽  
Author(s):  
Yu Xia ◽  
Jing Chen ◽  
Jun Du ◽  
Xiefei Zhi ◽  
Jingzhuo Wang ◽  
...  
Keyword(s):  
Bias Correction ◽  
Weather Prediction ◽  
Systematic Errors ◽  
Ensemble Prediction ◽  
Ensemble Model ◽  
Prediction System ◽  
Ensemble Prediction System ◽  
Probabilistic Forecasts ◽  
Ensemble Mean ◽  
Stochastic Physics

Abstract This study experimented with a unified scheme of stochastic physics and bias correction within a regional ensemble model [Global and Regional Assimilation and Prediction System–Regional Ensemble Prediction System (GRAPES-REPS)]. It is intended to improve ensemble prediction skill by reducing both random and systematic errors at the same time. Three experiments were performed on top of GRAPES-REPS. The first experiment adds only the stochastic physics. The second experiment adds only the bias correction scheme. The third experiment adds both the stochastic physics and bias correction. The experimental period is one month from 1 to 31 July 2015 over the China domain. Using 850-hPa temperature as an example, the study reveals the following: 1) the stochastic physics can effectively increase the ensemble spread, while the bias correction cannot. Therefore, ensemble averaging of the stochastic physics runs can reduce more random error than the bias correction runs. 2) Bias correction can significantly reduce systematic error, while the stochastic physics cannot. As a result, the bias correction greatly improved the quality of ensemble mean forecasts but the stochastic physics did not. 3) The unified scheme can greatly reduce both random and systematic errors at the same time and performed the best of the three experiments. These results were further confirmed by verification of the ensemble mean, spread, and probabilistic forecasts of many other atmospheric fields for both upper air and the surface, including precipitation. Based on this study, we recommend that operational numerical weather prediction centers adopt this unified scheme approach in ensemble models to achieve the best forecasts.

scholarly journals Added-value of ensemble prediction system on the quality of solar irradiance probabilistic forecasts

2020 ◽  
Vol 162 ◽  
pp. 1321-1339
Author(s):  
Josselin Le Gal La Salle ◽  
Jordi Badosa ◽  
Mathieu David ◽  
Pierre Pinson ◽  
Philippe Lauret
Keyword(s):  
Solar Irradiance ◽  
Added Value ◽  
Ensemble Prediction ◽  
Prediction System ◽  
Ensemble Prediction System ◽  
Probabilistic Forecasts

The WRF-Solar Ensemble Prediction System To Provide Solar Irradiance Probabilistic Forecasts

2021 ◽  
Author(s):  
Ju-Hye Kim ◽  
Pedro A. Jimenez ◽  
Manajit Sengupta ◽  
Jaemo Yang ◽  
Jimy Dudhia ◽  
...  
Keyword(s):  
Solar Irradiance ◽  
Ensemble Prediction ◽  
Prediction System ◽  
Ensemble Prediction System ◽  
Probabilistic Forecasts

scholarly journals Bias Correction of Tropical Cyclone Parameters in the ECMWF Ensemble Prediction System in Australia

2019 ◽  
Vol 147 (11) ◽  
pp. 4261-4285
Author(s):  
Saima Aijaz ◽  
Jeffrey D. Kepert ◽  
Hua Ye ◽  
Zhendong Huang ◽  
Alister Hawksford
Keyword(s):  
Tropical Cyclone ◽  
Bias Correction ◽  
Oil And Gas ◽  
Surface Wind ◽  
Oil And Gas Industry ◽  
Ensemble Prediction ◽  
Linear Regression Method ◽  
Prediction System ◽  
Ensemble Prediction System ◽  
Correction Technique

Abstract Global ensemble prediction systems have considerable ability to predict tropical cyclone (TC) formation and subsequent evolution. However, because of their relatively coarse resolution, their predictions of intensity and structure are biased. The biases arise mainly from underestimated intensities and enlarged radii, in particular the radius of maximum winds. This paper describes a method to reduce this limitation by bias correcting TCs in the ECMWF Ensemble Prediction System (ECMWF-EPS) for a region northwest of Australia. A bias-corrected TC system will provide more accurate forecasts of TC-generated wind and waves to the oil and gas industry, which operates a large number of offshore facilities in the region. It will also enable improvements in response decisions for weather sensitive operations that affect downtime and safety risks. The bias-correction technique uses a multivariate linear regression method to bias correct storm intensity and structure. Special strategies are used to maintain ensemble spread after bias correction and to predict the radius of maximum winds using a climatological relationship based on wind intensity and storm latitude. The system was trained on the Australian best track TC data and the ECMWF-EPS TC data from two cyclone seasons. The system inserts corrected vortices into the original surface wind and pressure fields, which are then used to estimate wind exceedance probabilities, and to drive a wave model. The bias-corrected system has shown an overall skill improvement over the uncorrected ECMWF-EPS for all TC intensity and structure parameters with the most significant gains for the maximum wind speed prediction. The system has been operational at the Australian Bureau of Meteorology since November 2016.

scholarly journals Regional Differences in the Prediction of Extratropical Cyclones by the ECMWF Ensemble Prediction System

2009 ◽  
Vol 137 (3) ◽  
pp. 893-911 ◽  
Author(s):  
Lizzie S. R. Froude
Keyword(s):  
Regional Differences ◽  
Propagation Speed ◽  
Ensemble Prediction ◽  
Extratropical Cyclones ◽  
Spatial Distributions ◽  
Prediction System ◽  
Ensemble Spread ◽  
Ensemble Prediction System ◽  
Mean Error ◽  
Ensemble Mean

Abstract A regional study of the prediction of extratropical cyclones by the European Centre for Medium-Range Weather Forecasts (ECMWF) Ensemble Prediction System (EPS) has been performed. An objective feature-tracking method has been used to identify and track the cyclones along the forecast trajectories. Forecast error statistics have then been produced for the position, intensity, and propagation speed of the storms. In previous work, data limitations meant it was only possible to present the diagnostics for the entire Northern Hemisphere (NH) or Southern Hemisphere. A larger data sample has allowed the diagnostics to be computed separately for smaller regions around the globe and has made it possible to explore the regional differences in the prediction of storms by the EPS. Results show that in the NH there is a larger ensemble mean error in the position of storms over the Atlantic Ocean. Further analysis revealed that this is mainly due to errors in the prediction of storm propagation speed rather than in direction. Forecast storms propagate too slowly in all regions, but the bias is about 2 times as large in the NH Atlantic region. The results show that storm intensity is generally overpredicted over the ocean and underpredicted over the land and that the absolute error in intensity is larger over the ocean than over the land. In the NH, large errors occur in the prediction of the intensity of storms that originate as tropical cyclones but then move into the extratropics. The ensemble is underdispersive for the intensity of cyclones (i.e., the spread is smaller than the mean error) in all regions. The spatial patterns of the ensemble mean error and ensemble spread are very different for the intensity of cyclones. Spatial distributions of the ensemble mean error suggest that large errors occur during the growth phase of storm development, but this is not indicated by the spatial distributions of the ensemble spread. In the NH there are further differences. First, the large errors in the prediction of the intensity of cyclones that originate in the tropics are not indicated by the spread. Second, the ensemble mean error is larger over the Pacific Ocean than over the Atlantic, whereas the opposite is true for the spread. The use of a storm-tracking approach, to both weather forecasters and developers of forecast systems, is also discussed.

scholarly journals Analysis of the Relative Operating Characteristic and Economic Value Using the LAPS Ensemble Prediction System in Taiwan

2015 ◽  
Vol 143 (5) ◽  
pp. 1833-1848 ◽  
Author(s):  
Hui-Ling Chang ◽  
Shu-Chih Yang ◽  
Huiling Yuan ◽  
Pay-Liam Lin ◽  
Yu-Chieng Liou
Keyword(s):  
Decision Making ◽  
Operating Characteristic ◽  
Economic Value ◽  
Ensemble Prediction ◽  
Prediction System ◽  
Forecast System ◽  
Relative Operating Characteristic ◽  
Ensemble Prediction System ◽  
Discrimination Ability ◽  
Probabilistic Forecasts

Abstract Measurement of the usefulness of numerical weather prediction considers not only the forecast quality but also the possible economic value (EV) in the daily decision-making process of users. Discrimination ability of an ensemble prediction system (EPS) can be assessed by the relative operating characteristic (ROC), which is closely related to the EV provided by the same forecast system. Focusing on short-range probabilistic quantitative precipitation forecasts (PQPFs) for typhoons, this study demonstrates the consistent and strongly related characteristics of ROC and EV based on the Local Analysis and Prediction System (LAPS) EPS operated at the Central Weather Bureau in Taiwan. Sensitivity experiments including the effect of terrain, calibration, and forecast uncertainties on ROC and EV show that the potential EV provided by a forecast system is mainly determined by the discrimination ability of the same system. The ROC and maximum EV (EVmax) of an EPS are insensitive to calibration, but the optimal probability threshold to achieve the EVmax becomes more reliable after calibration. In addition, the LAPS ensemble probabilistic forecasts outperform deterministic forecasts in respect to both ROC and EV, and such an advantage grows with increasing precipitation intensity. Also, even without explicitly knowing the cost–loss ratio, one can still optimize decision-making and obtain the EVmax by using ensemble probabilistic forecasts.

scholarly journals Outcomes and challenges of Forecast Demonstration Project (FDP) on landfalling cyclones over the Bay of Bengal

MAUSAM ◽  
2022 ◽  
Vol 64 (1) ◽  
pp. 1-12
Author(s):  
M. MOHAPATRA ◽  
B.K. BANDYOPADHYAY ◽  
D.R. SIKKA ◽  
AJIT TYAGI
Keyword(s):  
Bay Of Bengal ◽  
Weather Prediction ◽  
North Indian Ocean ◽  
Demonstration Project ◽  
Ensemble Prediction ◽  
Prediction System ◽  
Pilot Phase ◽  
Ensemble Prediction System ◽  
The North ◽  
Before And After

Cakxky dh [kkM+h esa m".kdfVca/kh; rwQkuksa ds ekxZ vkSj mudh rhozrk ds iwokZuqeku rduhd esa lq/kkj ykus ds fy, iwokZuqeku fun’kZu ifj;kstuk ¼,Q-Mh-ih-½ uked ,d dk;ZØe rS;kj fd;k x;k gSA ,Q-Mh-ih- dk;ZØe dk mÌs’;] ftu {ks=ksa ls vk¡dM+s vO;ofLFkr :i  ls izkIr gksrs gSa ogk¡ muds loaf/kZr izs{k.kksa ds lkFk gh lkFk mRrjh fgUn egklkxj esa pØokrksa ds mRiUu gksus] muds rhoz gksus vkSj mudh xfr dk vkdyu djus ds fy, fofHkUu l[;kRed ekSle iwokZuqeku ¼,u- MCY;w- ih-½ fun’kksZ dh {kerk dk izn’kZu djuk rFkk fo’ks"k :i  ls caxky dh [kkM+h ls lacaf/kr  ogha mlh LFkku ij fy, x, ekiksa ds vk/kkj ij fun’kksZ esa lq/kkj djuk gSA ,Q-Mh-ih- dk;ZØe rhu pj.kksa esa fu/kkfjr fd;k x;k gS uker% ¼i½ izh&ikbyV pj.k ¼15 vDrwcj ls 30 uoacj 2008] 2009½] ¼ii½ ikbyV pj.k ¼15 vDrwcj ls 30 uoacj 2010&2012½ rFkk ¼iii½ vafre pj.k ¼15 vDrwcj ls 30 uoacj 2013&2014½A Hkkjr] fdjk, ds gokbZ tgkt vkSj MªkWilkSansa iz;ksxksa ls 15 vDrwcj ls 30 uoacj 2013&2014 ds nkSjku caxky dh [kkM+h esa cuus okys pØokrksa dk gokbZ tgkt ds tfj, irk yxkus dh ;kstuk cuk jgk gSA bl mÌs’; ds iwfrZ ds fy, ¼i½ izs{k.kkRed mUu;u ¼ii½ pØokr fo’ys"k.k vkSj iwokZuqeku iz.kkyh dk vk/kqfudhdj.k ¼iii½ pØokr fo’ys"k.k vkSj iwokZuqeku izfØ;k ¼iv½ psrkouh mRiknksa dks rS;kj djuk] mudk izLrqrhdj.k rFkk izlj.k ¼v½ fo’oluh;rk mik; vkSj {kerk fuekZ.k ij izkFkfedrk ds vk/kkj ij dk;Z fd, x,A pØokr ds izs{k.k] fo’ys"k.k vkSj iwokZuqeku esa lq/kkj ykus ds fy, fofHkUu dk;Z iz.kkfy;k¡ viukbZ xbZaA o"kZ 2008&11 ds nkSjku ,Q-Mh-ih- vfHk;ku ds izh&ikbyV vkSj ikbyV pj.kksa esa la;qDr izs{k.kkRed] lapkjkRed vkSj ,u-MCY;w-ih- xfrfof/k;ksa esa vusd jk"Vªh; laLFkkuksa us Hkkx fy;kA ,Q-Mh-ih- ds igys vkSj mlds ckn dh izs{k.kkRed iz.kkfy;ksa dh rqyuk ls {ks= esa jsMkj] Lopkfyr ekSle dsUnz ¼,- MCY;w-,l-½] mPp iou xfr fjdkWMjksa esa egRoiw.kZ lq/kkj dk irk pyk gSA bl lq/kkj ls ekWuhVju vkSj iwokZuqeku esa gksus okyh =qfV;ksa esa deh vkbZ gSA th- ,Q- ,l- MCY;w vkj- ,Q] ,p- MCY;w- vkj- ,Q- vkSj vlsEcy iwokZuqeku iz.kkyh ¼bZ- ih- ,l-½  ds vkjaHk gksus ls ,u- MCY;w- ih- funsZ’kksa ds dk;Z fu"iknu esa o`f) gqbZ gSA bl 'kks/k i= esa bl ifj;kstuk dh miyfC/k;ksa ds egRoiw.kZ y{k.kksa lfgr leL;kvksa vkSj laHkkoukvksa dks izLrqr fd;k x;k gS rFkk mudh foospuk dh xbZ gSA pØokrksa dk gokbZ tgkt }kjk irk yxkus ds fy, ckj&ckj fd, x, iz;klksa ds ckotwn ;g dk;Z vHkh laHko ugha gks ldk gSA o"kZ 2013&14 ds nkSjku Hkkoh vfHk;ku ds le; ;g ,d eq[; pqukSrh gksxhA A programme has been evolved for improvement in prediction of track and intensity of tropical cyclones over the Bay of Bengal resulting in the Forecast Demonstration Project (FDP). FDP programme is aimed to demonstrate the ability of various Numerical Weather Prediction (NWP) models to assess the genesis, intensification and movement of cyclones over the north Indian ocean with enhanced observations over the data sparse region and to incorporate modification into the models which could be specific to the Bay of Bengal based on the in-situ measurements. FDP Programme is scheduled in three phases, viz., (i) Pre-pilot phase (15 Oct - 30 Nov 2008, 2009, (ii) Pilot phase (15 Oct - 30 Nov, 2010-2012) and (iii) Final phase (15 Oct - 30 Nov, 2013-14). India is planning to take up aircraft probing of cyclones over the Bay of Bengal during 15 Oct - 30 Nov, 2013-14 with hired aircraft and dropsonde experiments. To accomplish the above objective, the initiative was carried out with priorities on (i) observational upgradation, (ii) modernisation of cyclone analysis and prediction system, (iii) cyclone analysis and forecasting procedure, (iv) warning products generation, presentation & dissemination, (v) confidence building measures and capacity building. Various strategies were adopted for improvement of observation, analysis and prediction of cyclone. Several national institutions participated for joint observational, communicational & NWP activities during the pre-pilot and pilot phases of FDP campaign during 2008-11. The comparison of observational systems before and after FDP indicates a significant improvement in terms of Radar, Automatic Weather Station (AWS), high wind speed recorders over the region. It has resulted in reduction in monitoring and forecasting errors. The performance of NWP models have increased along with the introduction of NWP platforms like IMD GFS, WRF, HWRF and ensemble prediction system (EPS). Salient features of achievements along with the problems and prospects of this project are presented and discussed in this paper. With repeated attempts, the aircraft probing of cyclones could not be possible till now. It is a major challenge for the future campaign during 2013-14.

scholarly journals Why Should Ensemble Spread Match the RMSE of the Ensemble Mean?

2014 ◽  
Vol 15 (4) ◽  
pp. 1708-1713 ◽  
Author(s):  
V. Fortin ◽  
M. Abaza ◽  
F. Anctil ◽  
R. Turcotte
Keyword(s):  
Good Practice ◽  
Ensemble Prediction ◽  
Square Root ◽  
Prediction System ◽  
Mean Square ◽  
Ensemble Spread ◽  
Ensemble Prediction System ◽  
Correct Equation ◽  
Ensemble Mean ◽  
The Impact

Abstract When evaluating the reliability of an ensemble prediction system, it is common to compare the root-mean-square error of the ensemble mean to the average ensemble spread. While this is indeed good practice, two different and inconsistent methodologies have been used over the last few years in the meteorology and hydrology literature to compute the average ensemble spread. In some cases, the square root of average ensemble variance is used, and in other cases, the average of ensemble standard deviation is computed instead. The second option is incorrect. To avoid the perpetuation of practices that are not supported by probability theory, the correct equation for computing the average ensemble spread is obtained and the impact of using the wrong equation is illustrated.

Sign in / Sign up

Export Citation Format

Share Document