scholarly journals Comparisons of QPFs Derived from Single- and Multicore Convection-Allowing Ensembles

2019 ◽  
Vol 34 (6) ◽  
pp. 1955-1964
Author(s):  
Adam J. Clark
Keyword(s):  
Operating Characteristic ◽  
Characteristic Curve ◽  
Skill Score ◽  
Ensemble Forecast ◽  
Grid Spacing ◽  
Forecast System ◽  
Relative Operating Characteristic ◽  
Core System ◽  
Testing And Evaluation ◽  
Better Than

Abstract This study compares ensemble precipitation forecasts from 10-member, 3-km grid-spacing, CONUS domain single- and multicore ensembles that were a part of the 2016 Community Leveraged Unified Ensemble (CLUE) that was run for the 2016 NOAA Hazardous Weather Testbed Spring Forecasting Experiment. The main results are that a 10-member ARW ensemble was significantly more skillful than a 10-member NMMB ensemble, and a 10-member MIX ensemble (5 ARW and 5 NMMB members) performed about the same as the 10-member ARW ensemble. Skill was measured by area under the relative operating characteristic curve (AUC) and fractions skill score (FSS). Rank histograms in the ARW ensemble were flatter than the NMMB ensemble indicating that the envelope of ensemble members better encompassed observations (i.e., better reliability) in the ARW. Rank histograms in the MIX ensemble were similar to the ARW ensemble. In the context of NOAA’s plans for a Unified Forecast System featuring a CAM ensemble with a single core, the results are positive and indicate that it should be possible to develop a single-core system that performs as well as or better than the current operational CAM ensemble, which is known as the High-Resolution Ensemble Forecast System (HREF). However, as new modeling applications are developed and incremental changes that move HREF toward a single-core system are made possible, more thorough testing and evaluation should be conducted.

scholarly journals Postprocessing Next-Day Ensemble Probabilistic Precipitation Forecasts Using Random Forests

2019 ◽  
Vol 34 (6) ◽  
pp. 2017-2044 ◽  
Author(s):  
Eric D. Loken ◽  
Adam J. Clark ◽  
Amy McGovern ◽  
Montgomery Flora ◽  
Kent Knopfmeier
Keyword(s):  
Random Forest ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Ensemble Forecast ◽  
Grid Spacing ◽  
Relative Operating Characteristic ◽  
Advantages And Disadvantages ◽  
Skill Scores ◽  
Systematic Biases ◽  
Fold Cross Validation

Abstract Most ensembles suffer from underdispersion and systematic biases. One way to correct for these shortcomings is via machine learning (ML), which is advantageous due to its ability to identify and correct nonlinear biases. This study uses a single random forest (RF) to calibrate next-day (i.e., 12–36-h lead time) probabilistic precipitation forecasts over the contiguous United States (CONUS) from the Short-Range Ensemble Forecast System (SREF) with 16-km grid spacing and the High-Resolution Ensemble Forecast version 2 (HREFv2) with 3-km grid spacing. Random forest forecast probabilities (RFFPs) from each ensemble are compared against raw ensemble probabilities over 496 days from April 2017 to November 2018 using 16-fold cross validation. RFFPs are also compared against spatially smoothed ensemble probabilities since the raw SREF and HREFv2 probabilities are overconfident and undersample the true forecast probability density function. Probabilistic precipitation forecasts are evaluated at four precipitation thresholds ranging from 0.1 to 3 in. In general, RFFPs are found to have better forecast reliability and resolution, fewer spatial biases, and significantly greater Brier skill scores and areas under the relative operating characteristic curve compared to corresponding raw and spatially smoothed ensemble probabilities. The RFFPs perform best at the lower thresholds, which have a greater observed climatological frequency. Additionally, the RF-based postprocessing technique benefits the SREF more than the HREFv2, likely because the raw SREF forecasts contain more systematic biases than those from the raw HREFv2. It is concluded that the RFFPs provide a convenient, skillful summary of calibrated ensemble output and are computationally feasible to implement in real time. Advantages and disadvantages of ML-based postprocessing techniques are discussed.

scholarly journals Post-Processing and Evaluation of Precipitation Ensemble Forecast under Multiple Schemes in Beijiang River Basin

Water ◽  
2020 ◽  
Vol 12 (9) ◽  
pp. 2631
Author(s):  
Xinchi Chen ◽  
Xiaohong Chen ◽  
Dong Huang ◽  
Huamei Liu
Keyword(s):  
Time Scales ◽  
Weather Prediction ◽  
Forecast Accuracy ◽  
Skill Score ◽  
Ensemble Forecast ◽  
Efficiency Coefficient ◽  
Forecast System ◽  
Factors Affecting ◽  
Hydrological Forecasting ◽  
Better Than

Precipitation is one of the most important factors affecting the accuracy and uncertainty of hydrological forecasting. Considerable progress has been made in numerical weather prediction after decades of development, but the forecast products still cannot be used directly for hydrological forecasting. This study used ensemble pro-processor (EPP) to post-process the Global Ensemble Forecast System (GEFS) and Climate Forecast System version 2 (CFSv2) with four designed schemes, and then integrated them to investigate the forecast accuracy in longer time scales based on the best scheme. Many indices such as correlation coefficient, Nash efficiency coefficient, rank histogram, and continuous ranked probability skill score were used to evaluate the results in different aspects. The results show that EPP can improve the accuracy of raw forecast significantly, and the scheme considering cumulative forecast precipitation is better than that only considers single-day forecast. Moreover, the scheme that considers some observed precipitation would help to improve the accuracy and reduce the uncertainty. In terms of medium- and long-term forecasts, the integrated forecast based on GEFS and CFSv2 after post-processed would be better than CFSv2 significantly. The results of this study would be a very important demonstration to remove the deviation of ensemble forecast and improve the accuracy of hydrological forecasting in different time scales.

Initial Conditions for Convection-Allowing Ensembles over the Conterminous United States

2020 ◽  
Vol 148 (7) ◽  
pp. 2645-2669
Author(s):  
Craig S. Schwartz ◽  
May Wong ◽  
Glen S. Romine ◽  
Ryan A. Sobash ◽  
Kathryn R. Fossell
Keyword(s):  
United States ◽  
Ensemble Kalman Filter ◽  
Initial Conditions ◽  
Limited Area ◽  
Grid Spacing ◽  
Forecast System ◽  
Skill Scores ◽  
Ensemble Mean ◽  
Horizontal Grid ◽  
Better Than

Abstract Five sets of 48-h, 10-member, convection-allowing ensemble (CAE) forecasts with 3-km horizontal grid spacing were systematically evaluated over the conterminous United States with a focus on precipitation across 31 cases. The various CAEs solely differed by their initial condition perturbations (ICPs) and central initial states. CAEs initially centered about deterministic Global Forecast System (GFS) analyses were unequivocally better than those initially centered about ensemble mean analyses produced by a limited-area single-physics, single-dynamics 15-km continuously cycling ensemble Kalman filter (EnKF), strongly suggesting relative superiority of the GFS analyses. Additionally, CAEs with flow-dependent ICPs derived from either the EnKF or multimodel 3-h forecasts from the Short-Range Ensemble Forecast (SREF) system had higher fractions skill scores than CAEs with randomly generated mesoscale ICPs. Conversely, due to insufficient spread, CAEs with EnKF ICPs had worse reliability, discrimination, and dispersion than those with random and SREF ICPs. However, members in the CAE with SREF ICPs undesirably clustered by dynamic core represented in the ICPs, and CAEs with random ICPs had poor spinup characteristics. Collectively, these results indicate that continuously cycled EnKF mean analyses were suboptimal for CAE initialization purposes and suggest that further work to improve limited-area continuously cycling EnKFs over large regional domains is warranted. Additionally, the deleterious aspects of using both multimodel and random ICPs suggest efforts toward improving spread in CAEs with single-physics, single-dynamics, flow-dependent ICPs should continue.

scholarly journals An Extended Procedure for Implementing the Relative Operating Characteristic Graphical Method

2006 ◽  
Vol 45 (9) ◽  
pp. 1215-1223 ◽  
Author(s):  
Fredrick H. M. Semazzi ◽  
Roberto J. Mera
Keyword(s):  
Operating Characteristic ◽  
Graphical Method ◽  
General Procedure ◽  
Economic Value ◽  
Skill Score ◽  
Prediction System ◽  
End User ◽  
Relative Operating Characteristic ◽  
Typical Situation ◽  
Pass Through

Abstract The functional relationship between the relative operating characteristic (ROC) and the economic value (EV) graphical methods have been exploited to develop a hybrid procedure called the extended ROC (EROC) method. The EROC retains the appealing simplicity of the traditional ROC method and the ability of the EV method to provide evaluation of the performance of an ensemble climate prediction system (EPS) for a hypothetical end user defined by the cost–loss ratio (μ = C/L). An inequality defining the lower and upper theoretical bounds of μ has been derived. Outside these limits, the EPS yields no added benefits for end user μ relative to the use of climatological persistence as an alternative prediction system. In the traditional ROC graphical method, the ROC skill (ROCS) is often expressed in terms of the area between the ROC graph and the diagonal baseline passing through the origin with slope m = 1. Thus, ROCS = 2A − 1, where A is the area under the ROC graph. In the proposed EROC approach, a more general procedure is recommended based on the construction of user-specific baselines that do not necessarily pass through the origin and, in general, have slope m ≠ 1. The skill of a particular EPS computed from the EROC method is proportional to the corresponding estimated value based on the EV graphical method. Therefore, the EROC geometry conveys the same basic information as the EV method. The Semazzi–Mera skill score (SMSS) is proposed as a convenient and compact way of expressing the combined verification based on the ROC and EV methods. The ROCS estimate is a special case of the SMSS. The near-horizontal trail-like geometry sometimes exhibited by EV graphs is also examined. It is shown to occur when either the hit-rate or false-alarm term dominates in the formula for EV, unlike the more typical situation in which both terms are comparable in magnitude.

scholarly journals Are Body Composition Parameters Better than Conventional Anthropometric Measures in Predicting Pediatric Hypertension?

2020 ◽  
Vol 17 (16) ◽  
pp. 5771
Author(s):  
Chih-Yu Hsu ◽  
Rong-Ho Lin ◽  
Yu-Ching Lin ◽  
Jau-Yuan Chen ◽  
Wen-Cheng Li ◽  
...  
Keyword(s):  
Body Composition ◽  
Receiver Operating Characteristic Curve ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Fat Free Mass ◽  
Z Score ◽  
Pediatric Hypertension ◽  
Anthropometric Measures ◽  
Operating Characteristic Curve ◽  
Better Than

Body composition (BC) parameters are associated with cardiometabolic diseases in children; however, the importance of BC parameters for predicting pediatric hypertension is inconclusive. This cross-sectional study aimed to compare the difference in predictive values of BC parameters and conventional anthropometric measures for pediatric hypertension in school-aged children. A total of 340 children (177 girls and 163 boys) with a mean age of 8.8 ± 1.7 years and mean body mass index (BMI) z-score of 0.50 ± 1.24 were enrolled (102 hypertensive children and 238 normotensive children). Significantly higher values of anthropometric measures (BMI, BMI z-score, BMI percentile, waist-to-height ratio) and BC parameters (body-fat percentage, muscle weight, fat mass, fat-free mass) were observed among the hypertensive subgroup compared to their normotensive counterparts. A prediction model combining fat mass ≥ 3.65 kg and fat-free mass ≥ 34.65 kg (area under the receiver operating characteristic curve = 0.688; sensitivity = 66.7%; specificity = 89.9%) performed better than BMI alone (area under the receiver operating characteristic curve = 0.649; sensitivity = 55.9%; specificity = 73.9%) in predicting hypertension. In conclusion, BC parameters are better than anthropometric measures in predicting pediatric hypertension. BC measuring is a reasonable approach for risk stratification in pediatric hypertension.

scholarly journals A Note On the Maximum Peirce Skill Score

2007 ◽  
Vol 22 (5) ◽  
pp. 1148-1154 ◽  
Author(s):  
Agostino Manzato
Keyword(s):  
Cost Model ◽  
Characteristic Curve ◽  
Economic Cost ◽  
Skill Score ◽  
Relative Operating Characteristic ◽  
Skill Scores ◽  
Scalar Measure ◽  
Binary Classifiers ◽  
Continuous Predictor ◽  
Simple Economic

Abstract Binary classifiers are obtained from a continuous predictor using a threshold to dichotomize the predictor value into event occurrence and nonoccurrence classes. A contingency table is associated with each threshold, and from this table many statistical indices (like skill scores) can be computed. This work shows that the threshold that maximizes one of these indices [the Peirce skill score (PSS)] has some important properties. In particular, at that threshold the ratio of the two likelihood distributions is always 1 and the event posterior probability is equal to the event prior probability. These properties, together with the consideration that the maximum PSS is the point with the “most skill” on the relative operating characteristic curve and the point that maximizes the forecast value, suggest the use of the maximum PSS as a good scalar measure of the classifier skill. To show that this most skilled point is not always the best one for all the users, a simple economic cost model is presented.

scholarly journals IoT information theft prediction using ensemble feature selection

2022 ◽  
Vol 9 (1) ◽  
Author(s):  
Joffrey L. Leevy ◽  
John Hancock ◽  
Taghi M. Khoshgoftaar ◽  
Jared M. Peterson
Keyword(s):  
Feature Selection ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Classification Performance ◽  
Feature Reduction ◽  
Security Risk ◽  
Precision Recall Curve ◽  
Iot Devices ◽  
Feature Selection Techniques ◽  
Better Than

AbstractThe recent years have seen a proliferation of Internet of Things (IoT) devices and an associated security risk from an increasing volume of malicious traffic worldwide. For this reason, datasets such as Bot-IoT were created to train machine learning classifiers to identify attack traffic in IoT networks. In this study, we build predictive models with Bot-IoT to detect attacks represented by dataset instances from the Information Theft category, as well as dataset instances from the data exfiltration and keylogging subcategories. Our contribution is centered on the evaluation of ensemble feature selection techniques (FSTs) on classification performance for these specific attack instances. A group or ensemble of FSTs will often perform better than the best individual technique. The classifiers that we use are a diverse set of four ensemble learners (Light GBM, CatBoost, XGBoost, and random forest (RF)) and four non-ensemble learners (logistic regression (LR), decision tree (DT), Naive Bayes (NB), and a multi-layer perceptron (MLP)). The metrics used for evaluating classification performance are area under the receiver operating characteristic curve (AUC) and Area Under the precision-recall curve (AUPRC). For the most part, we determined that our ensemble FSTs do not affect classification performance but are beneficial because feature reduction eases computational burden and provides insight through improved data visualization.

Tornado-Resolving Ensemble and Probabilistic Predictions of the 20 May 2013 Newcastle–Moore EF5 Tornado

2019 ◽  
Vol 147 (4) ◽  
pp. 1215-1235 ◽  
Author(s):  
Nathan Snook ◽  
Ming Xue ◽  
Youngsun Jung
Keyword(s):  
Characteristic Curve ◽  
Surface Wind ◽  
Horizontal Wind ◽  
Grid Spacing ◽  
Ensemble Forecasts ◽  
Relative Operating Characteristic ◽  
Near Surface ◽  
Pressure Drops ◽  
Wind Speeds ◽  
Horizontal Grid

Abstract An ensemble of 10 forecasts is produced for the 20 May 2013 Newcastle–Moore EF5 tornado and its parent supercell using a horizontal grid spacing of 50 m, nested within ensemble forecasts with 500-m horizontal grid spacing initialized via ensemble Kalman filter data assimilation of surface and radar observations. Tornadic circulations are predicted in all members, though the intensity, track, and longevity of the predicted tornado vary substantially among members. Overall, tornadoes in the ensemble forecasts persisted longer and moved to the northeast faster than the observed tornado. In total, 8 of the 10 ensemble members produce tornadoes with winds corresponding to EF2 intensity or greater, with maximum instantaneous near-surface horizontal wind speeds of up to 130 m s−1 and pressure drops of up to 120 hPa; values similar to those reported in observational studies of intense tornadoes. The predicted intense tornadoes all acquire well-defined two-cell vortex structure, and exhibit features common in observed tornadic storms, including a weak-echo notch and low reflectivity within the mesocyclone. Ensemble-based probabilistic tornado forecasts based upon near-surface wind and/or vorticity fields at 10 m above the surface produce skillful forecasts of the tornado in terms of area under the relative operating characteristic curve, with probability swaths extending along and to the northeast of the observed tornado path. When probabilistic swaths of 0–3- and 2–5-km updraft helicity are compared to the swath of wind at 10 m above the surface exceeding 29 m s−1, a slight northwestward bias is present, although the pathlength, orientation, and the placement of minima and maxima show very strong agreement.

DETECTION OF BLOOD VESSELS IN RETINAL IMAGES

2010 ◽  
Vol 10 (01) ◽  
pp. 57-72 ◽  
Author(s):  
HEJER JLASSI ◽  
KAMEL HAMROUNI
Keyword(s):  
Blood Vessels ◽  
Anisotropic Diffusion ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Retinal Images ◽  
Linear Filter ◽  
Vascular Tree ◽  
Morphological Reconstruction ◽  
Processing Steps ◽  
Better Than

This paper presents a method to segment blood vessels in retinal images. It is based on mathematical morphology and the anisotropic diffusion and is composed of four steps: image processing by using linear filter and morphological ones, details extraction by using top-hat transform, morphological reconstruction of vascular tree and post processing steps using anisotropic diffusion. Our method is tested on red-free retinal images, taken from two public database. Our results on both public databases were comparable in performance with other authors. The method achieves a good result by mean of the "receiver operating characteristic curve" (ROC). The results show that our method is significantly better than other rule-based methods.

scholarly journals Analysis of the Relative Operating Characteristic and Economic Value Using the LAPS Ensemble Prediction System in Taiwan

2015 ◽  
Vol 143 (5) ◽  
pp. 1833-1848 ◽  
Author(s):  
Hui-Ling Chang ◽  
Shu-Chih Yang ◽  
Huiling Yuan ◽  
Pay-Liam Lin ◽  
Yu-Chieng Liou
Keyword(s):  
Decision Making ◽  
Operating Characteristic ◽  
Economic Value ◽  
Ensemble Prediction ◽  
Prediction System ◽  
Forecast System ◽  
Relative Operating Characteristic ◽  
Ensemble Prediction System ◽  
Discrimination Ability ◽  
Probabilistic Forecasts

Abstract Measurement of the usefulness of numerical weather prediction considers not only the forecast quality but also the possible economic value (EV) in the daily decision-making process of users. Discrimination ability of an ensemble prediction system (EPS) can be assessed by the relative operating characteristic (ROC), which is closely related to the EV provided by the same forecast system. Focusing on short-range probabilistic quantitative precipitation forecasts (PQPFs) for typhoons, this study demonstrates the consistent and strongly related characteristics of ROC and EV based on the Local Analysis and Prediction System (LAPS) EPS operated at the Central Weather Bureau in Taiwan. Sensitivity experiments including the effect of terrain, calibration, and forecast uncertainties on ROC and EV show that the potential EV provided by a forecast system is mainly determined by the discrimination ability of the same system. The ROC and maximum EV (EVmax) of an EPS are insensitive to calibration, but the optimal probability threshold to achieve the EVmax becomes more reliable after calibration. In addition, the LAPS ensemble probabilistic forecasts outperform deterministic forecasts in respect to both ROC and EV, and such an advantage grows with increasing precipitation intensity. Also, even without explicitly knowing the cost–loss ratio, one can still optimize decision-making and obtain the EVmax by using ensemble probabilistic forecasts.

Sign in / Sign up

Export Citation Format

Share Document