On the ROC Area of Ensemble Forecasts for Rare Events

The relative operating characteristic (ROC) curve is a popular diagnostic tool in forecast verification, with the area under the ROC curve (AUC) used as a verification metric measuring the discrimination ability of a forecast. Along with calibration, discrimination is deemed as a fundamental probabilistic forecast attribute. In particular, in ensemble forecast verification, AUC provides a basis for the comparison of potential predictive skill of competing forecasts. While this approach is straightforward when dealing with forecasts of common events (e.g. probability of precipitation), the AUC interpretation can turn out to be oversimplistic or misleading when focusing on rare events (e.g. precipitation exceeding some warning criterion). How should we interpret AUC of ensemble forecasts when focusing on rare events? How can changes in the way probability forecasts are derived from the ensemble forecast affect AUC results? How can we detect a genuine improvement in terms of predictive skill? Based on verification experiments, a critical eye is cast on the AUC interpretation to answer these questions. As well as the traditional trapezoidal approximation and the well-known bi-normal fitting model, we discuss a new approach which embraces the concept of imprecise probabilities and relies on the subdivision of the lowest ensemble probability category.

Urban Expansion Simulation Based on Various Driving Factors Using a Logistic Regression Model: Delhi as a Case Study

During the last three decades, Delhi has witnessed extensive and rapid urban expansion in all directions, especially in the East South East zone. The total built-up area has risen dramatically, from 195.3 sq. km to 435.1 sq. km, during 1989–2020, which has led to habitat fragmentation, deforestation, and difficulties in running urban utility services effectively in the new extensions. This research aimed to simulate urban expansion in Delhi based on various driving factors using a logistic regression model. The recent urban expansion of Delhi was mapped using LANDSAT images of 1989, 2000, 2010, and 2020. The urban expansion was analyzed using concentric rings to show the urban expansion intensity in each direction. Nine driving factors were analyzed to detect the influence of each factor on the urban expansion process. The results revealed that the proximity to urban areas, proximity to main roads, and proximity to medical facilities were the most significant factors in Delhi during 1989–2020, where they had the highest regression coefficients: −0.884, −0.475, and −0.377, respectively. In addition, the predicted pattern of urban expansion was chaotic, scattered, and dense on the peripheries. This pattern of urban expansion might lead to further losses of natural resources. The relative operating characteristic method was utilized to assess the accuracy of the simulation, and the resulting value of 0.96 proved the validity of the simulation. The results of this research will aid local authorities in recognizing the patterns of future expansion, thus facilitating the implementation of effective policies to achieve sustainable urban development in Delhi.

RADAR-BASED STOCHASTIC PRECIPITATION NOWCASTING USING THE SHORT-TERM ENSEMBLE PREDICTION SYSTEM (STEPS) (CASE STUDY: PANGKALAN BUN WEATHER RADAR)

Nowcasting, or the short-term forecasting of precipitation, is urgently needed to support the mitigation circle in hydrometeorological disasters. Pangkalan Bun weather radar is single-polarization radar with a 200 km maximum range and which runs 10 elevation angles in 10 minutes with a 250 meters spatial resolution. There is no terrain blocking around the covered area. The Short-Term Ensemble Prediction System (STEPS) is one of many algorithms that is used to generate precipitation nowcasting, and is already in operational use. STEPS has the advantage of producing ensemble nowcasts, by which nowcast uncertainties can be statistically quantified. This research aims to apply STEPS to generate stochastic nowcasting in Pangkalan Bun weather radar and to analyze its advantages and weaknesses. Accuracy is measured by counting the possibility of detection and false alarms under the 5 dBZ threshold and plotting them in a relative operating characteristic (ROC) curve. The observed frequency and forecast probability is represented by a reliability diagram to evaluate nowcast reliability and sharpness. Qualitative analysis of the results showed that the STEPS ensemble produces smoothed reflectivity fields that cannot capture extreme values in an observed quasi-linear convective system (QLCS), but that the algorithm achieves good accuracy under the threshold used, up to 40 minutes lead time. The ROC shows a curved upper left-hand corner, and the reliability diagram is an almost perfect nowcast diagonal line.

Improving the Spring Air Temperature Forecast Skills of BCC_CSM1.1 (m) by Spatial Disaggregation and Bias Correction: Importance of Trend Correction

In this study, an improved method named spatial disaggregation and detrended bias correction (SDDBC) based on spatial disaggregation and bias correction (SDBC) combined with trend correction was proposed. Using data from meteorological stations over China from 1991 to 2020 and the seasonal hindcast data from the Beijing Climate Center Climate System Model (BCC_CSM1.1 (m)), the performances of the model, SDBC, and SDDBC in spring temperature forecasts were evaluated. The results showed that the observed spring temperature exhibits a significant increasing trend in most of China, but the warming trend simulated by the model was obviously smaller. SDBC performed poorly in temperature trend correction. With SDDBC, the model’s deviation in temperature trend was corrected, and consequently, the temporal correlation between the model’s simulation and the observation as well as the forecasting skill on the phase of temperature were improved, thus improving the MSSS and the ACC. From the perspective of probabilistic prediction, the relative operating characteristic skill score (ROCSS) and the Brier skill score (BSS) of the SDDBC for three categorical forecasts were higher than those of the model and SDBC. The SDDBC’s BSS increased as the effect of the increasing resolution component was greater than that of the decreasing reliability component. Therefore, it is necessary to correct the predicted temperature trend in post-processing for the output of numerical prediction models.

Diagnostic value of pulse oximetry combined with cardiac auscultation in screening congenital heart disease in neonates

Objective This study aimed to investigate the feasibility and reliability of pulse oximetry combined with cardiac auscultation in screening neonatal congenital heart disease (CHD). Methods This was a retrospective, observational, screening study. All newborns included in the study were at the Second Affiliated Hospital of Wenzhou Medical University from July 2019 to January 2020. Primary screening of CHD was conducted by pulse oximetry combined with cardiac auscultation assays. Indices, including sensitivity, specificity, the positive/negative predictive value, the positive/negative likelihood ratio, and the diagnostic odds ratio, were calculated. The area under the relative operating characteristic curve of the subjects was measured. Results A total of 3327 neonates were enrolled, among whom 139 were diagnosed with CHD and the incidence of CHD was 4.2%. The sensitivity, specificity, diagnostic odds ratio, and area under the relative operating characteristic curve of pulse oximetry combined with cardiac auscultation were 89.9%, 94.7%, 169.0, and 0.923, respectively. Conclusions Pulse oximetry combined with cardiac auscultation is a novel screening method with acceptable accuracy and feasibility for neonatal CHD. This combination method is worth promoting widely.

Seismic Vulnerability Assessment in Ranau, Sabah, Using Two Different Models

Sabah is prone to seismic activities due to its location, being geographically located near the boundaries of three major active tectonic plates; the Eurasian, India-Australia, and Philippine-Pacific plates. The 6.0 Mw earthquake that occurred in Ranau, Sabah, on 15 June 2015 which caused 18 casualties, all of them climbers of Mount Kinabalu, raised many issues, primarily the requirements for seismic vulnerability assessment for this region. This study employed frequency ratio (FR)–index of entropy (IoE) and a combination of (FR-IoE) with an analytical hierarchy process (AHP) to map seismic vulnerability for Ranau, Sabah. The results showed that the success rate and prediction rate for the areas under the relative operating characteristic (ROC) curves were 0.853; 0.856 for the FR-IoE model and 0.863; 0.906 for (FR-IoE) AHP, respectively, with the highest performance achieved using the (FR-IoE) AHP model. The vulnerability maps produced were classified into five classes; very low, low, moderate, high, and very high seismic vulnerability. Seismic activities density ratio analysis performed on the final seismic vulnerability maps showed that high seismic activity density ratios were observed for high vulnerability zones with the values of 9.119 and 8.687 for FR-IoE and (FR-IoE) AHP models, respectively.

A theoretical relationship between probabilistic relative operating characteristic skill and deterministic correlation skill in dynamical seasonal climate prediction

Understanding the relationship between probabilistic and deterministic prediction skills is of important significance for the study of seasonal forecasting and verification. Based on the Brier skill score methodology, we have previously found a theoretical relationship between the probabilistic resolution skill and the deterministic correlation (i.e., anomaly correlation; AC) skill and a lack of necessary or consistent relationship between the probabilistic reliability skill and the deterministic skill in dynamical seasonal prediction. Here, we further theoretically investigate the relationship between the probabilistic relative operating characteristic (ROC) skill and the deterministic skill. The ROC measures the discrimination attribute of probabilistic forecast quality, another important attribute besides the resolution and reliability. With some simplified assumptions, we first derive theoretical expressions for the hit and false-alarm rates that are basic ingredients for the ROC curve, then demonstrate a sole dependence of the ROC curve on the AC, and finally analytically derive a relationship between the related ROC score and the AC. Such a theoretically derived ROC-AC relationship is further examined using dynamical models’ ensemble seasonal hindcasts, which is well verified. The finding here along with our previous findings implies that the discrimination and resolution attributes of probabilistic seasonal forecast skill are intrinsically equivalent to the corresponding deterministic skill, while the reliability appears to be the fundamental attribute of the probabilistic skill that differs from the deterministic skill, which constitutes an understanding of the fundamental similarities and difference between the two types of seasonal forecasting skills and predictability and can offer important implications for the study of seasonal forecasting and verification.

Forest fire spatial modelling using ordered weighted averaging multi-criteria evaluation

Forest fires are a major environmental issue because they are increasing as a consequence of climate change and global warming. The present study was aimed to model forest fire hazard using the ordered weighted averaging (OWA) multi-criteria evaluation algorithm and to determine the role of human, climatic, and environmental factors in forest fire occurrence within the Golestan National Park (GNP), Iran. The database used for the present study was created according to daily classification of climate changes, environmental basic maps, and human-made influential forest fire factors. In the study area, the forest fires were registered using GPS. Expert opinions were applied through the analytic hierarchy process (AHP) to determine the importance of effective factors. Fuzzy membership functions were used to standardize the thematic layers. The fire risk maps were prepared using different OWA scenarios for man-made, climatic, and environment factors. The findings revealed that roads (weight = 0.288), rainfalls (weight = 0.288), and aspects (weight = 0.255) are the major factors that contribute to the occurrence of forest fire in the study area. The forest fire maps prepared from different scenarios were validated using the relative operating characteristic (ROC) curve. Values of forest fire maps acquired from scenarios of human, environment, climate factors and their combination were 0.87, 0.731, 0.773 and 0.819, respectively.

Adjusted T SPOT.TB Criteria Can Increase the Specificity of Diagnosis When Differentiating Spinal Tuberculosis From Other Spinal Infections

Background: The ability of T-SPOT.TB to differentiate Mycobacterium tuberculosis infection of the spine from other infections is little known. This study quantified the efficiency, sensitivity, and specificity of the T-SPOT.TB assay to distinguish between spinal tuberculosis (STB) caused by M. tuberculosis and other infections of the spine and evaluated whether diagnostic performance was improved by adjusting the T-SPOT.TB assay criteria. Methods: From January 2010 to May 2020, 147 patients with spinal infections were recruited. Peripheral blood mononuclear cells were collected, and the number of spot-forming cells was observed. Patients’ white blood cell (WBC) counts, C-reactive protein (CRP), erythrocyte sedimentation rate (ESR), procalcitonin (PCT), and TB antibodies were recorded. Specimen/tissue bacteriological culture was the reference standard for sensitivity and specificity. Results: There were 77 (52.4%) participants with confirmed TB and 70 (47.6%) with other infections. The groups were comparable in T-SPOT.TB assay results, age, sex, lesions in the segments, WBC count, CRP, procalcitonin, ESR, and TB antibodies. The sensitivity and specificity of the T-SPOT.TB assay for identifying STB was 88.3% and 40.0%, respectively. On the basis of Relative operating characteristic curve (ROC) analysis and the Youden index, when we adjusted the T-SPOT.TB assay’s diagnostic criteria, ESAT-6>12 or CFP-10>19,the sensitivity and specificity of the T-SPOT.TB assay for identifying STB was 83.1% and 64.3%, respectively. Conclusion: The T-SPOT.TB assay has great sensitivity to distinguish STB from other spinal infections; however, the specificity is extremely low. Specificity can be significantly improved while sensitivity is guaranteed by adjusting the diagnostic criteria.

Predictability of Seasonal Streamflow Forecasting Based on CSM: Case Studies of Top Three Largest Rivers in China

Accurate seasonal streamflow forecasting is important in reservoir operation, watershed planning, and water resource management, and streamflow forecasting is often based on hydrological models driven by coupled global climate models (CGCMs). To understand streamflow forecasting predictability, this study considered the three largest rivers in China and explored deterministic and probabilistic skill metrics on the monthly scale according to ensemble streamflow hindcasts from the hydrological model Hydrologiska Byråns Vattenbalansavdelning (HBV) driven by multiple climate forcings from the climate system model by the Beijing Climate Center (BCC_CSM1.1m). The effects of initial conditions (ICs) and meteorological forcings (MFs) on skill were investigated using the conventional ensemble streamflow prediction (ESP) and reverse-ESP (revESP). The results revealed the following: (1) Skill declines as lead time increases, and forecasting is generally the most skillful for lead month 1; (2) skill is higher for dry rivers than wet rivers, and higher for dry target months than wet months for the Yellow and Yangtze Rivers, suggesting greater skill in potential drought forecasting than flood forecasting; (3) the relative operating characteristic (ROC) area is greater for abnormal terciles than the near-normal tercile for all three rivers, greater for the above-normal tercile than the below-normal tercile for the Yellow and Yangtze Rivers, but slightly greater for the below-normal tercile than the above-normal tercile for the Xijiang River; and (4) the influence of ICs outweighs that of MFs in dry months, and the period of influence varies from 1 to 3 months; however, the influence of MFs is dominant in wet target months. These findings will help improve the understanding of both the seasonal streamflow forecasting predictability based on coupled climate system/hydrological models and of streamflow forecasting for variable rivers and seasons.