Long-Term Performance Metrics for National Weather Service Tornado Warnings

Abstract Tornado warnings are one of the flagship products of the National Weather Service. We update the time series of various metrics of performance in order to provide baselines over the 1986–2016 period for lead time, probability of detection, false alarm ratio, and warning duration. We have used metrics (mean lead time for tornadoes warned in advance, fraction of tornadoes warned in advance) that work in a consistent way across the official changes in policy for warning issuance, as well as across points in time when unofficial changes took place. The mean lead time for tornadoes warned in advance was relatively constant from 1986 to 2011, while the fraction of tornadoes warned in advance increased through about 2006, and the false alarm ratio slowly decreased. The largest changes in performance take place in 2012 when the default warning duration decreased, and there is an apparent increased emphasis on reducing false alarms. As a result, the lead time, probability of detection, and false alarm ratio all decrease in 2012. Our analysis is based, in large part, on signal detection theory, which separates the quality of the warning system from the threshold for issuing warnings. Threshold changes lead to trade-offs between false alarms and missed detections. Such changes provide further evidence for changes in what the warning system as a whole considers important, as well as highlighting the limitations of measuring performance by looking at metrics independently.

Download Full-text

An Analysis of 2016-2018 Tornadoes and National Weather Service Tornado Warnings across the contiguous United States

Weather and Forecasting ◽

10.1175/waf-d-20-0241.1 ◽

2021 ◽

Author(s):

Evan S. Bentley ◽

Richard L. Thompson ◽

Barry R. Bowers ◽

Justin G. Gibbs ◽

Steven E. Nelson

Keyword(s):

United States ◽

Rotational Velocity ◽

Radar Data ◽

Probability Of Detection ◽

False Alarms ◽

National Weather Service ◽

Low Level ◽

Tornado Warnings ◽

Tornado Warning ◽

Weather Service

AbstractPrevious work has considered tornado occurrence with respect to radar data, both WSR-88D and mobile research radars, and a few studies have examined techniques to potentially improve tornado warning performance. To date, though, there has been little work focusing on systematic, large-sample evaluation of National Weather Service (NWS) tornado warnings with respect to radar-observable quantities and the near-storm environment. In this work, three full years (2016–2018) of NWS tornado warnings across the contiguous United States were examined, in conjunction with supporting data in the few minutes preceding warning issuance, or tornado formation in the case of missed events. The investigation herein examines WSR-88D and Storm Prediction Center (SPC) mesoanalysis data associated with these tornado warnings with comparisons made to the current Warning Decision Training Division (WDTD) guidance.Combining low-level rotational velocity and the significant tornado parameter (STP), as used in prior work, shows promise as a means to estimate tornado warning performance, as well as relative changes in performance as criteria thresholds vary. For example, low-level rotational velocity peaking in excess of 30 kt (15 m s−1), in a near-storm environment which is not prohibitive for tornadoes (STP > 0), results in an increased probability of detection and reduced false alarms compared to observed NWS tornado warning metrics. Tornado warning false alarms can also be reduced through limiting warnings with weak (<30 kt), broad (>1nm) circulations in a poor (STP=0) environment, careful elimination of velocity data artifacts like sidelobe contamination, and through greater scrutiny of human-based tornado reports in otherwise questionable scenarios.

Download Full-text

The Influence of Weather Watch Type on the Quality of Tornado Warnings and its Implications for Future Forecasting Systems

Weather and Forecasting ◽

10.1175/waf-d-21-0052.1 ◽

2021 ◽

Author(s):

Makenzie J. Krocak ◽

Harold E. Brooks

Keyword(s):

False Alarm ◽

Lead Time ◽

Severe Weather ◽

Probability Of Detection ◽

False Alarm Ratio ◽

Tornado Warnings ◽

General Improvement ◽

The Difference ◽

Tornado Warning

AbstractWhile many studies have looked at the quality of forecast products, few have attempted to understand the relationship between them. We begin to consider whether or not such an influence exists by analyzing storm-based tornado warning product metrics with respect to whether they occurred within a severe weather watch and, if so, what type of watch they occurred within.The probability of detection, false alarm ratio, and lead time all show a general improvement with increasing watch severity. In fact, the probability of detection increased more as a function of watch-type severity than the change in probability of detection during the time period of analysis. False alarm ratio decreased as watch type increased in severity, but with a much smaller magnitude than the difference in probability of detection. Lead time also improved with an increase in watch-type severity. Warnings outside of any watch had a mean lead time of 5.5 minutes, while those inside of a particularly dangerous situation tornado watch had a mean lead time of 15.1 minutes. These results indicate that the existence and type of severe weather watch may have an influence on the quality of tornado warnings. However, it is impossible to separate the influence of weather watches from possible differences in warning strategy or differences in environmental characteristics that make it more or less challenging to warn for tornadoes. Future studies should attempt to disentangle these numerous influences to assess how much influence intermediate products have on downstream products.

Download Full-text

False Alarms, Tornado Warnings, and Tornado Casualties

Weather Climate and Society ◽

10.1175/2009wcas1005.1 ◽

2009 ◽

Vol 1 (1) ◽

pp. 38-53 ◽

Cited By ~ 76

Author(s):

Kevin M. Simmons ◽

Daniel Sutter

Keyword(s):

False Alarm ◽

The United States ◽

Probability Of Detection ◽

False Alarms ◽

Lead Times ◽

Time Intervals ◽

Standard Deviation Increase ◽

False Alarm Ratio ◽

Tornado Warnings ◽

The Impact

Abstract This paper extends prior research on the societal value of tornado warnings to the impact of false alarms. Intuition and theory suggest that false alarms will reduce the response to warnings, yet little evidence of a “false alarm effect” has been unearthed. This paper exploits differences in the false-alarm ratio across the United States to test for a false-alarm effect in a regression model of tornado casualties from 1986 to 2004. A statistically significant and large false-alarm effect is found: tornadoes that occur in an area with a higher false-alarm ratio kill and injure more people, everything else being constant. The effect is consistent across false-alarm ratios defined over different geographies and time intervals. A one-standard-deviation increase in the false-alarm ratio increases expected fatalities by between 12% and 29% and increases expected injuries by between 14% and 32%. The reduction in the national tornado false-alarm ratio over the period reduced fatalities by 4%–11% and injuries by 4%–13%. The casualty effects of false alarms and warning lead times are approximately equal in magnitude, suggesting that the National Weather Service could not reduce casualties by trading off a higher probability of detection for a higher false-alarm ratio, or vice versa.

Download Full-text

Examining sub-daily tornado warning performance and associated environmental characteristics

Weather and Forecasting ◽

10.1175/waf-d-21-0097.1 ◽

2021 ◽

Author(s):

Makenzie J. Krocak ◽

Matthew D. Flournoy ◽

Harold E. Brooks

Keyword(s):

False Alarm ◽

Probability Of Detection ◽

Broad Sense ◽

False Alarms ◽

Environmental Characteristics ◽

False Alarm Ratio ◽

Intermediate Products ◽

Tornado Warning ◽

Operational Goal

AbstractIncreasing tornado warning skill in terms of the probability of detection and false alarm ratio remains an important operational goal. Although many studies have examined tornado warning performance in a broad sense, less focus has been placed on warning performance within sub-daily convective events. In this study, we use the NWS tornado verification database to examine tornado warning performance by order-of-tornado within each convective day. We combine this database with tornado reports to relate warning performance to environmental characteristics. On convective days with multiple tornadoes, the first tornado is warned significantly less often than the middle and last tornadoes. More favorable kinematic environmental characteristics, like increasing 0–1-km shear and storm-relative helicity, are associated with better warning performance related to the first tornado of the convective day. Thermodynamic and composite parameters are less correlated to warning performance. During tornadic events, over half of false alarms occur after the last tornado of the day decays, and false alarms are twice as likely to be issued during this time than before the first tornado forms. These results indicate that forecasters may be better “primed” (or more prepared) to issue warnings on middle and last tornadoes of the day, and must overcome a higher threshold to warn on the first tornado of the day. To overcome this challenge, using kinematic environmental characteristics and intermediate products on the watch-to-warning scale may help.

Download Full-text

Characteristics of Tornado Events and Warnings in the Southeastern United States

Weather and Forecasting ◽

10.1175/waf-d-18-0211.1 ◽

2019 ◽

Vol 34 (4) ◽

pp. 1017-1034 ◽

Cited By ~ 7

Author(s):

Alexandra K. Anderson-Frey ◽

Yvette P. Richardson ◽

Andrew R. Dean ◽

Richard L. Thompson ◽

Bryan T. Smith

Keyword(s):

United States ◽

False Alarm ◽

Southeastern United States ◽

Probability Of Detection ◽

High Shear ◽

Related Literature ◽

False Alarm Ratio ◽

Tornado Warnings ◽

Tornado Warning

Abstract The southeastern United States has become a prime area of focus in tornado-related literature due, in part, to the abundance of tornadoes occurring in high-shear low-CAPE (HSLC) environments. Through this analysis of 4133 tornado events and 16 429 tornado warnings in the southeastern United States, we find that tornadoes in the Southeast do indeed have, on average, higher shear and lower CAPE than tornadoes elsewhere in the contiguous United States (CONUS). We also examine tornado warning skill in the form of probability of detection (POD; percent of tornadoes receiving warning prior to tornado occurrence) and false alarm ratio (FAR; percent of tornado warnings for which no corresponding tornado is detected), and find that, on average, POD is better and FAR is worse for tornadoes in the Southeast than for the CONUS as a whole. These measures of warning skill remain consistent even when we consider only HSLC tornadoes. The Southeast also has nearly double the CONUS percentage of deadly tornadoes, with the highest percentage of these deadly tornadoes occurring during the spring, the winter, and around local sunset. On average, however, the tornadoes with the lowest POD also tend to be those that are weakest and least likely to be deadly; for the most part, the most dangerous storms are indeed being successfully warned.

Download Full-text

National Weather Service Severe Weather Warnings as Threats-in-Motion

Weather and Forecasting ◽

10.1175/waf-d-20-0159.1 ◽

2021 ◽

Author(s):

Gregory J. Stumpf ◽

Alan E. Gerard

Keyword(s):

False Alarm ◽

Lead Time ◽

Severe Weather ◽

Lead Times ◽

National Weather Service ◽

Severe Thunderstorm ◽

Environmental Threats ◽

Tornado Warnings ◽

Average Lead ◽

Tornado Outbreaks

AbstractThreats-in-Motion (TIM) is a warning generation approach that would enable the NWS to advance severe thunderstorm and tornado warnings from the current static polygon system to continuously updating polygons that move forward with a storm. This concept is proposed as a first stage for implementation of the Forecasting a Continuum of Environmental Threats (FACETs) paradigm, which eventually aims to deliver rapidly updating probabilistic hazard information alongside NWS warnings, watches, and other products.With TIM, a warning polygon is attached to the threat and moves forward along with it. This provides more uniform, or equitable, lead time for all locations downstream of the event. When forecaster workload is high, storms remain continually tracked and warned. TIM mitigates gaps in warning coverage and improves the handling of storm motion changes. In addition, warnings are automatically cleared from locations where the threat has passed. This all results in greater average lead times and lower average departure times than current NWS warnings, with little to no impact to average false alarm time. This is particularly noteworthy for storms expected to live longer than the average warning duration (30 or 45 minutes) such as long-tracked supercells that are more prevalent during significant tornado outbreaks.

Download Full-text

Radar Observations of Tornado-Warned Convection Associated with Tropical Cyclones over Florida

Weather and Forecasting ◽

10.1175/waf-d-16-0105.1 ◽

2017 ◽

Vol 32 (1) ◽

pp. 165-186 ◽

Cited By ~ 7

Author(s):

Steven M. Martinaitis

Keyword(s):

False Alarm ◽

Tropical Cyclones ◽

Rotational Velocity ◽

Elevation Angle ◽

Probability Of Detection ◽

Polarization Data ◽

False Alarm Ratio ◽

Tornado Warnings ◽

Tornado Warning ◽

Velocity Enhancement

Abstract Statistical evaluations of tornado warnings issued during recent tropical cyclone events yielded an above-average false alarm ratio. This study analyzed tornado-warned convection associated with Tropical Storms Debby (2012) and Andrea (2013) using superresolution and dual-polarization data from Weather Surveillance Radar-1988 Doppler radars located throughout the Florida peninsula to identify precursor characteristics and signatures that would distinguish tornadic events prior to tornadogenesis. A series of radar-based interrogation guidance at varying ranges from radar is presented to help facilitate the reduction of the tornado-warning false alarm ratio without compromising the probability of detection. For convection within 74.1 km from the nearest radar, low-level velocity characteristics include a rotational velocity ≥ 10.3 m s−1 (20 kt), shear across the rotation ≥ 0.010 s−1, and a contracting rotation diameter. The convection should also exhibit supercell reflectivity signatures and at least a mesocyclone velocity enhancement signature or horizontal separation of greater ZDR and KDP values. Guidance at a range from 74.1 to 129.6 km is similar except for not requiring the presence of a supercell reflectivity signature and the change of the rotational velocity guidance to ≥7.7 m s−1 (15 kt) at the 0.5°-elevation angle. Convection at a range beyond 129.6 km only requires a rotational velocity ≥ 7.7 m s−1 (15 kt) at the 0.5°-elevation angle. Evaluation of the radar interrogation guidance for tornadic events and tornado-warned convection for six tropical cyclones reduced the number of false alarm events by 28.9% and reduced the false alarm ratio from 0.740 to 0.669.

Download Full-text

An approach of constant false alarm ratio for improved target tracking

Scientific Technical Review ◽

10.5937/str2003017r ◽

2020 ◽

Vol 70 (3) ◽

pp. 17-23

Author(s):

Zvonko Radosavljević ◽

Dejan Ivković

Keyword(s):

False Alarm ◽

Tracking Error ◽

Probability Of Detection ◽

Interacting Multiple Model ◽

Multiple Model ◽

Moving Targets ◽

False Alarm Ratio ◽

Motion Models ◽

Areas Of Interest

Each radar has the function of surveillance of certain areas of interest. In particular, the radar also has the function of tracking moving targets in that territory with some probability of detection, which depends on the type of detector. Constant false alarm ratio (CFAR) is a very commonly used detector. Changing the probability of target detection can directly affect the quality of tracking the moving targets. The paper presents the theoretical basis of the influence of CFAR detectors on the quality of tracking, as well as an approach to the selection of CFAR detectors, CATM CFAR, which enables better monitoring by the Interacting Multiple Model (IMM) algorithm with two motion models. Comparative analysis of CA and CATM algorithm realized by numerical simulations has shown that CATM CFAR gives less tracking error with proportionally the same computer resources.

Download Full-text

Cry Wolf Effect? Evaluating the Impact of False Alarms on Public Responses to Tornado Alerts in the Southeastern United States

Weather Climate and Society ◽

10.1175/wcas-d-18-0080.1 ◽

2019 ◽

Vol 11 (3) ◽

pp. 549-563 ◽

Cited By ~ 8

Author(s):

JungKyu Rhys Lim ◽

Brooke Fisher Liu ◽

Michael Egnoto

Keyword(s):

United States ◽

False Alarm ◽

Protective Action ◽

Southeastern United States ◽

The United States ◽

False Alarms ◽

Protective Behavior ◽

Tornado Warnings ◽

Research Findings ◽

The Impact

Abstract On average, 75% of tornado warnings in the United States are false alarms. Although forecasters have been concerned that false alarms may generate a complacent public, only a few research studies have examined how the public responds to tornado false alarms. Through four surveys (N = 4162), this study examines how residents in the southeastern United States understand, process, and respond to tornado false alarms. The study then compares social science research findings on perceptions of false alarms to actual county false alarm ratios and the number of tornado warnings issued by counties. Contrary to prior research, findings indicate that concerns about false alarm ratios generating a complacent public may be overblown. Results show that southeastern U.S. residents estimate tornado warnings to be more accurate than they are. Participants’ perceived false alarm ratios are not correlated with actual county false alarm ratios. Counterintuitively, the higher individuals perceive false alarm ratios and tornado alert accuracy to be, the more likely they are to take protective behavior such as sheltering in place in response to tornado warnings. Actual country false alarm ratios and the number of tornado warnings issued did not predict taking protective action.

Download Full-text

Tornado Warning Decisions Using Phased-Array Radar Data

Weather and Forecasting ◽

10.1175/waf-d-14-00042.1 ◽

2015 ◽

Vol 30 (1) ◽

pp. 57-78 ◽

Cited By ~ 29

Author(s):

Pamela Heinselman ◽

Daphne LaDue ◽

Darrel M. Kingfield ◽

Robert Hoffman

Keyword(s):

Lead Time ◽

Phased Array ◽

Cognitive Task ◽

Radar Data ◽

Probability Of Detection ◽

Probability Of False Alarm ◽

Temporal Sampling ◽

Phased Array Radar ◽

Tornado Warnings ◽

Tornado Warning

Abstract The 2012 Phased Array Radar Innovative Sensing Experiment identified how rapidly scanned full-volumetric data captured known mesoscale processes and impacted tornado-warning lead time. Twelve forecasters from nine National Weather Service forecast offices used this rapid-scan phased-array radar (PAR) data to issue tornado warnings on two low-end tornadic and two nontornadic supercell cases. Verification of the tornadic cases revealed that forecasters’ use of PAR data provided a median tornado-warning lead time (TLT) of 20 min. This 20-min TLT exceeded by 6.5 and 9 min, respectively, participants’ forecast office and regions’ median spring season, low-end TLTs (2008–13). Furthermore, polygon-based probability of detection ranged from 0.75 to 1.0 and probability of false alarm for all four cases ranged from 0.0 to 0.5. Similar performance was observed regardless of prior warning experience. Use of a cognitive task analysis method called the recent case walk-through showed that this performance was due to forecasters’ use of rapid volumetric updates. Warning decisions were based upon the intensity, persistence, and important changes in features aloft that are precursors to tornadogenesis. Precursors that triggered forecasters’ decisions to warn occurred within one or two typical Weather Surveillance Radar-1988 Doppler (WSR-88D) scans, indicating PAR’s temporal sampling better matches the time scale at which these precursors evolve.

Download Full-text