scholarly journals Overview of spatial verification methods and their application to ensemble forecasting

2021 ◽  
Vol 4 ◽  
pp. 30-49
Author(s):  
A.Yu. Bundel ◽  
◽  
A.V. Muraviev ◽  
E.D. Olkhovaya ◽  
◽  
...  
Keyword(s):  
High Resolution ◽  
Ensemble Forecasting ◽  
Object Based ◽  
Verification Methods ◽  
International Projects ◽  
Resolution Model ◽  
High Gradients ◽  
High Resolution Models ◽  
High Degree ◽  
Spatial Verification

State-of-the-art high-resolution NWP models simulate mesoscale systems with a high degree of detail, with large amplitudes and high gradients of fields of weather variables. Higher resolution leads to the spatial and temporal error growth and to a well-known double penalty problem. To solve this problem, the spatial verification methods have been developed over the last two decades, which ignore moderate errors (especially in the position), but can still evaluate the useful skill of a high-resolution model. The paper refers to the updated classification of spatial verification methods, briefly describes the main methods, and gives an overview of the international projects for intercomparison of the methods. Special attention is given to the application of the spatial approach to ensemble forecasting. Popular software packages are considered. The Russian translation is proposed for the relevant English terms. Keywords: high-resolution models, verification, double penalty, spatial methods, ensemble forecasting, object-based methods

High-resolution model Verification Evaluation (HiVE). Part 1: Using neighbourhood techniques for the assessment of ocean model forecast skill

2020 ◽  
Author(s):  
Jan Maksymczuk ◽  
Ric Crocker ◽  
Marion Mittermaier ◽  
Christine Pequignet
Keyword(s):  
High Resolution ◽  
Weather Prediction ◽  
Ocean Model ◽  
Forecast Skill ◽  
Model Verification ◽  
Model Forecast ◽  
Verification Methods ◽  
Resolution Model ◽  
The Right ◽  
Spatial Verification

<div> <p>HiVE is a CMEMS funded collaboration between the atmospheric Numerical Weather Prediction (NWP) verification and the ocean community within the Met Office, aimed at demonstrating the use of spatial verification methods originally developed for the evaluation of high-resolution NWP forecasts, with CMEMS ocean model forecast products. Spatial verification methods provide more scale appropriate ways to better assess forecast characteristics and accuracy of km-scale forecasts, where the detail looks realistic but may not be in the right place at the right time. As a result, it can be the case that coarser resolution forecasts verify better (e.g. lower root-mean-square-error) than the higher resolution forecast. In this instance the smoothness of the coarser resolution forecast is rewarded, though the higher-resolution forecast may be better. The project utilised open source code library known as Model Evaluation Toolkit (MET) developed at the US National Center for Atmospheric Research. </p> </div><div> <p> </p> </div><div> <p>This project saw, for the first time, the application of spatial verification methods to sub-10 km resolution ocean model forecasts. The project consisted of two parts. Part 1 describes an assessment of the forecast skill for SST of CMEMS model configurations at observing locations using an approach called HiRA (High Resolution Assessment). Part 2 is described in the companion poster to this one.  </p> </div><div> <p> </p> </div><div> <p>HiRA is a single-observation-forecast-neighbourhood-type method which makes use of commonly used ensemble verification metrics such as the Brier Score (BS) and the Continuous Ranked Probability Score (CRPS). In this instance all model grid points within a predefined neighbourhood of the observing location are considered equi-probable outcomes (or pseudo-ensemble members) at the observing location. The technique allows for an inter-comparison of models with different grid resolutions as well as between deterministic and probabilistic forecasts in an equitable and consistent way. In this work it has been applied to the CMEMS products delivered from the AMM7 (~7km) and AMM15 (~1.5km) model configurations for the European North West Shelf that are provided by the Met Office. </p> </div><div> <p> </p> </div><div> <p>It has been found that when neighbourhoods of equivalent extent are compared for both configurations it is possible to show improved forecast skill for SST for the higher resolution AMM15 both on- and off-shelf, which has been difficult to demonstrate previously using traditional metrics. Forecast skill generally degrades with increasing lead time for both configurations, with the off-shelf results for the higher resolution model showing increasing benefits over the coarser configuration. </p> </div>

High-resolution model Verification Evaluation (HiVE). Part 2: Using object-based methods for the evaluation of algal blooms

2020 ◽  
Author(s):  
Marion Mittermaier ◽  
Rachel North ◽  
Christine Pequignet ◽  
Jan Maksymczuk
Keyword(s):  
High Resolution ◽  
Algal Blooms ◽  
Weather Prediction ◽  
Ocean Model ◽  
Model Verification ◽  
Forecast Performance ◽  
Object Based ◽  
Verification Methods ◽  
The Right ◽  
Spatial Verification

<div> <p>HiVE is a CMEMS funded collaboration between the atmospheric Numerical Weather Prediction (NWP) verification and the ocean community within the Met Office, aimed at demonstrating the use of spatial verification methods originally developed for the evaluation of high-resolution NWP forecasts, to CMEMS ocean model forecast products. Spatial verification methods provide more scale appropriate ways to better assess forecast characteristics and accuracy of km-scale forecasts, where the detail looks realistic but may not be in the right place at the right time. As a result, it can be the case that coarser resolution forecasts verify better (e.g. lower root-mean-square-error) than the higher resolution forecast. In this instance the smoothness of the coarser resolution forecast is rewarded, though the higher-resolution forecast may be better. The project utilised open source code library known as Model Evaluation Tools (MET) developed at the US National Center for Atmospheric Research (NCAR).</p> </div><div> <p> </p> </div><div> <p>This project saw, for the first time, the application of spatial verification methods to sub-10 km resolution ocean model forecasts. The project consisted of two parts. Part 1 is described in the companion poster to this one. Part 2 describes the skill of CMEMS products for forecasting events or features of interest such as algal blooms.  </p> </div><div> <p> </p> </div><div> <p>The Method for Object-based Diagnostic Evaluation (MODE) and the time dimension version MODE Time Domain (MTD) were applied to daily mean chlorophyll forecasts for the European North West Shelf from the FOAM-ERSEM model on the AMM7 grid. The forecasts are produced from a “cold start”, i.e. no data assimilation of biological variables. Here the entire 2019 algal bloom season was analysed to understand: intensity and spatial (area) biases; location and timing errors. Forecasts were compared to the CMEMS daily cloud free (L4) multi-sensor chlorophyll-<em>a</em> product. </p> </div><div> <p> </p> </div><div> <p>It has been found that there are large differences between forecast and observed concentrations of chlorophyll. This has meant that a quantile mapping approach for removing the bias was necessary before analysing the spatial properties of the forecast. Despite this the model still produces areas of chlorophyll which are too large compared to the observed. The model often produces areas of enhanced chlorophyll in approximately the right locations but the forecast and observed areas are rarely collocated and/or overlapping. Finally, the temporal analysis shows that the model struggled to get the onset of the season (being close to a month too late), but once the model picked up the signal there was better correspondence between the observed and forecast chlorophyll peaks for the remainder of the season. There was very little variation in forecast performance with lead time, suggesting that chlorophyll is a very slowly varying quantity.  </p> </div><div> <p> </p> </div><div> <p>Comparing an analysis which included the assimilation of observed chlorophyll shows that it is much closer to the observed L4 product than the non-biological assimilative analysis. It must be concluded that if the forecast were started from a DA analysis that included chlorophyll, it would lead to forecasts with less bias, and possibly a better detection of the onset of the bloom.  </p> </div><div> <p> </p> </div>

Neighborhood-Based Contingency Tables Including Errors Compensation

2019 ◽  
Vol 147 (1) ◽  
pp. 329-344 ◽  
Author(s):  
Joël Stein ◽  
Fabien Stoop
Keyword(s):  
High Resolution ◽  
Large Scale ◽  
Contingency Tables ◽  
Global Model ◽  
New Method ◽  
False Alarms ◽  
Scale Models ◽  
Resolution Model ◽  
High Resolution Model ◽  
High Resolution Models

Some specific scores use a neighborhood strategy in order to reduce double penalty effects, which penalize high-resolution models, compared to large-scale models. Contingency tables based on this strategy have already been proposed, but can sometimes display undesirable behavior. A new method of populating contingency tables is proposed: pairs of missed events and false alarms located in the same local neighborhood compensate in order to give pairs of hits and correct rejections. Local tables are summed up so as to provide the final table for the whole verification domain. It keeps track of the bias of the forecast when neighborhoods are taken into account. Moreover, the scores computed from this table depend on the distance between forecast and observed patterns. This method is applied to binary and multicategorical events in a simplified framework so as to present the method and to compare the new tables with previous neighborhood-based contingency tables. The new tables are then used for the verification of two models operational at Météo-France: AROME, a high-resolution model, and ARPEGE, a large-scale global model. The comparison of several contingency scores shows that the importance of the double penalty decreases more for AROME than for ARPEGE when the neighboring size increases. Scores designed for rare events are also applied to these neighborhood-based contingency tables.

scholarly journals Application of Two Spatial Verification Methods to Ensemble Forecasts of Low-Level Rotation

2016 ◽  
Vol 31 (3) ◽  
pp. 713-735 ◽  
Author(s):  
Patrick S. Skinner ◽  
Louis J. Wicker ◽  
Dustan M. Wheatley ◽  
Kent H. Knopfmeier
Keyword(s):  
Displacement Component ◽  
Ensemble Forecasts ◽  
Matching Algorithm ◽  
Low Level ◽  
Object Based ◽  
Verification Methods ◽  
Ensemble Mean ◽  
Dimensional Object ◽  
Verification Techniques ◽  
Spatial Verification

Abstract Two spatial verification methods are applied to ensemble forecasts of low-level rotation in supercells: a four-dimensional, object-based matching algorithm and the displacement and amplitude score (DAS) based on optical flow. Ensemble forecasts of low-level rotation produced using the National Severe Storms Laboratory (NSSL) Experimental Warn-on-Forecast System are verified against WSR-88D single-Doppler azimuthal wind shear values interpolated to the model grid. Verification techniques are demonstrated using four 60-min forecasts issued at 15-min intervals in the hour preceding development of the 20 May 2013 Moore, Oklahoma, tornado and compared to results from two additional forecasts of tornadic supercells occurring during the springs of 2013 and 2014. The object-based verification technique and displacement component of DAS are found to reproduce subjectively determined forecast characteristics in successive forecasts for the 20 May 2013 event, as well as to discriminate in subjective forecast quality between different events. Ensemble-mean, object-based measures quantify spatial and temporal displacement, as well as storm motion biases in predicted low-level rotation in a manner consistent with subjective interpretation. Neither method produces useful measures of the intensity of low-level rotation, owing to deficiencies in the verification dataset and forecast resolution.

scholarly journals Neighborhood Verification: A Strategy for Rewarding Close Forecasts

2009 ◽  
Vol 24 (6) ◽  
pp. 1498-1510 ◽  
Author(s):  
Elizabeth E. Ebert
Keyword(s):  
High Resolution ◽  
Rainfall Intensity ◽  
Mean Value ◽  
Forecast Performance ◽  
Fractional Coverage ◽  
Optimal Scale ◽  
Verification Methods ◽  
Intercomparison Project ◽  
The Mean ◽  
Spatial Verification

Abstract High-resolution forecasts may be quite useful even when they do not match the observations exactly. Neighborhood verification is a strategy for evaluating the “closeness” of the forecast to the observations within space–time neighborhoods rather than at the grid scale. Various properties of the forecast within a neighborhood can be assessed for similarity to the observations, including the mean value, fractional coverage, occurrence of a forecast event sufficiently near an observed event, and so on. By varying the sizes of the neighborhoods, it is possible to determine the scales for which the forecast has sufficient skill for a particular application. Several neighborhood verification methods have been proposed in the literature in the last decade. This paper examines four such methods in detail for idealized and real high-resolution precipitation forecasts, highlighting what can be learned from each of the methods. When applied to idealized and real precipitation forecasts from the Spatial Verification Methods Intercomparison Project, all four methods showed improved forecast performance for neighborhood sizes larger than grid scale, with the optimal scale for each method varying as a function of rainfall intensity.

scholarly journals How Do Continuous High-Resolution Models of Patchy Seabed Habitats Enhance Classification Schemes?

Geosciences ◽  
2019 ◽  
Vol 9 (5) ◽  
pp. 237 ◽  
Author(s):  
Gustav Kågesten ◽  
Dario Fiorentino ◽  
Finn Baumgartner ◽  
Lovisa Zillén
Keyword(s):  
High Resolution ◽  
Multiple Scales ◽  
Natura 2000 ◽  
Boosted Regression Trees ◽  
Biological Information ◽  
Classification Schemes ◽  
Object Based ◽  
Thematic Resolution ◽  
High Resolution Models ◽  
The Baltic

Predefined classification schemes and fixed geographic scales are often used to simplify and cost-effectively map the spatial complexity of nature. These simplifications can however limit the usefulness of the mapping effort for users who need information across a different range of thematic and spatial resolutions. We demonstrate how substrate and biological information from point samples and photos, combined with continuous multibeam data, can be modeled to predictively map percentage cover conforming with multiple existing classification schemes (i.e., HELCOM HUB; Natura 2000), while also providing high-resolution (5 m) maps of individual substrate and biological components across a 1344 km2 offshore bank in the Baltic Sea. Data for substrate and epibenthic organisms were obtained from high-resolution photo mosaics, sediment grab samples, legacy data and expert annotations. Environmental variables included pixel and object based metrics at multiple scales (0.5 m–2 km), which improved the accuracy of models. We found that using Boosted Regression Trees (BRTs) to predict continuous models of substrate and biological components provided additional detail for each component without losing accuracy in the classified maps, compared with a thematic model. Results demonstrate the sensitivity of habitat maps to the effects of spatial and thematic resolution and the importance of high-resolution maps to management applications.

scholarly journals Assessing the Benefits of Convection-Permitting Models by Neighborhood Verification: Examples from MAP D-PHASE

2010 ◽  
Vol 138 (9) ◽  
pp. 3418-3433 ◽  
Author(s):  
Tanja Weusthoff ◽  
Felix Ament ◽  
Marco Arpagaus ◽  
Mathias W. Rotach
Keyword(s):  
High Resolution ◽  
Spatial Scales ◽  
Weather Prediction ◽  
Grid Point ◽  
Skill Score ◽  
Verification Methods ◽  
Driving Model ◽  
High Resolution Models ◽  
Precipitation Events ◽  
Better Than

Abstract High-resolution numerical weather prediction (NWP) models produce more detailed precipitation structures but the real benefit is probably the more realistic statistics gained with the higher resolution and not the information on the specific grid point. By evaluating three model pairs, each consisting of a high-resolution NWP system resolving convection explicitly and its low-resolution-driving model with parameterized convection, on different spatial scales and for different thresholds, this paper addresses the question of whether high-resolution models really perform better than their driving lower-resolution counterparts. The model pairs are evaluated by means of two fuzzy verification methods—upscaling (UP) and fractions skill score (FSS)—for the 6 months of the D-PHASE Operations Period and in a highly complex terrain. Observations are provided by the Swiss radar composite and the evaluation is restricted to the area covered by the Swiss radar stations. The high-resolution models outperform or equal the performance of their respective lower-resolution driving models. The differences between the models are significant and robust against small changes in the verification settings. An evaluation based on individual months shows that high-resolution models give better results, particularly with regard to convective, more localized precipitation events.

Beyond the Basics: Evaluating Model-Based Precipitation Forecasts Using Traditional, Spatial, and Object-Based Methods

2014 ◽  
Vol 29 (6) ◽  
pp. 1451-1472 ◽  
Author(s):  
Jamie K. Wolff ◽  
Michelle Harrold ◽  
Tressa Fowler ◽  
John Halley Gotway ◽  
Louisa Nance ◽  
...  
Keyword(s):  
Mesoscale Model ◽  
Skill Score ◽  
Careful Consideration ◽  
Diagnostic Information ◽  
Coverage Area ◽  
Object Based ◽  
Grid Approach ◽  
Verification Methods ◽  
Verification Techniques ◽  
Spatial Verification

Abstract While traditional verification methods are commonly used to assess numerical model quantitative precipitation forecasts (QPFs) using a grid-to-grid approach, they generally offer little diagnostic information or reasoning behind the computed statistic. On the other hand, advanced spatial verification techniques, such as neighborhood and object-based methods, can provide more meaningful insight into differences between forecast and observed features in terms of skill with spatial scale, coverage area, displacement, orientation, and intensity. To demonstrate the utility of applying advanced verification techniques to mid- and coarse-resolution models, the Developmental Testbed Center (DTC) applied several traditional metrics and spatial verification techniques to QPFs provided by the Global Forecast System (GFS) and operational North American Mesoscale Model (NAM). Along with frequency bias and Gilbert skill score (GSS) adjusted for bias, both the fractions skill score (FSS) and Method for Object-Based Diagnostic Evaluation (MODE) were utilized for this study with careful consideration given to how these methods were applied and how the results were interpreted. By illustrating the types of forecast attributes appropriate to assess with the spatial verification techniques, this paper provides examples of how to obtain advanced diagnostic information to help identify what aspects of the forecast are or are not performing well.

scholarly journals Quantifying Extreme Precipitation Forecasting Skill in High-Resolution Models Using Spatial Patterns: A Case Study of the 2016 and 2018 Ellicott City Floods

Atmosphere ◽  
2020 ◽  
Vol 11 (2) ◽  
pp. 136
Author(s):  
Stephanie E. Zick
Keyword(s):  
Climate Change ◽  
High Resolution ◽  
Extreme Precipitation ◽  
Stage Iv ◽  
Precipitation Forecasting ◽  
Object Based ◽  
Detailed Surface ◽  
Spatial Attributes ◽  
High Resolution Models ◽  
Environmental Prediction

Recent historic floods in Ellicott City, MD, on 30 July 2016 and 27 May 2018 provide stark examples of the types of floods that are expected to become more frequent due to urbanization and climate change. Given the profound impacts associated with flood disasters, it is crucial to evaluate the capability of state-of-the-art weather models in predicting these hydrometeorological events. This study utilizes an object-based approach to evaluate short range (<12 h) hourly forecast precipitation from the High-Resolution Rapid Refresh (HRRR) versus observations from the National Centers for Environmental Prediction (NCEP) Stage IV precipitation analysis. For both datasets, a binary precipitation field is delineated using thresholds that span trace to extreme precipitation rates. Next, spatial metrics of area, perimeter, solidity, elongation, and fragmentation, as well as centroid positions for the forecast and observed fields are calculated. A Mann–Whitney U-test reveals biases (using a confidence level of 90%) related to the spatial attributes and locations of model forecast precipitation. Results indicate that traditional pixel-based precipitation verification metrics are limited in their ability to quantify and characterize model skill. In contrast, an object-based methodology offers encouraging results in that the HRRR can skillfully predict the extreme precipitation rates that are anticipated with anthropogenic climate change. Yet, there is still room for improvement, since model forecasts of extreme convective rainfall tend to be slightly too numerous and fragmented compared with observations. Lastly, results are sensitive to the HRRR model’s representation of synoptic-scale and mesoscale processes. Therefore, detailed surface analyses and an “ingredients-based” approach should remain central to the process of forecasting excessive rainfall.

scholarly journals Impact of the Atmospheric Mean State on Tropical Instability Wave Activity

2012 ◽  
Vol 25 (7) ◽  
pp. 2341-2355 ◽  
Author(s):  
Yukiko Imada ◽  
Masahide Kimoto ◽  
Xianyan Chen
Keyword(s):  
High Resolution ◽  
Negative Feedback ◽  
Mean Field ◽  
Surface Current ◽  
Atmospheric Model ◽  
Instability Wave ◽  
Tropical Instability Waves ◽  
Resolution Model ◽  
The Difference ◽  
High Resolution Models

Abstract The features of simulated tropical instability waves (TIWs) in the Pacific Ocean are compared between atmospheric models of two different resolutions coupled with a uniform oceanic model. Results show that TIWs are more active in the high-resolution model, even though it includes atmospheric negative feedback. Such negative feedback is not identified in the low-resolution atmospheric model because of the absence of atmospheric responses. Comparison of the energetics between the two models shows that the large TIW activity in the higher-resolution model is due to the difference in barotropic energy sources near the surface. A high-resolution atmosphere results in a tighter intertropical convergence zone and associated stronger wind curl and shear. This causes a stronger surface current shear between the South Equatorial Current (SEC) and North Equatorial Counter Current (NECC), which is one of the main sources of TIW kinetic energy. These results indicate the important role of the atmospheric mean field on TIW activity and the advantage of using high-resolution models to represent coupling among multiscale phenomena.

Sign in / Sign up

Export Citation Format

Share Document