scholarly journals Neighborhood Verification: A Strategy for Rewarding Close Forecasts

2009 ◽  
Vol 24 (6) ◽  
pp. 1498-1510 ◽  
Author(s):  
Elizabeth E. Ebert

Abstract High-resolution forecasts may be quite useful even when they do not match the observations exactly. Neighborhood verification is a strategy for evaluating the “closeness” of the forecast to the observations within space–time neighborhoods rather than at the grid scale. Various properties of the forecast within a neighborhood can be assessed for similarity to the observations, including the mean value, fractional coverage, occurrence of a forecast event sufficiently near an observed event, and so on. By varying the sizes of the neighborhoods, it is possible to determine the scales for which the forecast has sufficient skill for a particular application. Several neighborhood verification methods have been proposed in the literature in the last decade. This paper examines four such methods in detail for idealized and real high-resolution precipitation forecasts, highlighting what can be learned from each of the methods. When applied to idealized and real precipitation forecasts from the Spatial Verification Methods Intercomparison Project, all four methods showed improved forecast performance for neighborhood sizes larger than grid scale, with the optimal scale for each method varying as a function of rainfall intensity.

2020 ◽  
Author(s):  
Marion Mittermaier ◽  
Rachel North ◽  
Christine Pequignet ◽  
Jan Maksymczuk

<div> <p>HiVE is a CMEMS funded collaboration between the atmospheric Numerical Weather Prediction (NWP) verification and the ocean community within the Met Office, aimed at demonstrating the use of spatial verification methods originally developed for the evaluation of high-resolution NWP forecasts, to CMEMS ocean model forecast products. Spatial verification methods provide more scale appropriate ways to better assess forecast characteristics and accuracy of km-scale forecasts, where the detail looks realistic but may not be in the right place at the right time. As a result, it can be the case that coarser resolution forecasts verify better (e.g. lower root-mean-square-error) than the higher resolution forecast. In this instance the smoothness of the coarser resolution forecast is rewarded, though the higher-resolution forecast may be better. The project utilised open source code library known as Model Evaluation Tools (MET) developed at the US National Center for Atmospheric Research (NCAR).</p> </div><div> <p> </p> </div><div> <p>This project saw, for the first time, the application of spatial verification methods to sub-10 km resolution ocean model forecasts. The project consisted of two parts. Part 1 is described in the companion poster to this one. Part 2 describes the skill of CMEMS products for forecasting events or features of interest such as algal blooms.  </p> </div><div> <p> </p> </div><div> <p>The Method for Object-based Diagnostic Evaluation (MODE) and the time dimension version MODE Time Domain (MTD) were applied to daily mean chlorophyll forecasts for the European North West Shelf from the FOAM-ERSEM model on the AMM7 grid. The forecasts are produced from a “cold start”, i.e. no data assimilation of biological variables. Here the entire 2019 algal bloom season was analysed to understand: intensity and spatial (area) biases; location and timing errors. Forecasts were compared to the CMEMS daily cloud free (L4) multi-sensor chlorophyll-<em>a</em> product. </p> </div><div> <p> </p> </div><div> <p>It has been found that there are large differences between forecast and observed concentrations of chlorophyll. This has meant that a quantile mapping approach for removing the bias was necessary before analysing the spatial properties of the forecast. Despite this the model still produces areas of chlorophyll which are too large compared to the observed. The model often produces areas of enhanced chlorophyll in approximately the right locations but the forecast and observed areas are rarely collocated and/or overlapping. Finally, the temporal analysis shows that the model struggled to get the onset of the season (being close to a month too late), but once the model picked up the signal there was better correspondence between the observed and forecast chlorophyll peaks for the remainder of the season. There was very little variation in forecast performance with lead time, suggesting that chlorophyll is a very slowly varying quantity.  </p> </div><div> <p> </p> </div><div> <p>Comparing an analysis which included the assimilation of observed chlorophyll shows that it is much closer to the observed L4 product than the non-biological assimilative analysis. It must be concluded that if the forecast were started from a DA analysis that included chlorophyll, it would lead to forecasts with less bias, and possibly a better detection of the onset of the bloom.  </p> </div><div> <p> </p> </div>


2009 ◽  
Vol 24 (5) ◽  
pp. 1390-1400 ◽  
Author(s):  
Jason E. Nachamkin

Abstract The composite method is applied to verify a series of idealized and real precipitation forecasts as part of the Spatial Forecast Verification Methods Intercomparison Project. The test cases range from simple geometric shapes to high-resolution (∼4 km) numerical model precipitation output. The performance of the composite method is described as it is applied to each set of forecasts. In general, the method performed well because it was able to relay information concerning spatial displacement and areal coverage errors. Summary scores derived from the composite means and the individual events displayed relevant information in a condensed form. The composite method also showed an ability to discern performance attributes from high-resolution precipitation forecasts from several competing model configurations, though the results were somewhat limited by the lack of data. Overall, the composite method proved to be most sensitive in revealing systematic displacement errors, while it was less sensitive to systematic model biases.


2020 ◽  
Author(s):  
Jan Maksymczuk ◽  
Ric Crocker ◽  
Marion Mittermaier ◽  
Christine Pequignet

<div> <p>HiVE is a CMEMS funded collaboration between the atmospheric Numerical Weather Prediction (NWP) verification and the ocean community within the Met Office, aimed at demonstrating the use of spatial verification methods originally developed for the evaluation of high-resolution NWP forecasts, with CMEMS ocean model forecast products. Spatial verification methods provide more scale appropriate ways to better assess forecast characteristics and accuracy of km-scale forecasts, where the detail looks realistic but may not be in the right place at the right time. As a result, it can be the case that coarser resolution forecasts verify better (e.g. lower root-mean-square-error) than the higher resolution forecast. In this instance the smoothness of the coarser resolution forecast is rewarded, though the higher-resolution forecast may be better. The project utilised open source code library known as Model Evaluation Toolkit (MET) developed at the US National Center for Atmospheric Research. </p> </div><div> <p> </p> </div><div> <p>This project saw, for the first time, the application of spatial verification methods to sub-10 km resolution ocean model forecasts. The project consisted of two parts. Part 1 describes an assessment of the forecast skill for SST of CMEMS model configurations at observing locations using an approach called HiRA (High Resolution Assessment). Part 2 is described in the companion poster to this one.  </p> </div><div> <p> </p> </div><div> <p>HiRA is a single-observation-forecast-neighbourhood-type method which makes use of commonly used ensemble verification metrics such as the Brier Score (BS) and the Continuous Ranked Probability Score (CRPS). In this instance all model grid points within a predefined neighbourhood of the observing location are considered equi-probable outcomes (or pseudo-ensemble members) at the observing location. The technique allows for an inter-comparison of models with different grid resolutions as well as between deterministic and probabilistic forecasts in an equitable and consistent way. In this work it has been applied to the CMEMS products delivered from the AMM7 (~7km) and AMM15 (~1.5km) model configurations for the European North West Shelf that are provided by the Met Office. </p> </div><div> <p> </p> </div><div> <p>It has been found that when neighbourhoods of equivalent extent are compared for both configurations it is possible to show improved forecast skill for SST for the higher resolution AMM15 both on- and off-shelf, which has been difficult to demonstrate previously using traditional metrics. Forecast skill generally degrades with increasing lead time for both configurations, with the off-shelf results for the higher resolution model showing increasing benefits over the coarser configuration. </p> </div>


2009 ◽  
Vol 137 (12) ◽  
pp. 4382-4385 ◽  
Author(s):  
Renzo Richiardone ◽  
Massimiliano Manfrin

Abstract The lapse rates of high-resolution temperature profiles during nearly neutral, saturated conditions are compared with the saturated adiabatic lapse rate and with that proposed by Richiardone and Giusti. A good agreement between the latter and the mean value of the observed lapse rate is found, whereas the saturated adiabatic lapse rate differs significantly, confirming experimentally that it is not completely correct to assess the moist neutrality from a comparison with the saturated adiabatic lapse rate. The lapse-rate distribution supports the hypothesis that the lapse-rate statistics is a local collection of saturated adiabatic lapse rates in a background normal distribution centered around the neutrality.


2021 ◽  
Vol 4 ◽  
pp. 30-49
Author(s):  
A.Yu. Bundel ◽  
◽  
A.V. Muraviev ◽  
E.D. Olkhovaya ◽  
◽  
...  

State-of-the-art high-resolution NWP models simulate mesoscale systems with a high degree of detail, with large amplitudes and high gradients of fields of weather variables. Higher resolution leads to the spatial and temporal error growth and to a well-known double penalty problem. To solve this problem, the spatial verification methods have been developed over the last two decades, which ignore moderate errors (especially in the position), but can still evaluate the useful skill of a high-resolution model. The paper refers to the updated classification of spatial verification methods, briefly describes the main methods, and gives an overview of the international projects for intercomparison of the methods. Special attention is given to the application of the spatial approach to ensemble forecasting. Popular software packages are considered. The Russian translation is proposed for the relevant English terms. Keywords: high-resolution models, verification, double penalty, spatial methods, ensemble forecasting, object-based methods


2017 ◽  
Vol 32 (1) ◽  
pp. 187-198 ◽  
Author(s):  
Eric Gilleland

Abstract This paper proposes new diagnostic plots that take advantage of the lack of symmetry in the mean-error distance measure (MED) for binary images to yield a new concept of false alarms and misses appropriate to the spatial setting where the measure does not require perfect matching to be a hit or correct negative. Additionally, three previously proposed geometric indices that provide complementary information about forecast performance are used to produce useful diagnostic plots for forecast performance. The diagnostics are applied to previously analyzed case studies from the spatial forecast verification Intercomparison Project (ICP) to facilitate a comparison with more complicated methods. Relatively new test cases from the Mesoscale Verification Intercomparison over Complex Terrain (MesoVICT) project are also employed for future comparisons. It is found that the proposed techniques provide useful information about forecast model behavior by way of a succinct, easy-to-implement method that can be complementary to other measures of forecast performance.


2008 ◽  
Vol 139 (2_suppl) ◽  
pp. P117-P117
Author(s):  
Hiroshi Umeki ◽  
Kenji Takasaki ◽  
Kaori Enatsu ◽  
Fujinobu Tanaka ◽  
Hidetaka Kumagami ◽  
...  

Objectives This study aimed to investigate the effects of a tongue-holding maneuver (THM) during swallowing, using a novel high-resolution manometry (HRM) system. Methods 27 asymptomatic adult Japanese controls were studied. A solid-state HRM assembly with 36 circumferential sensors spaced 1 cm apart was positioned to record pressures during swallowing from the velopharynx to the upper esophagus. The maximum values of the dry swallowing pressures at meso-hypopharynx, the upper esophageal sphincter (UES) and the mean values at meso-hypopharynx with and without THM were measured. Results The maximum values of dry swallowing pressures with and without THM were 195.0±77.2 (mmHg, mean ± standard deviation), and 178.1±53.0 at the meso-hypopharynx, and 213.4±74.0 and 190.0±95.0 at the UES, respectively. The mean values with and without THM at meso-hypopharynx were 47.4±11.9, and 44.0±11.2, respectively. The maximum value at UES (p=0.0347), and the mean value at the meso-hypopharynx (p=0.0124) of dry swallowing pressures with THM were significantly higher than those without THM. Conclusions These findings indicate that the THM has a potential to facilitate compensatory swallowing power at the pharynx and upper esophagus. HRM will provide us with important information about the swallowing physiology, and pathophysiology along the velopharynx, and upper esophagus.


2005 ◽  
Vol 277-279 ◽  
pp. 851-856
Author(s):  
Hyun Jin Jeong ◽  
Jae Woo Lee ◽  
Sug Whan Kim ◽  
A Ram Kang

We present high-resolution chemical abundance results of 15 metal-poor dwarf stars in the solar neighborhood. Our metallicity measurements are in a good agreement with previous estimates. The stars under investigation have metallicities ranging from -1.6 to -2.8 with the mean value of [Fe/H] = -2.28. The mean values of elemental abundances are [O/Fe] = 0.73, [Na/Fe] = -0.16, [Ca/Fe] = 0.28, and [Ti/Fe] = 0.60. Our results are found to be consistent with those of giant metal-poor stars.


2017 ◽  
Vol 32 (2) ◽  
pp. 733-741 ◽  
Author(s):  
Craig S. Schwartz

Abstract As high-resolution numerical weather prediction models are now commonplace, “neighborhood” verification metrics are regularly employed to evaluate forecast quality. These neighborhood approaches relax the requirement that perfect forecasts must match observations at the grid scale, contrasting traditional point-by-point verification methods. One recently proposed metric, the neighborhood equitable threat score, is calculated from 2 × 2 contingency tables that are populated within a neighborhood framework. However, the literature suggests three subtly different methods of populating neighborhood-based contingency tables. Thus, this work compares and contrasts these three variants and shows they yield statistically significantly different conclusions regarding forecast performance, illustrating that neighborhood-based contingency tables should be constructed carefully and transparently. Furthermore, this paper shows how two of the methods use inconsistent event definitions and suggests a “neighborhood maximum” approach be used to fill neighborhood-based contingency tables.


SPE Journal ◽  
2014 ◽  
Vol 20 (02) ◽  
pp. 267-276 ◽  
Author(s):  
Xianhui Kong ◽  
Mojdeh Delshad ◽  
Mary F. Wheeler

Summary Numerical modeling and simulation are essential tools for developing a better understanding of the geologic characteristics of aquifers and providing technical support for future carbon dioxide (CO2) storage projects. Modeling CO2 sequestration in underground aquifers requires the implementation of models of multiphase flow and CO2 and brine phase behavior. Capillary pressure and relative permeability need to be consistent with permeability/porosity variations of the rock. It is, therefore, crucial to gain confidence in the numerical models by validating the models and results by use of laboratory and field pilot results. A published CO2/brine laboratory coreflood was selected for our simulation study. The experimental results include subcore porosity and CO2-saturation distributions by means of a computed tomography (CT) scanner along with a CO2-saturation histogram. Data used in this paper are all based on those provided by Krause et al. (2011), with the exception of the CT porosity data. We generated a heterogeneous distribution for the porosity but honoring the mean value provided by Krause et al. (2011). We also generated the permeability distribution with the mean value for the whole core given by Krause et al. (2011). All the other data, such as the core dimensions, injection rate, outlet pressure, temperature, relative permeability, and capillary pressure, are the same as those in Krause et al. (2011). High-resolution coreflood simulations of brine displacement with supercritical CO2 are presented with the compositional reservoir simulator IPARS (Wheeler and Wheeler 1990). A 3D synthetic core model was constructed with permeability and porosity distributions generated by use of the geostatistical software FFTSIM [Jennings et al. (2000)], with cell sizes of 1.27×1.27×6.35 mm. The core was initially saturated with brine. Fluid properties were calibrated with the equation-of-state (EOS) compositional model to match the measured data provided by Krause et al. (2011). We used their measured capillary pressure and relative permeability curves. However, we scaled capillary pressure on the basis of the Leverett J-function (Leverett 1941) for permeability, porosity, and interfacial tension (IFT) in every simulation grid cell. Saturation images provide insight into the role of heterogeneity of CO2 distribution in which a slight variation in porosity gives rise to large variations in CO2-saturation distribution in the core. High-resolution numerical results indicated that accurate representation of capillary pressure at small scales was critical. Residual brine saturation and the subsequent shift in the relative permeability curves showed a significant impact on final CO2 distribution in the core.


Sign in / Sign up

Export Citation Format

Share Document