High-resolution model Verification Evaluation (HiVE). Part 1: Using neighbourhood techniques for the assessment of ocean model forecast skill

2020 ◽  
Author(s):  
Jan Maksymczuk ◽  
Ric Crocker ◽  
Marion Mittermaier ◽  
Christine Pequignet
Keyword(s):  
High Resolution ◽  
Weather Prediction ◽  
Ocean Model ◽  
Forecast Skill ◽  
Model Verification ◽  
Model Forecast ◽  
Verification Methods ◽  
Resolution Model ◽  
The Right ◽  
Spatial Verification

<div> <p>HiVE is a CMEMS funded collaboration between the atmospheric Numerical Weather Prediction (NWP) verification and the ocean community within the Met Office, aimed at demonstrating the use of spatial verification methods originally developed for the evaluation of high-resolution NWP forecasts, with CMEMS ocean model forecast products. Spatial verification methods provide more scale appropriate ways to better assess forecast characteristics and accuracy of km-scale forecasts, where the detail looks realistic but may not be in the right place at the right time. As a result, it can be the case that coarser resolution forecasts verify better (e.g. lower root-mean-square-error) than the higher resolution forecast. In this instance the smoothness of the coarser resolution forecast is rewarded, though the higher-resolution forecast may be better. The project utilised open source code library known as Model Evaluation Toolkit (MET) developed at the US National Center for Atmospheric Research. </p> </div><div> <p> </p> </div><div> <p>This project saw, for the first time, the application of spatial verification methods to sub-10 km resolution ocean model forecasts. The project consisted of two parts. Part 1 describes an assessment of the forecast skill for SST of CMEMS model configurations at observing locations using an approach called HiRA (High Resolution Assessment). Part 2 is described in the companion poster to this one.  </p> </div><div> <p> </p> </div><div> <p>HiRA is a single-observation-forecast-neighbourhood-type method which makes use of commonly used ensemble verification metrics such as the Brier Score (BS) and the Continuous Ranked Probability Score (CRPS). In this instance all model grid points within a predefined neighbourhood of the observing location are considered equi-probable outcomes (or pseudo-ensemble members) at the observing location. The technique allows for an inter-comparison of models with different grid resolutions as well as between deterministic and probabilistic forecasts in an equitable and consistent way. In this work it has been applied to the CMEMS products delivered from the AMM7 (~7km) and AMM15 (~1.5km) model configurations for the European North West Shelf that are provided by the Met Office. </p> </div><div> <p> </p> </div><div> <p>It has been found that when neighbourhoods of equivalent extent are compared for both configurations it is possible to show improved forecast skill for SST for the higher resolution AMM15 both on- and off-shelf, which has been difficult to demonstrate previously using traditional metrics. Forecast skill generally degrades with increasing lead time for both configurations, with the off-shelf results for the higher resolution model showing increasing benefits over the coarser configuration. </p> </div>

High-resolution model Verification Evaluation (HiVE). Part 2: Using object-based methods for the evaluation of algal blooms

2020 ◽  
Author(s):  
Marion Mittermaier ◽  
Rachel North ◽  
Christine Pequignet ◽  
Jan Maksymczuk
Keyword(s):  
High Resolution ◽  
Algal Blooms ◽  
Weather Prediction ◽  
Ocean Model ◽  
Model Verification ◽  
Forecast Performance ◽  
Object Based ◽  
Verification Methods ◽  
The Right ◽  
Spatial Verification

<div> <p>HiVE is a CMEMS funded collaboration between the atmospheric Numerical Weather Prediction (NWP) verification and the ocean community within the Met Office, aimed at demonstrating the use of spatial verification methods originally developed for the evaluation of high-resolution NWP forecasts, to CMEMS ocean model forecast products. Spatial verification methods provide more scale appropriate ways to better assess forecast characteristics and accuracy of km-scale forecasts, where the detail looks realistic but may not be in the right place at the right time. As a result, it can be the case that coarser resolution forecasts verify better (e.g. lower root-mean-square-error) than the higher resolution forecast. In this instance the smoothness of the coarser resolution forecast is rewarded, though the higher-resolution forecast may be better. The project utilised open source code library known as Model Evaluation Tools (MET) developed at the US National Center for Atmospheric Research (NCAR).</p> </div><div> <p> </p> </div><div> <p>This project saw, for the first time, the application of spatial verification methods to sub-10 km resolution ocean model forecasts. The project consisted of two parts. Part 1 is described in the companion poster to this one. Part 2 describes the skill of CMEMS products for forecasting events or features of interest such as algal blooms.  </p> </div><div> <p> </p> </div><div> <p>The Method for Object-based Diagnostic Evaluation (MODE) and the time dimension version MODE Time Domain (MTD) were applied to daily mean chlorophyll forecasts for the European North West Shelf from the FOAM-ERSEM model on the AMM7 grid. The forecasts are produced from a “cold start”, i.e. no data assimilation of biological variables. Here the entire 2019 algal bloom season was analysed to understand: intensity and spatial (area) biases; location and timing errors. Forecasts were compared to the CMEMS daily cloud free (L4) multi-sensor chlorophyll-<em>a</em> product. </p> </div><div> <p> </p> </div><div> <p>It has been found that there are large differences between forecast and observed concentrations of chlorophyll. This has meant that a quantile mapping approach for removing the bias was necessary before analysing the spatial properties of the forecast. Despite this the model still produces areas of chlorophyll which are too large compared to the observed. The model often produces areas of enhanced chlorophyll in approximately the right locations but the forecast and observed areas are rarely collocated and/or overlapping. Finally, the temporal analysis shows that the model struggled to get the onset of the season (being close to a month too late), but once the model picked up the signal there was better correspondence between the observed and forecast chlorophyll peaks for the remainder of the season. There was very little variation in forecast performance with lead time, suggesting that chlorophyll is a very slowly varying quantity.  </p> </div><div> <p> </p> </div><div> <p>Comparing an analysis which included the assimilation of observed chlorophyll shows that it is much closer to the observed L4 product than the non-biological assimilative analysis. It must be concluded that if the forecast were started from a DA analysis that included chlorophyll, it would lead to forecasts with less bias, and possibly a better detection of the onset of the bloom.  </p> </div><div> <p> </p> </div>

scholarly journals Overview of spatial verification methods and their application to ensemble forecasting

2021 ◽  
Vol 4 ◽  
pp. 30-49
Author(s):  
A.Yu. Bundel ◽  
◽  
A.V. Muraviev ◽  
E.D. Olkhovaya ◽  
◽  
...  
Keyword(s):  
High Resolution ◽  
Ensemble Forecasting ◽  
Object Based ◽  
Verification Methods ◽  
International Projects ◽  
Resolution Model ◽  
High Gradients ◽  
High Resolution Models ◽  
High Degree ◽  
Spatial Verification

State-of-the-art high-resolution NWP models simulate mesoscale systems with a high degree of detail, with large amplitudes and high gradients of fields of weather variables. Higher resolution leads to the spatial and temporal error growth and to a well-known double penalty problem. To solve this problem, the spatial verification methods have been developed over the last two decades, which ignore moderate errors (especially in the position), but can still evaluate the useful skill of a high-resolution model. The paper refers to the updated classification of spatial verification methods, briefly describes the main methods, and gives an overview of the international projects for intercomparison of the methods. Special attention is given to the application of the spatial approach to ensemble forecasting. Popular software packages are considered. The Russian translation is proposed for the relevant English terms. Keywords: high-resolution models, verification, double penalty, spatial methods, ensemble forecasting, object-based methods

scholarly journals Central Mediterranean Sea forecast: effects of high-resolution atmospheric forcings

2006 ◽  
Vol 3 (3) ◽  
pp. 637-669 ◽  
Author(s):  
S. Natale ◽  
R. Sorgente ◽  
S. Gaberšek ◽  
A. Ribotti ◽  
A. Olita
Keyword(s):  
High Resolution ◽  
Ocean Circulation ◽  
Regional Scale ◽  
Circulation Model ◽  
Surface Current ◽  
Ocean Model ◽  
Central Mediterranean ◽  
Resolution Model ◽  
Atmospheric Forcings ◽  
Very High

Abstract. Ocean forecasts over the Central Mediterranean, produced by a near real time regional scale system, have been evaluated in order to assess their predictability. The ocean circulation model has been forced at the surface by a medium, high or very high resolution atmospheric forcing. The simulated ocean parameters have been compared with satellite data and they were found to be generally in good agreement. High and very high resolution atmospheric forcings have been able to form noticeable, although short-lived, surface current structures, due to their ability to detect transient atmospheric disturbances. The existence of the current structures has not been directly assessed due to lack of measurements. The ocean model in the slave mode was not able to develop dynamics different from the driving coarse resolution model which provides the boundary conditions.

scholarly journals Improving Numerical Weather Predictions of Summertime Precipitation over the Southeastern United States through a High-Resolution Initialization of the Surface State

2011 ◽  
Vol 26 (6) ◽  
pp. 785-807 ◽  
Author(s):  
Jonathan L. Case ◽  
Sujay V. Kumar ◽  
Jayanthi Srikishen ◽  
Gary J. Jedlovec
Keyword(s):  
United States ◽  
Soil Moisture ◽  
High Resolution ◽  
Land Surface ◽  
Weather Prediction ◽  
Southeastern United States ◽  
Model Verification ◽  
Sea Surface ◽  
Convective Precipitation ◽  
Numerical Weather

Abstract It is hypothesized that high-resolution, accurate representations of surface properties such as soil moisture and sea surface temperature are necessary to improve simulations of summertime pulse-type convective precipitation in high-resolution models. This paper presents model verification results of a case study period from June to August 2008 over the southeastern United States using the Weather Research and Forecasting numerical weather prediction model. Experimental simulations initialized with high-resolution land surface fields from the National Aeronautics and Space Administration’s (NASA) Land Information System (LIS) and sea surface temperatures (SSTs) derived from the Moderate Resolution Imaging Spectroradiometer (MODIS) are compared to a set of control simulations initialized with interpolated fields from the National Centers for Environmental Prediction’s (NCEP) 12-km North American Mesoscale model. The LIS land surface and MODIS SSTs provide a more detailed surface initialization at a resolution comparable to the 4-km model grid spacing. Soil moisture from the LIS spinup run is shown to respond better to the extreme rainfall of Tropical Storm Fay in August 2008 over the Florida peninsula. The LIS has slightly lower errors and higher anomaly correlations in the top soil layer but exhibits a stronger dry bias in the root zone. The model sensitivity to the alternative surface initial conditions is examined for a sample case, showing that the LIS–MODIS data substantially impact surface and boundary layer properties. The Developmental Testbed Center’s Meteorological Evaluation Tools package is employed to produce verification statistics, including traditional gridded precipitation verification and output statistics from the Method for Object-Based Diagnostic Evaluation (MODE) tool. The LIS–MODIS initialization is found to produce small improvements in the skill scores of 1-h accumulated precipitation during the forecast hours of the peak diurnal convective cycle. Because there is very little union in time and space between the forecast and observed precipitation systems, results from the MODE object verification are examined to relax the stringency of traditional gridpoint precipitation verification. The MODE results indicate that the LIS–MODIS-initialized model runs increase the 10 mm h−1 matched object areas (“hits”) while simultaneously decreasing the unmatched object areas (“misses” plus “false alarms”) during most of the peak convective forecast hours, with statistically significant improvements of up to 5%. Simulated 1-h precipitation objects in the LIS–MODIS runs more closely resemble the observed objects, particularly at higher accumulation thresholds. Despite the small improvements, however, the overall low verification scores indicate that much uncertainty still exists in simulating the processes responsible for airmass-type convective precipitation systems in convection-allowing models.

scholarly journals Neighborhood Verification: A Strategy for Rewarding Close Forecasts

2009 ◽  
Vol 24 (6) ◽  
pp. 1498-1510 ◽  
Author(s):  
Elizabeth E. Ebert
Keyword(s):  
High Resolution ◽  
Rainfall Intensity ◽  
Mean Value ◽  
Forecast Performance ◽  
Fractional Coverage ◽  
Optimal Scale ◽  
Verification Methods ◽  
Intercomparison Project ◽  
The Mean ◽  
Spatial Verification

Abstract High-resolution forecasts may be quite useful even when they do not match the observations exactly. Neighborhood verification is a strategy for evaluating the “closeness” of the forecast to the observations within space–time neighborhoods rather than at the grid scale. Various properties of the forecast within a neighborhood can be assessed for similarity to the observations, including the mean value, fractional coverage, occurrence of a forecast event sufficiently near an observed event, and so on. By varying the sizes of the neighborhoods, it is possible to determine the scales for which the forecast has sufficient skill for a particular application. Several neighborhood verification methods have been proposed in the literature in the last decade. This paper examines four such methods in detail for idealized and real high-resolution precipitation forecasts, highlighting what can be learned from each of the methods. When applied to idealized and real precipitation forecasts from the Spatial Verification Methods Intercomparison Project, all four methods showed improved forecast performance for neighborhood sizes larger than grid scale, with the optimal scale for each method varying as a function of rainfall intensity.

Impact of Coupling with an Ice–Ocean Model on Global Medium-Range NWP Forecast Skill

2018 ◽  
Vol 146 (4) ◽  
pp. 1157-1180 ◽  
Author(s):  
Gregory C. Smith ◽  
Jean-Marc Bélanger ◽  
François Roy ◽  
Pierre Pellerin ◽  
Hal Ritchie ◽  
...  
Keyword(s):  
Standard Deviation ◽  
Large Scale ◽  
Weather Forecasting ◽  
Positive Impact ◽  
Weather Prediction ◽  
Ocean Model ◽  
Forecast Skill ◽  
Medium Range ◽  
Forecasting System ◽  
The Impact

The importance of coupling between the atmosphere and the ocean for forecasting on time scales of hours to weeks has been demonstrated for a range of physical processes. Here, the authors evaluate the impact of an interactive air–sea coupling between an operational global deterministic medium-range weather forecasting system and an ice–ocean forecasting system. This system was developed in the context of an experimental forecasting system that is now running operationally at the Canadian Centre for Meteorological and Environmental Prediction. The authors show that the most significant impact is found to be associated with a decreased cyclone intensification, with a reduction in the tropical cyclone false alarm ratio. This results in a 15% decrease in standard deviation errors in geopotential height fields for 120-h forecasts in areas of active cyclone development, with commensurate benefits for wind, temperature, and humidity fields. Whereas impacts on surface fields are found locally in the vicinity of cyclone activity, large-scale improvements in the mid-to-upper troposphere are found with positive global implications for forecast skill. Moreover, coupling is found to produce fairly constant reductions in standard deviation error growth for forecast days 1–7 of about 5% over the northern extratropics in July and August and 15% over the tropics in January and February. To the authors’ knowledge, this is the first time a statistically significant positive impact of coupling has been shown in an operational global medium-range deterministic numerical weather prediction framework.

scholarly journals High-resolution nested model for the Lebanese coastal area, Eastern Mediterranean: implementation and climatological runs

2006 ◽  
Vol 3 (3) ◽  
pp. 373-396 ◽  
Author(s):  
N. Kabbara ◽  
R. Sorgente ◽  
S. Natale ◽  
D. R. Hayes ◽  
G. Zodiatis
Keyword(s):  
High Resolution ◽  
Eastern Mediterranean ◽  
Ocean Model ◽  
Levantine Basin ◽  
Resolution Model ◽  
Coarse Model ◽  
Shelf Region ◽  
Horizontal Grid ◽  
Lebanese Coast ◽  
Circulation Characteristics

Abstract. As a part of the project Mediterranean Network to Assess and Upgrade Monitoring and Forecasting Activity in the Region (MAMA) we implemented a high resolution nested hydrodynamic model (1/40° horizontal grid, 16 sigma levels) for the coastal, shelf and open sea areas off the Lebanese coast, East Levantine Basin of the Eastern Mediterranean Sea. The Lebanese Shelf Model (LSM) is a version of the Princeton Ocean Model (POM). It is nested in a coarse resolution model the Aegean Levantine Eddy Resolving Model (1/20° horizontal grid, 25 sigma levels), ALERMO, that covers the Eastern Mediterranean. The nesting is one way so that velocity, temperature, and salinity along the open boundaries are interpolated from the relevant coarse model variables. Numerical simulations have been carried out under climatological surface and lateral forcing. Due to the relatively small domain, the results closely follow the simulation of the intermediate model with more details especially over the narrow shelf region. Simulations reproduce main circulation features and coastal circulation characteristics over the eastern Levantine shelf. This paper describes the modeling system setup, compares the simulations with the corresponding results of the coarse model ALERMO, and with the observed climatological circulation characteristics in the Levantine Basin off the Lebanese coast.

The Sydney Australia Wildfires of January 1994 - Meteorological Conditions and High Resolution Numerical Modeling Experiments

1996 ◽  
Vol 6 (3) ◽  
pp. 145 ◽  
Author(s):  
MS Speer ◽  
LM Leslie ◽  
JR Colquhoun ◽  
E Mitchell
Keyword(s):  
High Resolution ◽  
Weather Prediction ◽  
Weather Conditions ◽  
Detailed Examination ◽  
Meteorological Conditions ◽  
Limited Area ◽  
Research Activity ◽  
Southeastern Australia ◽  
South Wales ◽  
Resolution Model

Southeastern Australia is particularly vulnerable to wildfires during the spring and summer months, and the threat of devastation is present most years. In January 1994, the most populous city in Australia, Sydney, was ringed by wildfires, some of which penetrated well into suburban areas and there were many other serious fires in coastal areas of New South Wales (NSW). In recent years much research activity in Australia has focussed on the development of high resolution limited area models, for eventual operational prediction of meteorological conditions associated with high levels of wildfire risk. In this study, the period January 7-8, 1994 was chosen for detailed examination, as it was the most critical period during late December 1993/early January 1994 for the greater Sydney area. Routine forecast guidance from the Australian Bureau of Meteorology's operational numerical weather prediction (NWP) models was very useful in that both the medium and short range models predicted synoptic patterns suggesting extreme fire weather conditions up to several days in advance. However, vital information of a detailed nature was lacking. A new high resolution model was run at the operational resolution of 150 km and the much higher resolutions of 25 km and 5 km. The new model showed statistically significant greater skill in predicting details of wind, relative humidity and temperature patterns both near the surface and above the boundary layer. It also produced skilful predictions of the Forest Fire Danger Index.

Evaluating Convective Initiation in High-Resolution Numerical Weather Prediction Models Using GOES-16 Infrared Brightness Temperatures

2021 ◽  
Vol 149 (4) ◽  
pp. 1153-1172
Author(s):  
David S. Henderson ◽  
Jason A. Otkin ◽  
John R. Mecikalski
Keyword(s):  
High Resolution ◽  
Prediction Models ◽  
Weather Prediction ◽  
Cloud Microphysics ◽  
Cloud Water ◽  
Convective Initiation ◽  
Model Simulations ◽  
Brightness Temperatures ◽  
Cloud Characteristics ◽  
Verification Methods

AbstractThe evolution of model-based cloud-top brightness temperatures (BT) associated with convective initiation (CI) is assessed for three bulk cloud microphysics schemes in the Weather Research and Forecasting Model. Using a composite-based analysis, cloud objects derived from high-resolution (500 m) model simulations are compared to 5-min GOES-16 imagery for a case study day located near the Alabama–Mississippi border. Observed and simulated cloud characteristics for clouds reaching CI are examined by utilizing infrared BTs commonly used in satellite-based CI nowcasting methods. The results demonstrate the ability of object-based verification methods with satellite observations to evaluate the evolution of model cloud characteristics, and the BT comparison provides insight into a known issue of model simulations producing too many convective cells reaching CI. The timing of CI from the different microphysical schemes is dependent on the production of ice in the upper levels of the cloud, which typically occurs near the time of maximum cloud growth. In particular, large differences in precipitation formation drive differences in the amount of cloud water able to reach upper layers of the cloud, which impacts cloud-top glaciation. Larger cloud mixing ratios are found in clouds with sustained growth leading to more cloud water lofted to the upper levels of the cloud and the formation of ice. Clouds unable to sustain growth lack the necessary cloud water needed to form ice and grow into cumulonimbus. Clouds with slower growth rates display similar BT trends as clouds exhibiting growth, which suggests that forecasting CI using geostationary satellites might require additional information beyond those derived at cloud top.

Sign in / Sign up

Export Citation Format

Share Document