scholarly journals An Object-Oriented Multiscale Verification Scheme

2010 ◽  
Vol 25 (1) ◽  
pp. 79-92 ◽  
Author(s):  
Steven A. Lack ◽  
George L. Limpert ◽  
Neil I. Fox

Abstract Object-oriented verification methodology is becoming more and more common in the evaluation of model performance on high-resolution grids. The research herein describes an advanced version of an object-oriented approach that involves a combination of object identification on multiple scales with Procrustes shape analysis techniques. The multiscale object identification technique relies heavily on a novel Fourier transform approach to associate the signals within convection to different spatial scales. Other features of this new verification scheme include using a weighted cost function that can be user defined for object matching using different criteria, delineating objects that are more linear in character from those that are more cellular, and tagging object matches as hits, misses, or false alarms. Although the scheme contains a multiscale approach for identifying convective objects, standard minimum intensity and minimum size thresholds can be set when desirable. The method was tested as part of a spatial verification intercomparison experiment utilizing a combination of synthetic data and real cases from the Storm Prediction Center (SPC)/NSSL Weather Research and Forecasting (WRF) model Spring Program 2005. The resulting metrics, including error measures from differences in matched objects due to displacement, dilation, rotation, and intensity, from these cases run through this new, robust verification scheme are shown.

2008 ◽  
Vol 136 (3) ◽  
pp. 1013-1025 ◽  
Author(s):  
Caren Marzban ◽  
Scott Sandgathe

Abstract In a recent paper, a statistical method referred to as cluster analysis was employed to identify clusters in forecast and observed fields. Further criteria were also proposed for matching the identified clusters in one field with those in the other. As such, the proposed methodology was designed to perform an automated form of what has been called object-oriented verification. Herein, a variation of that methodology is proposed that effectively avoids (or simplifies) the criteria for matching the objects. The basic idea is to perform cluster analysis on the combined set of observations and forecasts, rather than on the individual fields separately. This method will be referred to as combinative cluster analysis (CCA). CCA naturally lends itself to the computation of false alarms, hits, and misses, and therefore, to the critical success index (CSI). A desirable feature of the previous method—the ability to assess performance on different spatial scales—is maintained. The method is demonstrated on reflectivity data and corresponding forecasts for three dates using three mesoscale numerical weather prediction model formulations—the NCEP/NWS Nonhydrostatic Mesoscale Model (NMM) at 4-km resolution (nmm4), the University of Oklahoma’s Center for Analysis and Prediction of Storms (CAPS) Weather Research and Forecasting Model (WRF) at 2-km resolution (arw2), and the NCAR WRF at 4-km resolution (arw4). In the small demonstration sample herein, model forecast quality is efficiently differentiated when performance is assessed in terms of the CSI. In this sample, arw2 appears to outperform the other two model formulations across all scales when the cluster analysis is performed in the space of spatial coordinates and reflectivity. However, when the analysis is performed only on spatial data (i.e., when only the spatial placement of the reflectivity is assessed), the difference is not significant. This result has been verified both visually and using a standard gridpoint verification, and seems to provide a reasonable assessment of model performance. This demonstration of CCA indicates promise in quickly evaluating mesoscale model performance while avoiding the subjectivity and labor intensiveness of human evaluation or the pitfalls of non-object-oriented automated verification.


2021 ◽  
Author(s):  
Antonio Serrano ◽  
Guadalupe Sánchez-Hernández ◽  
Julio A. H. Escobar ◽  
María Luisa Cancillo

<p>Solar energy proves to be an interesting alternative to conventional sources based on the burning of fossil fuels. However, it shows a high short-term variability that makes its integration into the electricity mix difficult. To facilitate this integration, reliable short- and medium-term forecasts become highly necessary. To respond to this demand, solar radiation forecasting models have emerged. Among them, Weather Research and Forecasting (WRF) has become particularly promising and has shown good performance at different temporal and spatial scales. The performance of these models is usually assessed by comparing their estimates with point measurements at selected stations. This comparison is hampered by the difference in spatial dimensions between the model estimates (representative of a given area) and the station (point) measurements. This difference introduces a certain error in the forecast, mainly related to the short-scale variability of cloudiness. Despite being essential to understand model validation, this issue has not been sufficiently investigated. In this framework, the present study analyzes the effect of the spatial representativeness of point measurements when used to validate model estimates. For this purpose, a specific one-month measurement campaign was conducted, deploying seven pyranometers in the vicinity of the city of Badajoz, Spain. To ensure their intercomparability, all pyranometers were calibrated with respect to a reference pyranometer previously calibrated by the World Radiation Center in Davos, Switzerland. Solar radiation was measured at a 1-minute basis to record the short-term variability due to cloudiness. Two series were constructed with these data, one corresponding to a selected station and the other to the average of the seven stations. These series of measurements were compared with the estimates provided by the WRF model for the same period and location. A configuration with two nested domains of 27 km and 9 km was used. Model performance showed better agreement when averaging was used instead of individual measurements, with RMSE improving from 89 W/m² to 77 W/m². Cloudy cases contributed the most to the differences between station measurements and model estimates, showing an RMSE greater than 100 W/m2, more than three times higher than the RMSE for clear cases (about 33 W/m2). The difference between the stations and the model for cloudy cases is reduced from 125 W/m2 to 107 W/m2 when averaged measurements are considered instead of single station measurements. This study contributes to the understanding of the representativeness of point station measurements for validation and comparison with WRF estimates. Acknowledgments. This work is partially funded by FEDER/Ministerio de Ciencia, Innovación y Universidades-Agencia Estatal de Investigación of Spain through project RTI 2018-097332-B-C22, and by Junta de Extremadura-FEDER through project GR18097.</p>


2008 ◽  
Vol 65 (5) ◽  
pp. 742-745 ◽  
Author(s):  
Deborah A. Reusser ◽  
Henry Lee

Abstract Reusser, D. A., and Lee II, H. 2008. Predictions for an invaded world: a strategy to predict the distribution of native and non-indigenous species at multiple scales. – ICES Journal of Marine Science, 65: 742–745. Habitat models can be used to predict the distributions of marine and estuarine non-indigenous species (NIS) over several spatial scales. At an estuary scale, our goal is to predict the estuaries most likely to be invaded, but at a habitat scale, the goal is to predict the specific locations within an estuary that are most vulnerable to invasion. As an initial step in evaluating several habitat models, model performance for a suite of benthic species with reasonably well-known distributions on the Pacific coast of the US needs to be compared. We discuss the utility of non-parametric multiplicative regression (NPMR) for predicting habitat- and estuary-scale distributions of native and NIS. NPMR incorporates interactions among variables, allows qualitative and categorical variables, and utilizes data on absence as well as presence. Preliminary results indicate that NPMR generally performs well at both spatial scales and that distributions of NIS are predicted as well as those of native species. For most species, latitude was the single best predictor, although similar model performance could be obtained at both spatial scales with combinations of other habitat variables. Errors of commission were more frequent at a habitat scale, with omission and commission errors approximately equal at an estuary scale.


2021 ◽  
Vol 13 (12) ◽  
pp. 2355
Author(s):  
Linglin Zeng ◽  
Yuchao Hu ◽  
Rui Wang ◽  
Xiang Zhang ◽  
Guozhang Peng ◽  
...  

Air temperature (Ta) is a required input in a wide range of applications, e.g., agriculture. Land Surface Temperature (LST) products from Moderate Resolution Imaging Spectroradiometer (MODIS) are widely used to estimate Ta. Previous studies of these products in Ta estimation, however, were generally applied in small areas and with a small number of meteorological stations. This study designed both temporal and spatial experiments to estimate 8-day and daily maximum and minimum Ta (Tmax and Tmin) on three spatial scales: climate zone, continental and global scales from 2009 to 2018, using the Random Forest (RF) method based on MODIS LST products and other auxiliary data. Factors contributing to the relation between LST and Ta were determined based on physical models and equations. Temporal and spatial experiments were defined by the rules of dividing the training and validation datasets for the RF method, in which the stations selected in the training dataset were all included or not in the validation dataset. The RF model was first trained and validated on each spatial scale, respectively. On a global scale, model accuracy with a determination coefficient (R2) > 0.96 and root mean square error (RMSE) < 1.96 °C and R2 > 0.95 and RMSE < 2.55 °C was achieved for 8-day and daily Ta estimations, respectively, in both temporal and spatial experiments. Then the model was trained and cross-validated on each spatial scale. The results showed that the data size and station distribution of the study area were the main factors influencing the model performance at different spatial scales. Finally, the spatial patterns of the model performance and variable importance were analyzed. Both daytime and nighttime LST had a significant contribution in the 8-day Tmax estimation on all the three spatial scales; while their contribution in daily Tmax estimation varied over different continents or climate zones. This study was expected to improve our understanding of Ta estimation in terms of accuracy variations and influencing variables on different spatial and temporal scales. The future work mainly includes identifying underlying mechanisms of estimation errors and the uncertainty sources of Ta estimation from a local to a global scale.


1998 ◽  
Vol 55 (S1) ◽  
pp. 9-21 ◽  
Author(s):  
Carol L Folt ◽  
Keith H Nislow ◽  
Mary E Power

The Atlantic salmon (Salmo salar) is a model species for studying scale issues (i.e., the extent, duration, and resolution of a study or natural process) in ecology. Major shifts in behavior and habitat use over ontogeny, along with a relatively long life span and large dispersal and migration distances, make scale issues critical for effective conservation, management, and restoration of this species. The scale over which a process occurs must be linked to the research design and we illustrate this with a discussion of resource tracking by Atlantic salmon. Identifying scale inconsistencies (e.g., when a process is evident at one scale but not another) is shown to be an effective means by which some scale-dependent processes are understood. We review the literature to assess the temporal and spatial scales used in Atlantic salmon research and find most current studies appear to sacrifice spatial and temporal extent for increased resolution. Finally, we discuss research strategies for expanding the temporal and spatial scales in salmon research, such as conducting multiple scales studies to elucidate scale inconsistencies, identifying mechanisms, and using techniques and approaches to generalize across studies and over time and space.


2018 ◽  
Vol 75 (11) ◽  
pp. 1902-1914 ◽  
Author(s):  
Lu Guan ◽  
John F. Dower ◽  
Pierre Pepin

Spatial structures of larval fish in the Strait of Georgia (British Columbia, Canada) were quantified in the springs of 2009 and 2010 to investigate linkages to environmental heterogeneity at multiple scales. By applying a multiscale approach, principal coordinate neighborhood matrices, spatial variability was decomposed into three predefined scale categories: broad scale (>40 km), medium scale (20∼40 km), and fine scale (<20 km). Spatial variations in larval density of the three dominant fish taxa with different early life histories (Pacific herring (Clupea pallasii), Pacific hake (Merluccius productus), and northern smoothtongue (Leuroglossus schmidti)) were mainly structured at broad and medium scales, with scale-dependent associations with environmental descriptors varying interannually and among species. Larval distributions in the central-southern Strait were mainly associated with salinity, temperature, and vertical stability of the top 50 m of the water column on the medium scale. Our results emphasize the critical role of local estuarine circulation, especially at medium spatial scale, in structuring hierarchical spatial distributions of fish larvae in the Strait of Georgia and suggest the role of fundamental differences in life-history traits in influencing the formation and maintenance of larval spatial structures.


2010 ◽  
Vol 25 (1) ◽  
pp. 343-354 ◽  
Author(s):  
Marion Mittermaier ◽  
Nigel Roberts

Abstract The fractions skill score (FSS) was one of the measures that formed part of the Intercomparison of Spatial Forecast Verification Methods project. The FSS was used to assess a common dataset that consisted of real and perturbed Weather Research and Forecasting (WRF) model precipitation forecasts, as well as geometric cases. These datasets are all based on the NCEP 240 grid, which translates to approximately 4-km resolution over the contiguous United States. The geometric cases showed that the FSS can provide a truthful assessment of displacement errors and forecast skill. In addition, the FSS can be used to determine the scale at which an acceptable level of skill is reached and this usage is perhaps more helpful than interpreting the actual FSS value. This spatial-scale approach is becoming more popular for monitoring operational forecast performance. The study also shows how the FSS responds to forecast bias. A more biased forecast always gives lower FSS values at large scales and usually at smaller scales. It is possible, however, for a more biased forecast to give a higher score at smaller scales, when additional rain overlaps the observed rain. However, given a sufficiently large sample of forecasts, a more biased forecast system will score lower. The use of percentile thresholds can remove the impacts of the bias. When the proportion of the domain that is “wet” (the wet-area ratio) is small, subtle differences introduced through near-threshold misses can lead to large changes in FSS magnitude in individual cases (primarily because the bias is changed). Reliable statistics for small wet-area ratios require a larger sample of forecasts. Care needs to be taken in the choice of verification domain. For high-resolution models, the domain should be large enough to encompass the length scale of the typical mesoscale forcing (e.g., upper-level troughs or squall lines). If the domain is too large, the wet-area ratios will always be small. If the domain is too small, fluctuations in the wet-area ratio can be large and larger spatial errors may be missed. The FSS is a good measure of the spatial accuracy of precipitation forecasts. Different methods are needed to determine other patterns of behavior.


Ocean Science ◽  
2015 ◽  
Vol 11 (6) ◽  
pp. 879-896 ◽  
Author(s):  
M. Haller ◽  
F. Janssen ◽  
J. Siddorn ◽  
W. Petersen ◽  
S. Dick

Abstract. For understanding and forecasting of hydrodynamics in coastal regions, numerical models have served as an important tool for many years. In order to assess the model performance, we compared simulations to observational data of water temperature and salinity. Observations were available from FerryBox transects in the southern North Sea and, additionally, from a fixed platform of the MARNET network. More detailed analyses have been made at three different stations, located off the English eastern coast, at the Oyster Ground and in the German Bight. FerryBoxes installed on ships of opportunity (SoO) provide high-frequency surface measurements along selected tracks on a regular basis. The results of two operational hydrodynamic models have been evaluated for two different time periods: BSHcmod v4 (January 2009 to April 2012) and FOAM AMM7 NEMO (April 2011 to April 2012). While they adequately simulate temperature, both models underestimate salinity, especially near the coast in the southern North Sea. Statistical errors differ between the two models and between the measured parameters. The root mean square error (RMSE) of water temperatures amounts to 0.72 °C (BSHcmod v4) and 0.44 °C (AMM7), while for salinity the performance of BSHcmod is slightly better (0.68 compared to 1.1). The study results reveal weaknesses in both models, in terms of variability, absolute levels and limited spatial resolution. Simulation of the transition zone between the coasts and the open sea is still a demanding task for operational modelling. Thus, FerryBox data, combined with other observations with differing temporal and spatial scales, can serve as an invaluable tool not only for model evaluation, but also for model optimization by assimilation of such high-frequency observations.


2018 ◽  
Vol 115 (47) ◽  
pp. 12069-12074 ◽  
Author(s):  
Samuel G. Roy ◽  
Emi Uchida ◽  
Simone P. de Souza ◽  
Ben Blachly ◽  
Emma Fox ◽  
...  

Aging infrastructure and growing interests in river restoration have led to a substantial rise in dam removals in the United States. However, the decision to remove a dam involves many complex trade-offs. The benefits of dam removal for hazard reduction and ecological restoration are potentially offset by the loss of hydroelectricity production, water supply, and other important services. We use a multiobjective approach to examine a wide array of trade-offs and synergies involved with strategic dam removal at three spatial scales in New England. We find that increasing the scale of decision-making improves the efficiency of trade-offs among ecosystem services, river safety, and economic costs resulting from dam removal, but this may lead to heterogeneous and less equitable local-scale outcomes. Our model may help facilitate multilateral funding, policy, and stakeholder agreements by analyzing the trade-offs of coordinated dam decisions, including net benefit alternatives to dam removal, at scales that satisfy these agreements.


2016 ◽  
Vol 2016 ◽  
pp. 1-11 ◽  
Author(s):  
Tien Du Duc ◽  
Lars Robert Hole ◽  
Duc Tran Anh ◽  
Cuong Hoang Duc ◽  
Thuy Nguyen Ba

The national numerical weather prediction system of Vietnam is presented and evaluated. The system is based on three main models, namely, the Japanese Global Spectral Model, the US Global Forecast System, and the US Weather Research and Forecasting (WRF) model. The global forecast products have been received at 0.25- and 0.5-degree horizontal resolution, respectively, and the WRF model has been run locally with 16 km horizontal resolution at the National Center for Hydro-Meteorological Forecasting using lateral conditions from GSM and GFS. The model performance is evaluated by comparing model output against observations of precipitation, wind speed, and temperature at 168 weather stations, with daily data from 2010 to 2014. In general, the global models provide more accurate forecasts than the regional models, probably due to the low horizontal resolution in the regional model. Also, the model performance is poorer for stations with altitudes greater than 500 meters above sea level (masl). For tropical cyclone performance validations, the maximum wind surface forecast from global and regional models is also verified against the best track of Joint Typhoon Warning Center. Finally, the model forecast skill during a recent extreme rain event in northeast Vietnam is evaluated.


Sign in / Sign up

Export Citation Format

Share Document