scholarly journals Evaluation of Analysis by Cross-Validation. Part II: Diagnostic and Optimization of Analysis Error Covariance

2017 ◽  
Author(s):  
Richard Ménard ◽  
Martin Deshaies-Jacques
Keyword(s):  
Air Quality ◽  
General Theory ◽  
Cross Validation ◽  
Error Variance ◽  
Geometric Interpretation ◽  
Quality Analysis ◽  
Error Margin ◽  
Error Covariance ◽  
Analysis Scheme ◽  
Analysis Error

We present a general theory of estimation of analysis error covariances based on cross-validation as well as a geometric interpretation of the method. In particular we use the variance of passive observation–minus-analysis residuals and show that the true analysis error variance can be estimated, without relying on the optimality assumption. This approach is used to obtain near optimal analyses that are then used to evaluate the air quality analysis error using several different methods at active and passive observation sites. We compare the estimates according to the method of Hollingsworth-Lönnberg, Desroziers et al., a new diagnostic we developed, and the perceived analysis error computed from the analysis scheme, to conclude that, as long as the analysis is near optimal, all estimates agree within a certain error margin.

scholarly journals Evaluation of Analysis by Cross-Validation. Part I: Using Verification Metrics

2018 ◽  
Author(s):  
Richard Menard ◽  
Martin Deshaies-Jacques
Keyword(s):  
Air Quality ◽  
Cross Validation ◽  
Error Variance ◽  
Quality Analysis ◽  
Second Moment ◽  
Error Margin ◽  
Analysis Scheme ◽  
Analysis Error ◽  
First Moment

We examine how observations can be used to evaluate an air quality analysis by verifying against passive observations (i.e. cross-validation) that are not used to create the analysis and we compare these verifications to those made against the same set of (active) observations that were used to generate the analysis. The results show that both active and passive observations can be used to evaluate of first moment metrics (e.g. bias) but only passive observations are useful to evaluate second moment metrics such as variance of observed-minus-analysis and correlation between observations and analysis. We derive a set of diagnostics based on passive observation–minus-analysis residuals and we show that the true analysis error variance can be estimated, without relying on any statistical optimality assumption. This diagnostic is used to obtain near optimal analyses that are then used to evaluate the analysis error using several different methods. We compare the estimates according to the method of Hollingsworth Lonnberg, Desroziers, a diagnostic we introduce, and the perceived analysis error computed from the analysis scheme, to conclude that as long as the analysis is optimal, all estimates agrees within a certain error margin. The analysis error variance at passive observation sites is also obtained.

scholarly journals Assimilation of GOSAT Methane in the Hemispheric CMAQ; Part II: Results Using Optimal Error Statistics

2022 ◽  
Vol 14 (2) ◽  
pp. 375
Author(s):  
Sina Voshtani ◽  
Richard Ménard ◽  
Thomas W. Walker ◽  
Amir Hakami
Keyword(s):  
Kalman Filter ◽  
Cross Validation ◽  
Error Variance ◽  
Error Statistics ◽  
Optimal Error ◽  
Meaningful Improvement ◽  
Aircraft Observations ◽  
Variance Parameters ◽  
Analysis Error ◽  
Or Modelling

We applied the parametric variance Kalman filter (PvKF) data assimilation designed in Part I of this two-part paper to GOSAT methane observations with the hemispheric version of CMAQ to obtain the methane field (i.e., optimized analysis) with its error variance. Although the Kalman filter computes error covariances, the optimality depends on how these covariances reflect the true error statistics. To achieve more accurate representation, we optimize the global variance parameters, including correlation length scales and observation errors, based on a cross-validation cost function. The model and the initial error are then estimated according to the normalized variance matching diagnostic, also to maintain a stable analysis error variance over time. The assimilation results in April 2010 are validated against independent surface and aircraft observations. The statistics of the comparison of the model and analysis show a meaningful improvement against all four types of available observations. Having the advantage of continuous assimilation, we showed that the analysis also aims at pursuing the temporal variation of independent measurements, as opposed to the model. Finally, the performance of the PvKF assimilation in capturing the spatial structure of bias and uncertainty reduction across the Northern Hemisphere is examined, indicating the capability of analysis in addressing those biases originated, whether from inaccurate emissions or modelling error.

scholarly journals Formulations for Estimating Spatial Variations of Analysis Error Variance to Improve Multiscale and Multistep Variational Data Assimilation

2018 ◽  
Vol 2018 ◽  
pp. 1-17
Author(s):  
Qin Xu ◽  
Li Wei
Keyword(s):  
Data Assimilation ◽  
Spatial Variation ◽  
Variational Analysis ◽  
Variance Reduction ◽  
Error Variance ◽  
Spatial Variations ◽  
Variational Data Assimilation ◽  
Coarse Resolution ◽  
Error Covariance ◽  
Analysis Error

When the coarse-resolution observations used in the first step of multiscale and multistep variational data assimilation become increasingly nonuniform and/or sparse, the error variance of the first-step analysis tends to have increasingly large spatial variations. However, the analysis error variance computed from the previously developed spectral formulations is constant and thus limited to represent only the spatially averaged error variance. To overcome this limitation, analytic formulations are constructed to efficiently estimate the spatial variation of analysis error variance and associated spatial variation in analysis error covariance. First, a suite of formulations is constructed to efficiently estimate the error variance reduction produced by analyzing the coarse-resolution observations in one- and two-dimensional spaces with increased complexity and generality (from uniformly distributed observations with periodic extension to nonuniformly distributed observations without periodic extension). Then, three different formulations are constructed for using the estimated analysis error variance to modify the analysis error covariance computed from the spectral formulations. The successively improved accuracies of these three formulations and their increasingly positive impacts on the two-step variational analysis (or multistep variational analysis in first two steps) are demonstrated by idealized experiments.

scholarly journals Evaluation of the Ensemble Transform Analysis Perturbation Scheme at NRL

2008 ◽  
Vol 136 (3) ◽  
pp. 1093-1108 ◽  
Author(s):  
Justin G. McLay ◽  
Craig H. Bishop ◽  
Carolyn A. Reynolds
Keyword(s):  
Weather Prediction ◽  
Error Variance ◽  
Naval Research ◽  
Perturbation Scheme ◽  
Stochastic Perturbations ◽  
Error Covariance ◽  
Analysis Error ◽  
Nwp Model ◽  
Ensemble Transform ◽  
Multivariate Correlations

Abstract The ensemble transform (ET) scheme changes forecast perturbations into analysis perturbations whose amplitudes and directions are consistent with a user-provided estimate of analysis error covariance. A practical demonstration of the ET scheme was undertaken using Naval Research Laboratory (NRL) Atmospheric Variational Data Assimilation System (NAVDAS) analysis error variance estimates and the Navy Operational Global Atmospheric Prediction System (NOGAPS) numerical weather prediction (NWP) model. It was found that the ET scheme produced forecast ensembles that were comparable to or better in a variety of measures than those produced by the Fleet Numerical and Oceanography Center (FNMOC) bred-growing modes (BGM) scheme. Also, the demonstration showed that the introduction of stochastic perturbations into the ET forecast ensembles led to a substantial improvement in the agreement between the ET and NAVDAS analysis error variances. This finding is strong evidence that even a small-sized ET ensemble is capable of obtaining good agreement between the ET and NAVDAS analysis error variances, provided that NWP model deficiencies are accounted for. Last, since the NAVDAS analysis error covariance estimate is diagonal and hence ignores multivariate correlations, it was of interest to examine the ET analysis perturbations’ spatial correlation. Tests showed that the ET analysis perturbations exhibited statistically significant, realistic multivariate correlations.

Effects of Accounting-Method Choices on Subjective Performance-Measure Weighting Decisions: Experimental Evidence on Precision and Error Covariance

2005 ◽  
Vol 80 (4) ◽  
pp. 1163-1192 ◽  
Author(s):  
Ranjani Krishnan ◽  
Joan L. Luft ◽  
Michael D. Shields
Keyword(s):  
Performance Measures ◽  
Error Variance ◽  
Spillover Effects ◽  
Performance Measure ◽  
Incentive System ◽  
Experimental Setting ◽  
Error Covariance ◽  
Psychology Theory ◽  
Optimal Weighting ◽  
Similarity And Difference

Performance-measure weights for incentive compensation are often determined subjectively. Determining these weights is a cognitively difficult task, and archival research shows that observed performance-measure weights are only partially consistent with the predictions of agency theory. Ittner et al. (2003) have concluded that psychology theory can help to explain such inconsistencies. In an experimental setting based on Feltham and Xie (1994), we use psychology theories of reasoning to predict distinctive patterns of similarity and difference between optimal and actual subjective performance-measure weights. The following predictions are supported. First, in contrast to a number of prior studies, most individuals' decisions are significantly influenced by the performance measures' error variance (precision) and error covariance. Second, directional errors in the use of these measurement attributes are relatively frequent, resulting in a mean underreaction to an accounting change that alters performance measurement error. Third, individuals seem insufficiently aware that a change in the accounting for one measure has spillover effects on the optimal weighting of the other measure in a two-measure incentive system. In consequence, they make performance-measure weighting decisions that are likely to result in misallocations of agent effort.

Maximum Likelihood Ensemble Filter: Theoretical Aspects

2005 ◽  
Vol 133 (6) ◽  
pp. 1710-1726 ◽  
Author(s):  
Milija Zupanski
Keyword(s):  
Maximum Likelihood ◽  
Data Assimilation ◽  
Cost Function ◽  
Ensemble Data Assimilation ◽  
Error Covariance ◽  
Ensemble Data ◽  
Analysis Error ◽  
Nonlinear Observation ◽  
The Cost ◽  
Maximum Likelihood Ensemble Filter

Abstract A new ensemble-based data assimilation method, named the maximum likelihood ensemble filter (MLEF), is presented. The analysis solution maximizes the likelihood of the posterior probability distribution, obtained by minimization of a cost function that depends on a general nonlinear observation operator. The MLEF belongs to the class of deterministic ensemble filters, since no perturbed observations are employed. As in variational and ensemble data assimilation methods, the cost function is derived using a Gaussian probability density function framework. Like other ensemble data assimilation algorithms, the MLEF produces an estimate of the analysis uncertainty (e.g., analysis error covariance). In addition to the common use of ensembles in calculation of the forecast error covariance, the ensembles in MLEF are exploited to efficiently calculate the Hessian preconditioning and the gradient of the cost function. A sufficient number of iterative minimization steps is 2–3, because of superior Hessian preconditioning. The MLEF method is well suited for use with highly nonlinear observation operators, for a small additional computational cost of minimization. The consistent treatment of nonlinear observation operators through optimization is an advantage of the MLEF over other ensemble data assimilation algorithms. The cost of MLEF is comparable to the cost of existing ensemble Kalman filter algorithms. The method is directly applicable to most complex forecast models and observation operators. In this paper, the MLEF method is applied to data assimilation with the one-dimensional Korteweg–de Vries–Burgers equation. The tested observation operator is quadratic, in order to make the assimilation problem more challenging. The results illustrate the stability of the MLEF performance, as well as the benefit of the cost function minimization. The improvement is noted in terms of the rms error, as well as the analysis error covariance. The statistics of innovation vectors (observation minus forecast) also indicate a stable performance of the MLEF algorithm. Additional experiments suggest the amplified benefit of targeted observations in ensemble data assimilation.

Air quality analysis in critical zones of Mexico using unsupervised machine learning

2020 ◽  
Author(s):  
Eli G. Pale-Ramon ◽  
Luis J. Morales-Mendoza ◽  
Sonia L. Mestizo-Gutierrez ◽  
Mario Gonzalez-Leee ◽  
Rene F. Vazquez-Bautista ◽  
...  
Keyword(s):  
Machine Learning ◽  
Air Quality ◽  
Quality Analysis ◽  
Unsupervised Machine Learning ◽  
Critical Zones

scholarly journals Interpretation of Adaptive Observing Guidance for Atlantic Tropical Cyclones

2007 ◽  
Vol 135 (12) ◽  
pp. 4006-4029 ◽  
Author(s):  
C. A. Reynolds ◽  
M. S. Peng ◽  
S. J. Majumdar ◽  
S. D. Aberson ◽  
C. H. Bishop ◽  
...  
Keyword(s):  
Data Assimilation ◽  
Total Energy ◽  
Tropical Cyclones ◽  
Error Variance ◽  
Naval Research ◽  
Error Statistics ◽  
Background Error ◽  
Analysis Error ◽  
The Impact ◽  
Perturbation Growth

Abstract Adaptive observing guidance products for Atlantic tropical cyclones are compared using composite techniques that allow one to quantitatively examine differences in the spatial structures of the guidance maps and relate these differences to the constraints and approximations of the respective techniques. The guidance maps are produced using the ensemble transform Kalman filter (ETKF) based on ensembles from the National Centers for Environmental Prediction and the European Centre for Medium-Range Weather Forecasts (ECMWF), and total-energy singular vectors (TESVs) produced by ECMWF and the Naval Research Laboratory. Systematic structural differences in the guidance products are linked to the fact that TESVs consider the dynamics of perturbation growth only, while the ETKF combines information on perturbation evolution with error statistics from an ensemble-based data assimilation scheme. The impact of constraining the SVs using different estimates of analysis error variance instead of a total-energy norm, in effect bringing the two methods closer together, is also assessed. When the targets are close to the storm, the TESV products are a maximum in an annulus around the storm, whereas the ETKF products are a maximum at the storm location itself. When the targets are remote from the storm, the TESVs almost always indicate targets northwest of the storm, whereas the ETKF targets are more scattered relative to the storm location and often occur over the northern North Atlantic. The ETKF guidance often coincides with locations in which the ensemble-based analysis error variance is large. As the TESV method is not designed to consider spatial differences in the likely analysis errors, it will produce targets over well-observed regions, such as the continental United States. Constraining the SV calculation using analysis error variance values from an operational 3D variational data assimilation system (with stationary, quasi-isotropic background error statistics) results in a modest modulation of the target areas away from the well-observed regions, and a modest reduction of perturbation growth. Constraining the SVs using the ETKF estimate of analysis error variance produces SV targets similar to ETKF targets and results in a significant reduction in perturbation growth, due to the highly localized nature of the analysis error variance estimates. These results illustrate the strong sensitivity of SVs to the norm (and to the analysis error variance estimate used to define it) and confirm that discrepancies between target areas computed using different methods reflect the mathematical and physical differences between the methods themselves.

Sign in / Sign up

Export Citation Format

Share Document