scholarly journals Evaluation of Analysis by Cross-Validation, Part II: Diagnostic and Optimization of Analysis Error Covariance

Atmosphere ◽  
2018 ◽  
Vol 9 (2) ◽  
pp. 70 ◽  
Author(s):  
◽  
Keyword(s):  
Cross Validation ◽  
Error Covariance ◽  
Analysis Error

scholarly journals Evaluation of Analysis by Cross-Validation. Part II: Diagnostic and Optimization of Analysis Error Covariance

2017 ◽  
Author(s):  
Richard Ménard ◽  
Martin Deshaies-Jacques
Keyword(s):  
Air Quality ◽  
General Theory ◽  
Cross Validation ◽  
Error Variance ◽  
Geometric Interpretation ◽  
Quality Analysis ◽  
Error Margin ◽  
Error Covariance ◽  
Analysis Scheme ◽  
Analysis Error

We present a general theory of estimation of analysis error covariances based on cross-validation as well as a geometric interpretation of the method. In particular we use the variance of passive observation–minus-analysis residuals and show that the true analysis error variance can be estimated, without relying on the optimality assumption. This approach is used to obtain near optimal analyses that are then used to evaluate the air quality analysis error using several different methods at active and passive observation sites. We compare the estimates according to the method of Hollingsworth-Lönnberg, Desroziers et al., a new diagnostic we developed, and the perceived analysis error computed from the analysis scheme, to conclude that, as long as the analysis is near optimal, all estimates agree within a certain error margin.

Maximum Likelihood Ensemble Filter: Theoretical Aspects

2005 ◽  
Vol 133 (6) ◽  
pp. 1710-1726 ◽  
Author(s):  
Milija Zupanski
Keyword(s):  
Maximum Likelihood ◽  
Data Assimilation ◽  
Cost Function ◽  
Ensemble Data Assimilation ◽  
Error Covariance ◽  
Ensemble Data ◽  
Analysis Error ◽  
Nonlinear Observation ◽  
The Cost ◽  
Maximum Likelihood Ensemble Filter

Abstract A new ensemble-based data assimilation method, named the maximum likelihood ensemble filter (MLEF), is presented. The analysis solution maximizes the likelihood of the posterior probability distribution, obtained by minimization of a cost function that depends on a general nonlinear observation operator. The MLEF belongs to the class of deterministic ensemble filters, since no perturbed observations are employed. As in variational and ensemble data assimilation methods, the cost function is derived using a Gaussian probability density function framework. Like other ensemble data assimilation algorithms, the MLEF produces an estimate of the analysis uncertainty (e.g., analysis error covariance). In addition to the common use of ensembles in calculation of the forecast error covariance, the ensembles in MLEF are exploited to efficiently calculate the Hessian preconditioning and the gradient of the cost function. A sufficient number of iterative minimization steps is 2–3, because of superior Hessian preconditioning. The MLEF method is well suited for use with highly nonlinear observation operators, for a small additional computational cost of minimization. The consistent treatment of nonlinear observation operators through optimization is an advantage of the MLEF over other ensemble data assimilation algorithms. The cost of MLEF is comparable to the cost of existing ensemble Kalman filter algorithms. The method is directly applicable to most complex forecast models and observation operators. In this paper, the MLEF method is applied to data assimilation with the one-dimensional Korteweg–de Vries–Burgers equation. The tested observation operator is quadratic, in order to make the assimilation problem more challenging. The results illustrate the stability of the MLEF performance, as well as the benefit of the cost function minimization. The improvement is noted in terms of the rms error, as well as the analysis error covariance. The statistics of innovation vectors (observation minus forecast) also indicate a stable performance of the MLEF algorithm. Additional experiments suggest the amplified benefit of targeted observations in ensemble data assimilation.

scholarly journals Assimilation of GOSAT Methane in the Hemispheric CMAQ; Part II: Results Using Optimal Error Statistics

2022 ◽  
Vol 14 (2) ◽  
pp. 375
Author(s):  
Sina Voshtani ◽  
Richard Ménard ◽  
Thomas W. Walker ◽  
Amir Hakami
Keyword(s):  
Kalman Filter ◽  
Cross Validation ◽  
Error Variance ◽  
Error Statistics ◽  
Optimal Error ◽  
Meaningful Improvement ◽  
Aircraft Observations ◽  
Variance Parameters ◽  
Analysis Error ◽  
Or Modelling

We applied the parametric variance Kalman filter (PvKF) data assimilation designed in Part I of this two-part paper to GOSAT methane observations with the hemispheric version of CMAQ to obtain the methane field (i.e., optimized analysis) with its error variance. Although the Kalman filter computes error covariances, the optimality depends on how these covariances reflect the true error statistics. To achieve more accurate representation, we optimize the global variance parameters, including correlation length scales and observation errors, based on a cross-validation cost function. The model and the initial error are then estimated according to the normalized variance matching diagnostic, also to maintain a stable analysis error variance over time. The assimilation results in April 2010 are validated against independent surface and aircraft observations. The statistics of the comparison of the model and analysis show a meaningful improvement against all four types of available observations. Having the advantage of continuous assimilation, we showed that the analysis also aims at pursuing the temporal variation of independent measurements, as opposed to the model. Finally, the performance of the PvKF assimilation in capturing the spatial structure of bias and uncertainty reduction across the Northern Hemisphere is examined, indicating the capability of analysis in addressing those biases originated, whether from inaccurate emissions or modelling error.

scholarly journals Assimilation of Stratospheric Temperature and Ozone with an Ensemble Kalman Filter in a Chemistry–Climate Model

2011 ◽  
Vol 139 (11) ◽  
pp. 3389-3404 ◽  
Author(s):  
Thomas Milewski ◽  
Michel S. Bourqui
Keyword(s):  
Kalman Filter ◽  
Data Assimilation ◽  
Covariance Matrix ◽  
Ensemble Kalman Filter ◽  
Climate Model ◽  
Observation Error ◽  
Background Error ◽  
Error Covariance Matrix ◽  
Error Covariance ◽  
Analysis Error

Abstract A new stratospheric chemical–dynamical data assimilation system was developed, based upon an ensemble Kalman filter coupled with a Chemistry–Climate Model [i.e., the intermediate-complexity general circulation model Fast Stratospheric Ozone Chemistry (IGCM-FASTOC)], with the aim to explore the potential of chemical–dynamical coupling in stratospheric data assimilation. The system is introduced here in a context of a perfect-model, Observing System Simulation Experiment. The system is found to be sensitive to localization parameters, and in the case of temperature (ozone), assimilation yields its best performance with horizontal and vertical decorrelation lengths of 14 000 km (5600 km) and 70 km (14 km). With these localization parameters, the observation space background-error covariance matrix is underinflated by only 5.9% (overinflated by 2.1%) and the observation-error covariance matrix by only 1.6% (0.5%), which makes artificial inflation unnecessary. Using optimal localization parameters, the skills of the system in constraining the ensemble-average analysis error with respect to the true state is tested when assimilating synthetic Michelson Interferometer for Passive Atmospheric Sounding (MIPAS) retrievals of temperature alone and ozone alone. It is found that in most cases background-error covariances produced from ensemble statistics are able to usefully propagate information from the observed variable to other ones. Chemical–dynamical covariances, and in particular ozone–wind covariances, are essential in constraining the dynamical fields when assimilating ozone only, as the radiation in the stratosphere is too slow to transfer ozone analysis increments to the temperature field over the 24-h forecast window. Conversely, when assimilating temperature, the chemical–dynamical covariances are also found to help constrain the ozone field, though to a much lower extent. The uncertainty in forecast/analysis, as defined by the variability in the ensemble, is large compared to the analysis error, which likely indicates some amount of noise in the covariance terms, while also reducing the risk of filter divergence.

Development of a nonlinear ensemble data assimilation method with global state-space covariance localization

2020 ◽  
Author(s):  
Milija Zupanski
Keyword(s):  
Data Assimilation ◽  
State Space ◽  
High Dimensional ◽  
Ensemble Data Assimilation ◽  
Error Covariance ◽  
Ensemble Data ◽  
Space Localization ◽  
Initial Ensemble ◽  
Analysis Error ◽  
Observation Space

<p>High-dimensional ensemble data assimilation applications require error covariance localization in order to address the problem of insufficient degrees of freedom, typically accomplished using the observation-space covariance localization. However, this creates a challenge for vertically integrated observations, such as satellite radiances, aerosol optical depth, etc., since the exact observation location in vertical does not exist. For nonlinear problems, there is an implied inconsistency in iterative minimization due to using observation-space localization which effectively prevents finding the optimal global minimizing solution. Using state-space localization, however, in principal resolves both issues associated with observation space localization.</p><p> </p><p>In this work we present a new nonlinear ensemble data assimilation method that employs covariance localization in state space and finds an optimal analysis solution. The new method resembles “modified ensembles” in the sense that ensemble size is increased in the analysis, but it differs in methodology used to create ensemble modifications, calculate the analysis error covariance, and define the initial ensemble perturbations for data assimilation cycling. From a practical point of view, the new method is considerably more efficient and potentially applicable to realistic high-dimensional data assimilation problems. A distinct characteristic of the new algorithm is that the localized error covariance and minimization are global, i.e. explicitly defined over all state points. The presentation will focus on examining feasible options for estimating the analysis error covariance and for defining the initial ensemble perturbations.</p>

scholarly journals An estimate of the inflation factor and analysis sensitivity in the ensemble Kalman filter

2017 ◽  
Vol 24 (3) ◽  
pp. 329-341 ◽  
Author(s):  
Guocan Wu ◽  
Xiaogu Zheng
Keyword(s):  
Kalman Filter ◽  
Covariance Matrix ◽  
Ensemble Kalman Filter ◽  
Forecast Error ◽  
Model Errors ◽  
Error Covariance Matrix ◽  
Spatially Correlated ◽  
Error Covariance ◽  
Inflation Factor ◽  
Analysis Error

Abstract. The ensemble Kalman filter (EnKF) is a widely used ensemble-based assimilation method, which estimates the forecast error covariance matrix using a Monte Carlo approach that involves an ensemble of short-term forecasts. While the accuracy of the forecast error covariance matrix is crucial for achieving accurate forecasts, the estimate given by the EnKF needs to be improved using inflation techniques. Otherwise, the sampling covariance matrix of perturbed forecast states will underestimate the true forecast error covariance matrix because of the limited ensemble size and large model errors, which may eventually result in the divergence of the filter. In this study, the forecast error covariance inflation factor is estimated using a generalized cross-validation technique. The improved EnKF assimilation scheme is tested on the atmosphere-like Lorenz-96 model with spatially correlated observations, and is shown to reduce the analysis error and increase its sensitivity to the observations.

scholarly journals Formulations for Estimating Spatial Variations of Analysis Error Variance to Improve Multiscale and Multistep Variational Data Assimilation

2018 ◽  
Vol 2018 ◽  
pp. 1-17
Author(s):  
Qin Xu ◽  
Li Wei
Keyword(s):  
Data Assimilation ◽  
Spatial Variation ◽  
Variational Analysis ◽  
Variance Reduction ◽  
Error Variance ◽  
Spatial Variations ◽  
Variational Data Assimilation ◽  
Coarse Resolution ◽  
Error Covariance ◽  
Analysis Error

When the coarse-resolution observations used in the first step of multiscale and multistep variational data assimilation become increasingly nonuniform and/or sparse, the error variance of the first-step analysis tends to have increasingly large spatial variations. However, the analysis error variance computed from the previously developed spectral formulations is constant and thus limited to represent only the spatially averaged error variance. To overcome this limitation, analytic formulations are constructed to efficiently estimate the spatial variation of analysis error variance and associated spatial variation in analysis error covariance. First, a suite of formulations is constructed to efficiently estimate the error variance reduction produced by analyzing the coarse-resolution observations in one- and two-dimensional spaces with increased complexity and generality (from uniformly distributed observations with periodic extension to nonuniformly distributed observations without periodic extension). Then, three different formulations are constructed for using the estimated analysis error variance to modify the analysis error covariance computed from the spectral formulations. The successively improved accuracies of these three formulations and their increasingly positive impacts on the two-step variational analysis (or multistep variational analysis in first two steps) are demonstrated by idealized experiments.

scholarly journals Mitigating Observation Perturbation Sampling Errors in the Stochastic EnKF

2015 ◽  
Vol 143 (7) ◽  
pp. 2918-2936 ◽  
Author(s):  
I. Hoteit ◽  
D.-T. Pham ◽  
M. E. Gharamti ◽  
X. Luo
Keyword(s):  
Ensemble Kalman Filter ◽  
Forecast Error ◽  
Sampling Errors ◽  
Observational Error ◽  
Error Covariance ◽  
Atmospheric Data ◽  
Observational Errors ◽  
Analysis Error ◽  
Lorenz 96 ◽  
The Impact

Abstract The stochastic ensemble Kalman filter (EnKF) updates its ensemble members with observations perturbed with noise sampled from the distribution of the observational errors. This was shown to introduce noise into the system and may become pronounced when the ensemble size is smaller than the rank of the observational error covariance, which is often the case in real oceanic and atmospheric data assimilation applications. This work introduces an efficient serial scheme to mitigate the impact of observations’ perturbations sampling in the analysis step of the EnKF, which should provide more accurate ensemble estimates of the analysis error covariance matrices. The new scheme is simple to implement within the serial EnKF algorithm, requiring only the approximation of the EnKF sample forecast error covariance matrix by a matrix with one rank less. The new EnKF scheme is implemented and tested with the Lorenz-96 model. Results from numerical experiments are conducted to compare its performance with the EnKF and two standard deterministic EnKFs. This study shows that the new scheme enhances the behavior of the EnKF and may lead to better performance than the deterministic EnKFs even when implemented with relatively small ensembles.

Sign in / Sign up

Export Citation Format

Share Document