scholarly journals A Moment Matching Ensemble Filter for Nonlinear Non-Gaussian Data Assimilation

2011 ◽  
Vol 139 (12) ◽  
pp. 3964-3973 ◽  
Author(s):  
Jing Lei ◽  
Peter Bickel
Keyword(s):  
Kalman Filter ◽  
Ensemble Kalman Filter ◽  
Ensemble Forecasting ◽  
Moment Matching ◽  
State Variables ◽  
Mean And Variance ◽  
Large Systems ◽  
Non Gaussian ◽  
Lorenz 96 ◽  
Gaussian Data

Abstract The ensemble Kalman filter is now an important component of ensemble forecasting. While using the linear relationship between the observation and state variables makes it applicable for large systems, relying on linearity introduces nonnegligible bias since the true distribution will never be Gaussian. This paper analyzes the bias of the ensemble Kalman filter from a statistical perspective and proposes a debiasing method called the nonlinear ensemble adjustment filter. This new filter transforms the forecast ensemble in a statistically principled manner so that the updated ensemble has the desired mean and variance. It is also easily localizable and, hence, potentially useful for large systems. Its performance is demonstrated and compared with other Kalman filter and particle filter variants through various experiments on the Lorenz-63 and Lorenz-96 systems. The results show that the new filter is stable and accurate for challenging situations such as nonlinear, high-dimensional systems with sparse observations.

scholarly journals A hybrid particle-ensemble Kalman filter for problems with medium nonlinearity

PLoS ONE ◽  
2021 ◽  
Vol 16 (3) ◽  
pp. e0248266
Author(s):  
Ian Grooms ◽  
Gregor Robinson
Keyword(s):  
Kalman Filter ◽  
Particle Filter ◽  
Ensemble Kalman Filter ◽  
Orthogonal Transformation ◽  
Hybrid Filter ◽  
Hybrid Particle ◽  
Particle Ensemble ◽  
Non Gaussian ◽  
Two Stages ◽  
Lorenz 96

A hybrid particle ensemble Kalman filter is developed for problems with medium non-Gaussianity, i.e. problems where the prior is very non-Gaussian but the posterior is approximately Gaussian. Such situations arise, e.g., when nonlinear dynamics produce a non-Gaussian forecast but a tight Gaussian likelihood leads to a nearly-Gaussian posterior. The hybrid filter starts by factoring the likelihood. First the particle filter assimilates the observations with one factor of the likelihood to produce an intermediate prior that is close to Gaussian, and then the ensemble Kalman filter completes the assimilation with the remaining factor. How the likelihood gets split between the two stages is determined in such a way to ensure that the particle filter avoids collapse, and particle degeneracy is broken by a mean-preserving random orthogonal transformation. The hybrid is tested in a simple two-dimensional (2D) problem and a multiscale system of ODEs motivated by the Lorenz-‘96 model. In the 2D problem it outperforms both a pure particle filter and a pure ensemble Kalman filter, and in the multiscale Lorenz-‘96 model it is shown to outperform a pure ensemble Kalman filter, provided that the ensemble size is large enough.

scholarly journals Parameters Estimation and Prediction of Water Movement and Solute Transport in Layered, Variably Saturated Soils Using the Ensemble Kalman Filter

Water ◽  
2019 ◽  
Vol 11 (7) ◽  
pp. 1520
Author(s):  
Zheng Jiang ◽  
Quanzhong Huang ◽  
Gendong Li ◽  
Guangyong Li
Keyword(s):  
Kalman Filter ◽  
Solute Transport ◽  
Ensemble Kalman Filter ◽  
Water Movement ◽  
Parameters Estimation ◽  
Saturated Soils ◽  
Hydrological Models ◽  
State Variables ◽  
Ensemble Size ◽  
Highly Nonlinear

The parameters of water movement and solute transport models are essential for the accurate simulation of soil moisture and salinity, particularly for layered soils in field conditions. Parameter estimation can be achieved using the inverse modeling method. However, this type of method cannot fully consider the uncertainties of measurements, boundary conditions, and parameters, resulting in inaccurate estimations of parameters and predictions of state variables. The ensemble Kalman filter (EnKF) is well-suited to data assimilation and parameter prediction in Situations with large numbers of variables and uncertainties. Thus, in this study, the EnKF was used to estimate the parameters of water movement and solute transport in layered, variably saturated soils. Our results indicate that when used in conjunction with the HYDRUS-1D software (University of California Riverside, California, CA, USA) the EnKF effectively estimates parameters and predicts state variables for layered, variably saturated soils. The assimilation of factors such as the initial perturbation and ensemble size significantly affected in the simulated results. A proposed ensemble size range of 50–100 was used when applying the EnKF to the highly nonlinear hydrological models of the present study. Although the simulation results for moisture did not exhibit substantial improvement with the assimilation, the simulation of the salinity was significantly improved through the assimilation of the salinity and relative solutetransport parameters. Reducing the uncertainties in measured data can improve the goodness-of-fit in the application of the EnKF method. Sparse field condition observation data also benefited from the accurate measurement of state variables in the case of EnKF assimilation. However, the application of the EnKF algorithm for layered, variably saturated soils with hydrological models requires further study, because it is a challenging and highly nonlinear problem.

scholarly journals A Marginal Adjustment Rank Histogram Filter for Non-Gaussian Ensemble Data Assimilation

2020 ◽  
Author(s):  
Jeffrey L. Anderson
Keyword(s):  
Ensemble Kalman Filter ◽  
Ensemble Methods ◽  
State Variables ◽  
Histogram Method ◽  
Prior Distributions ◽  
Bounded State ◽  
Rank Histogram ◽  
Non Gaussian ◽  
Atmospheric Tracer ◽  
Filter Algorithms

An extension to standard ensemble Kalman filter algorithms that can improve performance for non-Gaussian prior distributions, non-Gaussian likelihoods, and bounded state variables is described. The algorithm exploits the capability of the rank histogram filter (RHF) to represent arbitrary prior distributions for observed variables. The rank histogram algorithm can be applied directly to state variables to produce posterior marginal ensembles without the need for regression that is part of standard ensemble filters. These marginals are used to adjust the marginals obtained from a standard ensemble filter that uses regression to update state variables. The final posterior ensemble is obtained by doing an ordered replacement of the posterior marginal ensemble values from a standard ensemble filter with the values obtained from the rank histogram method applied directly to state variables; the algorithm is referred to as the Marginal Adjustment Rank Histogram Filter (MARHF). Applications to idealized bivariate problems and low-order dynamical systems show that the MARHF can produce better results than standard ensemble methods for priors that are non-Gaussian. Like the original RHF, the MARHF can also make use of arbitrary non-Gaussian observation likelihoods. The MARHF also has advantages for problems with bounded state variables, for instance the concentration of an atmospheric tracer. Bounds can be automatically respected in the posterior ensembles. With an efficient implementation of the MARHF, the additional cost has better scaling than the standard RHF.

scholarly journals Convective-Scale Data Assimilation for the Weather Research and Forecasting Model Using the Local Particle Filter

2017 ◽  
Vol 145 (5) ◽  
pp. 1897-1918 ◽  
Author(s):  
Jonathan Poterjoy ◽  
Ryan A. Sobash ◽  
Jeffrey L. Anderson
Keyword(s):  
Kalman Filter ◽  
Data Assimilation ◽  
Ensemble Kalman Filter ◽  
Weather Prediction ◽  
Squall Line ◽  
Monte Carlo Data ◽  
Mixing Ratios ◽  
Convective Scale ◽  
Non Gaussian ◽  
Mean Square Errors

Abstract Particle filters (PFs) are Monte Carlo data assimilation techniques that operate with no parametric assumptions for prior and posterior errors. A data assimilation method introduced recently, called the local PF, approximates the PF solution within neighborhoods of observations, thus allowing for its use in high-dimensional systems. The current study explores the potential of the local PF for atmospheric data assimilation through cloud-permitting numerical experiments performed for an idealized squall line. Using only 100 ensemble members, experiments using the local PF to assimilate simulated radar measurements demonstrate that the method provides accurate analyses at a cost comparable to ensemble filters currently used in weather models. Comparisons between the local PF and an ensemble Kalman filter demonstrate benefits of the local PF for producing probabilistic analyses of non-Gaussian variables, such as hydrometeor mixing ratios. The local PF also provides more accurate forecasts than the ensemble Kalman filter, despite yielding higher posterior root-mean-square errors. A major advantage of the local PF comes from its ability to produce more physically consistent posterior members than the ensemble Kalman filter, which leads to fewer spurious model adjustments during forecasts. This manuscript presents the first successful application of the local PF in a weather prediction model and discusses implications for real applications where nonlinear measurement operators and nonlinear model processes limit the effectiveness of current Gaussian data assimilation techniques.

scholarly journals A Comparison of Two Local Moment-Matching Nonlinear Filters: Local Particle Filter (LPF) and Local Nonlinear Ensemble Transform Filter (LNETF)

2020 ◽  
Vol 148 (11) ◽  
pp. 4377-4395
Author(s):  
Jie Feng ◽  
Xuguang Wang ◽  
Jonathan Poterjoy
Keyword(s):  
Particle Filter ◽  
Nonlinear Filters ◽  
Classical Particle ◽  
Moment Matching ◽  
State Variables ◽  
Posterior Probability Distribution ◽  
Localization Function ◽  
Moment Estimation ◽  
Mean And Variance ◽  
Ensemble Transform

AbstractThe local particle filter (LPF) and the local nonlinear ensemble transform filter (LNETF) are two moment-matching nonlinear filters to approximate the classical particle filter (PF). They adopt different strategies to alleviate filter degeneracy. LPF and LNETF localize observational impact but use different localization functions. They assimilate observations in a partially sequential and a simultaneous manner, respectively. In addition, LPF applies the resampling step, whereas LNETF applies the deterministic square root transformation to update particles. Both methods preserve the posterior mean and variance of the PF. LNETF additionally preserves the posterior correlation of the PF for state variables within a local volume. These differences lead to their differing performance in filter stability and posterior moment estimation. LPF and LNETF are systematically compared and analyzed here through a set of experiments with a Lorenz model. Strategies to improve the LNETF are proposed. The original LNETF is inferior to the original LPF in filter stability and analysis accuracy, particularly for small particle numbers. This is attributed to both the localization function and particle update differences. The LNETF localization function imposes a stronger observation impact than the LPF for remote grids and thus is more susceptible to filter degeneracy. The LNETF update causes an overall narrower range of posteriors that excludes true states more frequently. After applying the same localization function as the LPF and additional posterior inflation to the LNETF, the two filters reach similar filter stability and analysis accuracy for all particle numbers. The improved LNETF shows more accurate posterior probability distribution but slightly worse spatial correlation of posteriors than the LPF.

scholarly journals A Bayesian approach to multivariate adaptive localization in ensemble-based data assimilation with time-dependent extensions

2019 ◽  
Vol 26 (2) ◽  
pp. 109-122 ◽  
Author(s):  
Andrey A. Popov ◽  
Adrian Sandu
Keyword(s):  
Kalman Filter ◽  
Bayesian Approach ◽  
Ensemble Kalman Filter ◽  
Time Dependent ◽  
Multivariate Approach ◽  
Filter Performance ◽  
Geophysical Model ◽  
Correlation Information ◽  
Lorenz 96 ◽  
Univariate Approach

Abstract. Ever since its inception, the ensemble Kalman filter (EnKF) has elicited many heuristic approaches that sought to improve it. One such method is covariance localization, which alleviates spurious correlations due to finite ensemble sizes by using relevant spatial correlation information. Adaptive localization techniques account for how correlations change in time and space, in order to obtain improved covariance estimates. This work develops a Bayesian approach to adaptive Schur-product localization for the deterministic ensemble Kalman filter (DEnKF) and extends it to support multiple radii of influence. We test the proposed adaptive localization using the toy Lorenz'96 problem and a more realistic 1.5-layer quasi-geostrophic model. Results with the toy problem show that the multivariate approach informs us that strongly observed variables can tolerate larger localization radii. The univariate approach leads to markedly improved filter performance for the realistic geophysical model, with a reduction in error by as much as 33 %.

A Non-Gaussian Ensemble Filter Update for Data Assimilation

2010 ◽  
Vol 138 (11) ◽  
pp. 4186-4198 ◽  
Author(s):  
Jeffrey L. Anderson
Keyword(s):  
Kalman Filter ◽  
Data Assimilation ◽  
Ensemble Kalman Filter ◽  
Sampling Error ◽  
Weather Prediction ◽  
Large Ensembles ◽  
Stochastic Filter ◽  
Pathological Behavior ◽  
Non Gaussian ◽  
Observation Space

Abstract A deterministic square root ensemble Kalman filter and a stochastic perturbed observation ensemble Kalman filter are used for data assimilation in both linear and nonlinear single variable dynamical systems. For the linear system, the deterministic filter is simply a method for computing the Kalman filter and is optimal while the stochastic filter has suboptimal performance due to sampling error. For the nonlinear system, the deterministic filter has increasing error as ensemble size increases because all ensemble members but one become tightly clustered. In this case, the stochastic filter performs better for sufficiently large ensembles. A new method for computing ensemble increments in observation space is proposed that does not suffer from the pathological behavior of the deterministic filter while avoiding much of the sampling error of the stochastic filter. This filter uses the order statistics of the prior observation space ensemble to create an approximate continuous prior probability distribution in a fashion analogous to the use of rank histograms for ensemble forecast evaluation. This rank histogram filter can represent non-Gaussian observation space priors and posteriors and is shown to be competitive with existing filters for problems as large as global numerical weather prediction. The ability to represent non-Gaussian distributions is useful for a variety of applications such as convective-scale assimilation and assimilation of bounded quantities such as relative humidity.

Assimilating Vortex Position with an Ensemble Kalman Filter

2007 ◽  
Vol 135 (5) ◽  
pp. 1828-1845 ◽  
Author(s):  
Yongsheng Chen ◽  
Chris Snyder
Keyword(s):  
Kalman Filter ◽  
Ensemble Kalman Filter ◽  
Track Forecast ◽  
Two Dimensional ◽  
Barotropic Model ◽  
Radar Imagery ◽  
Vortex Size ◽  
True Position ◽  
Non Gaussian ◽  
Background Forecast

Abstract Observations of hurricane position, which in practice might be available from satellite or radar imagery, can be easily assimilated with an ensemble Kalman filter (EnKF) given an operator that computes the position of the vortex in the background forecast. The simple linear updating scheme used in the EnKF is effective for small displacements of forecasted vortices from the true position; this situation is operationally relevant since hurricane position is often available frequently in time. When displacements of the forecasted vortices are comparable to the vortex size, non-Gaussian effects become significant and the EnKF’s linear update begins to degrade. Simulations using a simple two-dimensional barotropic model demonstrate the potential of the technique and show that the track forecast initialized with the EnKF analysis is improved. The assimilation of observations of the vortex shape and intensity, along with position, extends the technique’s effectiveness to larger displacements of the forecasted vortices than when assimilating position alone.

Sign in / Sign up

Export Citation Format

Share Document