scholarly journals The Square Root Information Increment Ensemble Filter

2016 ◽  
Vol 144 (12) ◽  
pp. 4667-4686
Author(s):  
Mark L. Psiaki
Keyword(s):  
Kalman Filters ◽  
A Priori ◽  
Full Rank ◽  
Forecast Model ◽  
Model Error ◽  
The State ◽  
Low Rank ◽  
Square Root ◽  
Ensemble Kalman Filters ◽  
Lorenz 96

Abstract A new type of ensemble filter is developed, one that stores and updates its state information in an efficient square root information filter form. It addresses two shortcomings of conventional ensemble Kalman filters: the coarse characterization of random forecast model error effects and the overly optimistic approximation of the estimation error statistics. The new filter uses an assumed a priori covariance approximation that is full rank but sparse, possibly with a dense low-rank increment. This matrix can be used to develop a nominal square root information equation for the system state and uncertainty. The measurements are used to develop an additional low-rank square root information equation. New algorithms provide forecasts and analyses of these increments at a computational cost comparable to that of existing ensemble Kalman filters. Model error effects are implicit in the a priori covariance time history, thereby obviating one of the reasons for including an inflation operation. The use of an a priori full-rank covariance allows the analysis operations to improve the state estimate without the need for a localization adjustment. This new filter exhibited worse performance than a typical covariance square root ensemble Kalman filter when operating on the Lorenz-96 problem in a chaotic regime. It excelled on a version of the Lorenz-96 problem where nonlinearities in the forecast model were weak, where the state vector uncertainty lay predominantly in a small subspace, and where the observations were spatially sparse. Such a problem might be representative of ionospheric space weather data assimilation where forcing variability can dominate the state uncertainty and where remote sensing data coverage can be sparse.

scholarly journals Accounting for Model Error from Unresolved Scales in Ensemble Kalman Filters by Stochastic Parameterization

2017 ◽  
Vol 145 (9) ◽  
pp. 3709-3723 ◽  
Author(s):  
Fei Lu ◽  
Xuemin Tu ◽  
Alexandre J. Chorin
Keyword(s):  
Kalman Filters ◽  
Numerical Experiments ◽  
Sampling Error ◽  
Forecast Model ◽  
Model Error ◽  
Ensemble Size ◽  
Ensemble Kalman Filters ◽  
Filter Performance ◽  
Performance Results ◽  
Lorenz 96

The use of discrete-time stochastic parameterization to account for model error due to unresolved scales in ensemble Kalman filters is investigated by numerical experiments. The parameterization quantifies the model error and produces an improved non-Markovian forecast model, which generates high quality forecast ensembles and improves filter performance. Results are compared with the methods of dealing with model error through covariance inflation and localization (IL), using as an example the two-layer Lorenz-96 system. The numerical results show that when the ensemble size is sufficiently large, the parameterization is more effective in accounting for the model error than IL; if the ensemble size is small, IL is needed to reduce sampling error, but the parameterization further improves the performance of the filter. This suggests that in real applications where the ensemble size is relatively small, the filter can achieve better performance than pure IL if stochastic parameterization methods are combined with IL.

scholarly journals A Data-Driven Method for Improving the Correlation Estimation in Serial Ensemble Kalman Filters

2017 ◽  
Vol 145 (3) ◽  
pp. 985-1001 ◽  
Author(s):  
Michèle De La Chevrotière ◽  
John Harlim
Keyword(s):  
Kalman Filters ◽  
Data Driven ◽  
Training Procedure ◽  
Ensemble Kalman Filters ◽  
Regression Problem ◽  
Correlation Estimation ◽  
Sample Correlation ◽  
Correlation Statistics ◽  
Nonlinear Observation ◽  
Lorenz 96

A data-driven method for improving the correlation estimation in serial ensemble Kalman filters is introduced. The method finds a linear map that transforms, at each assimilation cycle, the poorly estimated sample correlation into an improved correlation. This map is obtained from an offline training procedure without any tuning as the solution of a linear regression problem that uses appropriate sample correlation statistics obtained from historical data assimilation outputs. In an idealized OSSE with the Lorenz-96 model and for a range of linear and nonlinear observation models, the proposed scheme improves the filter estimates, especially when the ensemble size is small relative to the dimension of the state space.

scholarly journals On Serial Observation Processing in Localized Ensemble Kalman Filters

2015 ◽  
Vol 143 (5) ◽  
pp. 1554-1567 ◽  
Author(s):  
Lars Nerger
Keyword(s):  
Ensemble Kalman Filter ◽  
Kalman Filters ◽  
Large Scale ◽  
The State ◽  
Ensemble Kalman Filters ◽  
Filter Performance ◽  
Error Covariance ◽  
Analysis Process ◽  
Scale Models ◽  
Serial Observation

Abstract Ensemble square root filters can either assimilate all observations that are available at a given time at once, or assimilate the observations in batches or one at a time. For large-scale models, the filters are typically applied with a localized analysis step. This study demonstrates that the interaction of serial observation processing and localization can destabilize the analysis process, and it examines under which conditions the instability becomes significant. The instability results from a repeated inconsistent update of the state error covariance matrix that is caused by the localization. The inconsistency is present in all ensemble Kalman filters, except for the classical ensemble Kalman filter with perturbed observations. With serial observation processing, its effect is small in cases when the assimilation changes the ensemble of model states only slightly. However, when the assimilation has a strong effect on the state estimates, the interaction of localization and serial observation processing can significantly deteriorate the filter performance. In realistic large-scale applications, when the assimilation changes the states only slightly and when the distribution of the observations is irregular and changing over time, the instability is likely not significant.

scholarly journals Penalized ensemble Kalman filters for high dimensional non-linear systems

PLoS ONE ◽  
2021 ◽  
Vol 16 (3) ◽  
pp. e0248046
Author(s):  
Elizabeth Hou ◽  
Earl Lawrence ◽  
Alfred O. Hero
Keyword(s):  
Kalman Filter ◽  
Ensemble Kalman Filter ◽  
Estimation Error ◽  
Covariance Structure ◽  
Full Rank ◽  
The State ◽  
High Dimensional ◽  
Ensemble Kalman Filters ◽  
Non Linear ◽  
Non Linear System

The ensemble Kalman filter (EnKF) is a data assimilation technique that uses an ensemble of models, updated with data, to track the time evolution of a usually non-linear system. It does so by using an empirical approximation to the well-known Kalman filter. However, its performance can suffer when the ensemble size is smaller than the state space, as is often necessary for computationally burdensome models. This scenario means that the empirical estimate of the state covariance is not full rank and possibly quite noisy. To solve this problem in this high dimensional regime, we propose a computationally fast and easy to implement algorithm called the penalized ensemble Kalman filter (PEnKF). Under certain conditions, it can be theoretically proven that the PEnKF will be accurate (the estimation error will converge to zero) despite having fewer ensemble members than state dimensions. Further, as contrasted to localization methods, the proposed approach learns the covariance structure associated with the dynamical system. These theoretical results are supported with simulations of several non-linear and high dimensional systems.

scholarly journals Particle Kalman Filtering: A Nonlinear Bayesian Framework for Ensemble Kalman Filters

2012 ◽  
Vol 140 (2) ◽  
pp. 528-542 ◽  
Author(s):  
Ibrahim Hoteit ◽  
Xiaodong Luo ◽  
Dinh-Tuan Pham
Keyword(s):  
Kalman Filter ◽  
Particle Filter ◽  
Kalman Filters ◽  
Nonlinear Filtering ◽  
Weighted Average ◽  
Gaussian Mixture ◽  
Ensemble Kalman Filters ◽  
Strongly Nonlinear ◽  
Special Cases ◽  
Lorenz 96

This paper investigates an approximation scheme of the optimal nonlinear Bayesian filter based on the Gaussian mixture representation of the state probability distribution function. The resulting filter is similar to the particle filter, but is different from it in that the standard weight-type correction in the particle filter is complemented by the Kalman-type correction with the associated covariance matrices in the Gaussian mixture. The authors show that this filter is an algorithm in between the Kalman filter and the particle filter, and therefore is referred to as the particle Kalman filter (PKF). In the PKF, the solution of a nonlinear filtering problem is expressed as the weighted average of an “ensemble of Kalman filters” operating in parallel. Running an ensemble of Kalman filters is, however, computationally prohibitive for realistic atmospheric and oceanic data assimilation problems. For this reason, the authors consider the construction of the PKF through an “ensemble” of ensemble Kalman filters (EnKFs) instead, and call the implementation the particle EnKF (PEnKF). It is shown that different types of the EnKFs can be considered as special cases of the PEnKF. Similar to the situation in the particle filter, the authors also introduce a resampling step to the PEnKF in order to reduce the risk of weights collapse and improve the performance of the filter. Numerical experiments with the strongly nonlinear Lorenz-96 model are presented and discussed.

scholarly journals Efficient Local Error Parameterizations for Square Root or Ensemble Kalman Filters: Application to a Basin-Scale Ocean Turbulent Flow

2011 ◽  
Vol 139 (2) ◽  
pp. 474-493 ◽  
Author(s):  
Jean-Michel Brankart ◽  
Emmanuel Cosme ◽  
Charles-Emmanuel Testut ◽  
Pierre Brasseur ◽  
Jacques Verron
Keyword(s):  
Turbulent Flow ◽  
Covariance Matrix ◽  
Kalman Filters ◽  
Square Root ◽  
Ensemble Kalman Filters ◽  
Error Covariance Matrix ◽  
Square Roots ◽  
Schur Product ◽  
Error Covariance ◽  
Basin Scale

Abstract In large-sized atmospheric or oceanic applications of square root or ensemble Kalman filters, it is often necessary to introduce the prior assumption that long-range correlations are negligible and force them to zero using a local parameterization, supplementing the ensemble or reduced-rank representation of the covariance. One classic algorithm to perform this operation consists of taking the Schur product of the ensemble covariance matrix by a local support correlation matrix. However, with this parameterization, the square root of the forecast error covariance matrix is no more directly available, so that any observational update algorithm requiring this square root must include an additional step to compute local square roots from the Schur product. This computation generates an additional numerical cost or produces high-rank square roots, which may deprive the observational update from its original efficiency. In this paper, it is shown how efficient local square root parameterizations can be obtained, for use with a specific square root formulation (i.e., eigenbasis algorithm) of the observational update. Comparisons with the classic algorithm are provided, mainly in terms of consistency, accuracy, and computational complexity. As an application, the resulting parameterization is used to estimate maps of dynamic topography characterizing a basin-scale ocean turbulent flow. Even with this moderate-sized system (a 2200-km-wide square basin with 100-km-wide mesoscale eddies), it is observed that more than 1000 ensemble members are necessary to faithfully represent the global correlation patterns, and that a local parameterization is needed to produce correct covariances with moderate-sized ensembles. Comparisons with the exact solution show that the use of local square roots is able to improve the accuracy of the updated ensemble mean and the consistency of the updated ensemble variance. With the eigenbasis algorithm, optimal adaptive estimates of scaling factors for the forecast and observation error covariance matrix can also be obtained locally at negligible additional numerical cost. Finally, a comparison of the overall computational cost illustrates the decisive advantage that efficient local square root parameterizations may have to deal simultaneously with a larger number of observations and avoid data thinning as much as possible.

scholarly journals Efficient Parameterization of the Observation Error Covariance Matrix for Square Root or Ensemble Kalman Filters: Application to Ocean Altimetry

2009 ◽  
Vol 137 (6) ◽  
pp. 1908-1927 ◽  
Author(s):  
Jean-Michel Brankart ◽  
Clément Ubelmann ◽  
Charles-Emmanuel Testut ◽  
Emmanuel Cosme ◽  
Pierre Brasseur ◽  
...  
Keyword(s):  
Covariance Matrix ◽  
Kalman Filters ◽  
Signal Amplitude ◽  
Observation Error ◽  
Square Root ◽  
Ensemble Kalman Filters ◽  
Error Covariance Matrix ◽  
Standard Algorithm ◽  
Error Covariance ◽  
North Brazil

Abstract In the Kalman filter standard algorithm, the computational complexity of the observational update is proportional to the cube of the number y of observations (leading behavior for large y). In realistic atmospheric or oceanic applications, involving an increasing quantity of available observations, this often leads to a prohibitive cost and to the necessity of simplifying the problem by aggregating or dropping observations. If the filter error covariance matrices are in square root form, as in square root or ensemble Kalman filters, the standard algorithm can be transformed to be linear in y, providing that the observation error covariance matrix is diagonal. This is a significant drawback of this transformed algorithm and often leads to an assumption of uncorrelated observation errors for the sake of numerical efficiency. In this paper, it is shown that the linearity of the transformed algorithm in y can be preserved for other forms of the observation error covariance matrix. In particular, quite general correlation structures (with analytic asymptotic expressions) can be simulated simply by augmenting the observation vector with differences of the original observations, such as their discrete gradients. Errors in ocean altimetric observations are spatially correlated, as for instance orbit or atmospheric errors along the satellite track. Adequately parameterizing these correlations can directly improve the quality of observational updates and the accuracy of the associated error estimates. In this paper, the example of the North Brazil Current circulation is used to demonstrate the importance of this effect, which is especially significant in that region of moderate ratio between signal amplitude and observation noise, and to show that the efficient parameterization that is proposed for the observation error correlations is appropriate to take it into account. Adding explicit gradient observations also receives a physical justification. This parameterization is thus proved to be useful to ocean data assimilation systems that are based on square root or ensemble Kalman filters, as soon as the number of observations becomes penalizing, and if a sophisticated parameterization of the observation error correlations is required.

scholarly journals A Probabilistic Approach to Adaptive Covariance Localization for Serial Ensemble Square Root Filters

2014 ◽  
Vol 142 (12) ◽  
pp. 4499-4518 ◽  
Author(s):  
Yicun Zhen ◽  
Fuqing Zhang
Keyword(s):  
Probabilistic Approach ◽  
Filter Method ◽  
Square Root ◽  
Ensemble Size ◽  
Sampling Errors ◽  
Jeffreys Prior ◽  
Ensemble Kalman Filters ◽  
Localization Radius ◽  
Lorenz 96 ◽  
The Impact

Abstract This study proposes a variational approach to adaptively determine the optimum radius of influence for ensemble covariance localization when uncorrelated observations are assimilated sequentially. The covariance localization is commonly used by various ensemble Kalman filters to limit the impact of covariance sampling errors when the ensemble size is small relative to the dimension of the state. The probabilistic approach is based on the premise of finding an optimum localization radius that minimizes the distance between the Kalman update using the localized sampling covariance versus using the true covariance, when the sequential ensemble Kalman square root filter method is used. The authors first examine the effectiveness of the proposed method for the cases when the true covariance is known or can be approximated by a sufficiently large ensemble size. Not surprisingly, it is found that the smaller the true covariance distance or the smaller the ensemble, the smaller the localization radius that is needed. The authors further generalize the method to the more usual scenario that the true covariance is unknown but can be represented or estimated probabilistically based on the ensemble sampling covariance. The mathematical formula for this probabilistic and adaptive approach with the use of the Jeffreys prior is derived. Promising results and limitations of this new method are discussed through experiments using the Lorenz-96 system.

Sign in / Sign up

Export Citation Format

Share Document