Expanding the validity of the ensemble Kalman filter without the intrinsic need for inflation

Abstract. The ensemble Kalman filter (EnKF) is a powerful data assimilation method meant for high-dimensional nonlinear systems. But its implementation requires fixes such as localization and inflation. The recently developed finite-size ensemble Kalman filter (EnKF-N) does not require multiplicative inflation meant to counteract sampling errors. Aside from the practical interest of avoiding the tuning of inflation in perfect model data assimilation experiments, it also offers theoretical insights and a unique perspective on the EnKF. Here, we revisit, clarify and correct several key points of the EnKF-N derivation. This simplifies the use of the method, and expands its validity. The EnKF is shown to not only rely on the observations and the forecast ensemble but also on an implicit prior assumption, termed hyperprior, that fills in the gap of missing information. In the EnKF-N framework, this assumption is made explicit through a Bayesian hierarchy. This hyperprior has been so far chosen to be the uninformative Jeffreys' prior. Here, this choice is revisited to improve the performance of the EnKF-N in the regime where the analysis strongly relaxes to the prior. Moreover, it is shown that the EnKF-N can be extended with a normal-inverse-Wishart informative hyperprior that additionally introduces climatological error statistics. This can be identified as a hybrid 3D-Var/EnKF counterpart to the EnKF-N.

Download Full-text

Expanding the validity of the ensemble Kalman filter without the intrinsic need for inflation

Nonlinear Processes in Geophysics ◽

10.5194/npg-22-645-2015 ◽

2015 ◽

Vol 22 (6) ◽

pp. 645-662 ◽

Cited By ~ 26

Author(s):

M. Bocquet ◽

P. N. Raanes ◽

A. Hannart

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Ad Hoc ◽

Finite Size ◽

Practical Interest ◽

Sampling Errors ◽

Jeffreys Prior ◽

Error Statistics ◽

Additional Information

Abstract. The ensemble Kalman filter (EnKF) is a powerful data assimilation method meant for high-dimensional nonlinear systems. But its implementation requires somewhat ad hoc procedures such as localization and inflation. The recently developed finite-size ensemble Kalman filter (EnKF-N) does not require multiplicative inflation meant to counteract sampling errors. Aside from the practical interest in avoiding the tuning of inflation in perfect model data assimilation experiments, it also offers theoretical insights and a unique perspective on the EnKF. Here, we revisit, clarify and correct several key points of the EnKF-N derivation. This simplifies the use of the method, and expands its validity. The EnKF is shown to not only rely on the observations and the forecast ensemble, but also on an implicit prior assumption, termed hyperprior, that fills in the gap of missing information. In the EnKF-N framework, this assumption is made explicit through a Bayesian hierarchy. This hyperprior has so far been chosen to be the uninformative Jeffreys prior. Here, this choice is revisited to improve the performance of the EnKF-N in the regime where the analysis is strongly dominated by the prior. Moreover, it is shown that the EnKF-N can be extended with a normal-inverse Wishart informative hyperprior that introduces additional information on error statistics. This can be identified as a hybrid EnKF–3D-Var counterpart to the EnKF-N.

Download Full-text

A Probabilistic Collocation-Based Kalman Filter for History Matching

SPE Journal ◽

10.2118/140737-pa ◽

2011 ◽

Vol 16 (02) ◽

pp. 294-306 ◽

Cited By ~ 22

Author(s):

Lingzao Zeng ◽

Haibin Chang ◽

Dongxiao Zhang

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

History Matching ◽

Initial Conditions ◽

Collocation Point ◽

Computational Effort ◽

Production Data ◽

Sampling Errors ◽

Probabilistic Collocation

Summary The ensemble Kalman filter (EnKF) has been used widely for data assimilation. Because the EnKF is a Monte Carlo-based method, a large ensemble size is required to reduce the sampling errors. In this study, a probabilistic collocation-based Kalman filter (PCKF) is developed to adjust the reservoir parameters to honor the production data. It combines the advantages of the EnKF for dynamic data assimilation and the polynomial chaos expansion (PCE) for efficient uncertainty quantification. In this approach, all the system parameters and states and the production data are approximated by the PCE. The PCE coefficients are solved with the probabilistic collocation method (PCM). Collocation realizations are constructed by choosing collocation point sets in the random space. The simulation for each collocation realization is solved forward in time independently by means of an existing deterministic solver, as in the EnKF method. In the analysis step, the needed covariance is approximated by the PCE coefficients. In this study, a square-root filter is employed to update the PCE coefficients. After the analysis, new collocation realizations are constructed. With the parameter collocation realizations as the inputs and the state collocation realizations as initial conditions, respectively, the simulations are forwarded to the next analysis step. Synthetic 2D water/oil examples are used to demonstrate the applicability of the PCKF in history matching. The results are compared with those from the EnKF on the basis of the same analysis. It is shown that the estimations provided by the PCKF are comparable to those obtained from the EnKF. The biggest improvement of the PCKF comes from the leading PCE approximation, with which the computational burden of the PCKF can be greatly reduced by means of a smaller number of simulation runs, and the PCKF outperforms the EnKF for a similar computational effort. When the correlation ratio is much smaller, the PCKF still provides estimations with a better accuracy for a small computational effort.

Download Full-text

Ensemble Kalman filtering without the intrinsic need for inflation

Nonlinear Processes in Geophysics ◽

10.5194/npg-18-735-2011 ◽

2011 ◽

Vol 18 (5) ◽

pp. 735-750 ◽

Cited By ~ 48

Author(s):

M. Bocquet

Keyword(s):

Kalman Filter ◽

Probability Density Function ◽

Probability Density ◽

Ensemble Kalman Filter ◽

Finite Size ◽

Ensemble Size ◽

Sampling Errors ◽

New Class ◽

Ensemble Transform Kalman Filter ◽

Ensemble Transform

Abstract. The main intrinsic source of error in the ensemble Kalman filter (EnKF) is sampling error. External sources of error, such as model error or deviations from Gaussianity, depend on the dynamical properties of the model. Sampling errors can lead to instability of the filter which, as a consequence, often requires inflation and localization. The goal of this article is to derive an ensemble Kalman filter which is less sensitive to sampling errors. A prior probability density function conditional on the forecast ensemble is derived using Bayesian principles. Even though this prior is built upon the assumption that the ensemble is Gaussian-distributed, it is different from the Gaussian probability density function defined by the empirical mean and the empirical error covariance matrix of the ensemble, which is implicitly used in traditional EnKFs. This new prior generates a new class of ensemble Kalman filters, called finite-size ensemble Kalman filter (EnKF-N). One deterministic variant, the finite-size ensemble transform Kalman filter (ETKF-N), is derived. It is tested on the Lorenz '63 and Lorenz '95 models. In this context, ETKF-N is shown to be stable without inflation for ensemble size greater than the model unstable subspace dimension, at the same numerical cost as the ensemble transform Kalman filter (ETKF). One variant of ETKF-N seems to systematically outperform the ETKF with optimally tuned inflation. However it is shown that ETKF-N does not account for all sampling errors, and necessitates localization like any EnKF, whenever the ensemble size is too small. In order to explore the need for inflation in this small ensemble size regime, a local version of the new class of filters is defined (LETKF-N) and tested on the Lorenz '95 toy model. Whatever the size of the ensemble, the filter is stable. Its performance without inflation is slightly inferior to that of LETKF with optimally tuned inflation for small interval between updates, and superior to LETKF with optimally tuned inflation for large time interval between updates.

Download Full-text

Pseudo-Orbit Data Assimilation. Part I: The Perfect Model Scenario

Journal of the Atmospheric Sciences ◽

10.1175/jas-d-13-032.1 ◽

2014 ◽

Vol 71 (2) ◽

pp. 469-482 ◽

Cited By ~ 11

Author(s):

Hailiang Du ◽

Leonard A. Smith

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

State Estimation ◽

Ensemble Kalman Filter ◽

Weather Forecasting ◽

Weather Prediction ◽

Attractive Alternative ◽

Perfect Model ◽

Improved Performance ◽

Orbit Data

Abstract State estimation lies at the heart of many meteorological tasks. Pseudo-orbit-based data assimilation provides an attractive alternative approach to data assimilation in nonlinear systems such as weather forecasting models. In the perfect model scenario, noisy observations prevent a precise estimate of the current state. In this setting, ensemble Kalman filter approaches are hampered by their foundational assumptions of dynamical linearity, while variational approaches may fail in practice owing to local minima in their cost function. The pseudo-orbit data assimilation approach improves state estimation by enhancing the balance between the information derived from the dynamic equations and that derived from the observations. The potential use of this approach for numerical weather prediction is explored in the perfect model scenario within two deterministic chaotic systems: the two-dimensional Ikeda map and 18-dimensional Lorenz96 flow. Empirical results demonstrate improved performance over that of the two most common traditional approaches of data assimilation (ensemble Kalman filter and four-dimensional variational assimilation).

Download Full-text

Preemptive Forecasts Using an Ensemble Kalman Filter

Monthly Weather Review ◽

10.1175/mwr3480.1 ◽

2007 ◽

Vol 135 (10) ◽

pp. 3484-3495 ◽

Cited By ~ 9

Author(s):

Brian J. Etherton

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Three Dimensional ◽

Forecast Model ◽

Observation Time ◽

Model Integration ◽

Model Forecast ◽

Error Statistics ◽

Background Error

Abstract An ensemble Kalman filter (EnKF) estimates the error statistics of a model forecast using an ensemble of model forecasts. One use of an EnKF is data assimilation, resulting in the creation of an increment to the first-guess field at the observation time. Another use of an EnKF is to propagate error statistics of a model forecast forward in time, such as is done for optimizing the location of adaptive observations. Combining these two uses of an ensemble Kalman filter, a “preemptive forecast” can be generated. In a preemptive forecast, the increment to the first-guess field is, using ensembles, propagated to some future time and added to the future control forecast, resulting in a new forecast. This new forecast requires no more time to produce than the time needed to run a data assimilation scheme, as no model integration is necessary. In an observing system simulation experiment (OSSE), a barotropic vorticity model was run to produce a 300-day “nature run.” The same model, run with a different vorticity forcing scheme, served as the forecast model. The model produced 24- and 48-h forecasts for each of the 300 days. The model was initialized every 24 h by assimilating observations of the nature run using a hybrid ensemble Kalman filter–three-dimensional variational data assimilation (3DVAR) scheme. In addition to the control forecast, a 64-member forecast ensemble was generated for each of the 300 days. Every 24 h, given a set of observations, the 64-member ensemble, and the control run, an EnKF was used to create 24-h preemptive forecasts. The preemptive forecasts were more accurate than the unmodified, original 48-h forecasts, though not quite as accurate as the 24-h forecast obtained from a new model integration initialized by assimilating the same observations as were used in the preemptive forecasts. The accuracy of the preemptive forecasts improved significantly when 1) the ensemble-based error statistics used by the EnKF were localized using a Schur product and 2) a model error term was included in the background error covariance matrices.

Download Full-text

Combining inflation-free and iterative ensemble Kalman filters for strongly nonlinear systems

Nonlinear Processes in Geophysics ◽

10.5194/npg-19-383-2012 ◽

2012 ◽

Vol 19 (3) ◽

pp. 383-399 ◽

Cited By ~ 44

Author(s):

M. Bocquet ◽

P. Sakov

Keyword(s):

Kalman Filter ◽

Nonlinear Systems ◽

Ensemble Kalman Filter ◽

Kalman Filters ◽

Finite Size ◽

Ensemble Kalman Filters ◽

Strongly Nonlinear ◽

Perfect Model ◽

Model Condition ◽

Better Than

Abstract. The finite-size ensemble Kalman filter (EnKF-N) is an ensemble Kalman filter (EnKF) which, in perfect model condition, does not require inflation because it partially accounts for the ensemble sampling errors. For the Lorenz '63 and '95 toy-models, it was so far shown to perform as well or better than the EnKF with an optimally tuned inflation. The iterative ensemble Kalman filter (IEnKF) is an EnKF which was shown to perform much better than the EnKF in strongly nonlinear conditions, such as with the Lorenz '63 and '95 models, at the cost of iteratively updating the trajectories of the ensemble members. This article aims at further exploring the two filters and at combining both into an EnKF that does not require inflation in perfect model condition, and which is as efficient as the IEnKF in very nonlinear conditions. In this study, EnKF-N is first introduced and a new implementation is developed. It decomposes EnKF-N into a cheap two-step algorithm that amounts to computing an optimal inflation factor. This offers a justification of the use of the inflation technique in the traditional EnKF and why it can often be efficient. Secondly, the IEnKF is introduced following a new implementation based on the Levenberg-Marquardt optimisation algorithm. Then, the two approaches are combined to obtain the finite-size iterative ensemble Kalman filter (IEnKF-N). Several numerical experiments are performed on IEnKF-N with the Lorenz '95 model. These experiments demonstrate its numerical efficiency as well as its performance that offer, at least, the best of both filters. We have also selected a demanding case based on the Lorenz '63 model that points to ways to improve the finite-size ensemble Kalman filters. Eventually, IEnKF-N could be seen as the first brick of an efficient ensemble Kalman smoother for strongly nonlinear systems.

Download Full-text

Tests of an Ensemble Kalman Filter for Mesoscale and Regional-Scale Data Assimilation. Part I: Perfect Model Experiments

Monthly Weather Review ◽

10.1175/mwr3101.1 ◽

2006 ◽

Vol 134 (2) ◽

pp. 722-736 ◽

Cited By ~ 100

Author(s):

Fuqing Zhang ◽

Zhiyong Meng ◽

Altug Aksoy

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Mesoscale Model ◽

Regional Scale ◽

Model Assumption ◽

Filter Performance ◽

Perfect Model ◽

Scale Data ◽

Surface Observations

Abstract Through observing system simulation experiments, this two-part study exploits the potential of using the ensemble Kalman filter (EnKF) for mesoscale and regional-scale data assimilation. Part I focuses on the performance of the EnKF under the perfect model assumption in which the truth simulation is produced with the same model and same initial uncertainties as those of the ensemble, while Part II explores the impacts of model error and ensemble initiation on the filter performance. In this first part, the EnKF is implemented in a nonhydrostatic mesoscale model [the fifth-generation Pennsylvania State University–NCAR Mesoscale Model (MM5)] to assimilate simulated sounding and surface observations derived from simulations of the “surprise” snowstorm of January 2000. This is an explosive East Coast cyclogenesis event with strong error growth at all scales as a result of interactions between convective-, meso-, and subsynoptic-scale dynamics. It is found that the EnKF is very effective in keeping the analysis close to the truth simulation under the perfect model assumption. The EnKF is most effective in reducing larger-scale errors but less effective in reducing errors at smaller, marginally resolvable scales. In the control experiment, in which the truth simulation was produced with the same model and same initial uncertainties as those of the ensemble, a 24-h continuous EnKF assimilation of sounding and surface observations of typical temporal and spatial resolutions is found to reduce the error by as much as 80% (compared to a 24-h forecast without data assimilation) for both observed and unobserved variables including zonal and meridional winds, temperature, and pressure. However, it is observed to be relatively less efficient in correcting errors in the vertical velocity and moisture fields, which have stronger smaller-scale components. The analysis domain-averaged root-mean-square error after 24-h assimilation is ∼1.0–1.5 m s−1 for winds and ∼1.0 K for temperature, which is comparable to or less than typical observational errors. Various sensitivity experiments demonstrated that the EnKF is quite successful in all realistic observational scenarios tested. However, as will be presented in Part II, the EnKF performance may be significantly degraded if an imperfect forecast model is used, as is likely the case when real observations are assimilated.

Download Full-text