Insensitivety to initial condition/prior in data assimilation for the case of the optimal filter and deterministic model

<p>Data assimilation is a term used to describe efforts to improve our knowledge<br>of a system by combining incomplete observations with imperfect models.<br>This is more generally known as filtering, which is ’optimal’ estimation of<br>the state of a system as it evolves over time, in the mean square sense. In<br>a Bayesian framework, the optimal filter is therefore naturally a sequence of<br>conditional probabilities of a signal given the observations and can be up-<br>dated recursively with new observations with Bayes’ formula. When, the<br>dynamics and observations errors are linear, this is equivalent to the Kalman<br>filter. In the nonlinear case, deriving an explicit form for the posterior dis-<br>tribution is in general not possible.<br>One of the important difficulties with applying the nonlinear filter in practice<br>is that the initial condition, the prior, is required to initialise the filtering.<br>However we are unlikely to know the correct initial distribution accurately<br>or at all. A filter is called stable if it is insensitive with respect to the<br>prior, that is, it converges to the same distribution, regardless of the initial<br>condition.<br>A body of work exists showing stability of the filter which rely on the stochas-<br>ticity of the underlying dynamics. In contrast, we show stability of the op-<br>timal filter for a class of nonlinear and deterministic dynamical systems and<br>our result relies on the intrinsic chaotic properties of the dynamics. We build<br>on the considerable knowledge that exists on the existence of SRB measures<br>in uniformly hyperbolic dynamical systems and we view the conditional prob-<br>abilities as SRB measures ‘conditional on the observation’ which are shown<br>to be absolutely continuous along the unstable manifold. This is in line with<br>the result of Bouquet, Carrassi et al [1] regarding data assimilation in the<br>“unstable subspace”, where they show stability of the filter if the unstable<br>and neutral subspaces are uniformly observed.</p><p>[1] M. Bocquet et al. “Degenerate Kalman Filter Error Covariances and<br>Their Convergence onto the Unstable Subspace”. In: SIAM/ASA Jour-<br>nal on Uncertainty Quantification 5.1 (2017), pp. 304–333. url: https:<br>//doi.org/10.1137/16M1068712.</p>

CHAOS AND WEATHER FORECASTING: THE ROLE OF THE UNSTABLE SUBSPACE IN PREDICTABILITY AND STATE ESTIMATION PROBLEMS

In the first part of this paper, we review some important results on atmospheric predictability, from the pioneering work of Lorenz to recent results with operational forecasting models. Particular relevance is given to the connection between atmospheric predictability and the theory of Lyapunov exponents and vectors. In the second part, we briefly review the foundations of data assimilation methods and then we discuss recent results regarding the application of the tools typical of chaotic systems theory described in the first part to well established data assimilation algorithms, the Extended Kalman Filter (EKF) and Four Dimensional Variational Assimilation (4DVar). In particular, the Assimilation in the Unstable Space (AUS), specifically developed for application to chaotic systems, is described in detail.

On the Kalman Filter error covariance collapse into the unstable subspace

When the Extended Kalman Filter is applied to a chaotic system, the rank of the error covariance matrices, after a sufficiently large number of iterations, reduces to N+ + N0 where N+ and N0 are the number of positive and null Lyapunov exponents. This is due to the collapse into the unstable and neutral tangent subspace of the solution of the full Extended Kalman Filter. Therefore the solution is the same as the solution obtained by confining the assimilation to the space spanned by the Lyapunov vectors with non-negative Lyapunov exponents. Theoretical arguments and numerical verification are provided to show that the asymptotic state and covariance estimates of the full EKF and of its reduced form, with assimilation in the unstable and neutral subspace (EKF-AUS) are the same. The consequences of these findings on applications of Kalman type Filters to chaotic models are discussed.

Controlling instabilities along a 3DVar analysis cycle by assimilating in the unstable subspace: a comparison with the EnKF

A hybrid scheme obtained by combining 3DVar with the Assimilation in the Unstable Subspace (3DVar-AUS) is tested in a QG model, under perfect model conditions, with a fixed observational network, with and without observational noise. The AUS scheme, originally formulated to assimilate adaptive observations, is used here to assimilate the fixed observations that are found in the region of local maxima of BDAS vectors (Bred vectors subject to assimilation), while the remaining observations are assimilated by 3DVar. The performance of the hybrid scheme is compared with that of 3DVar and of an EnKF. The improvement gained by 3DVar-AUS and the EnKF with respect to 3DVar alone is similar in the present model and observational configuration, while 3DVar-AUS outperforms the EnKF during the forecast stage. The 3DVar-AUS algorithm is easy to implement and the results obtained in the idealized conditions of this study encourage further investigation toward an implementation in more realistic contexts.