SymPKF (v1.0): a symbolic and computational toolbox for the design of parametric Kalman filter dynamics

Abstract. Recent research in data assimilation has led to the introduction of the parametric Kalman filter (PKF): an implementation of the Kalman filter, whereby the covariance matrices are approximated by a parameterized covariance model. In the PKF, the dynamics of the covariance during the forecast step rely on the prediction of the covariance parameters. Hence, the design of the parameter dynamics is crucial, while it can be tedious to do this by hand. This contribution introduces a Python package, SymPKF, able to compute PKF dynamics for univariate statistics and when the covariance model is parameterized from the variance and the local anisotropy of the correlations. The ability of SymPKF to produce the PKF dynamics is shown on a nonlinear diffusive advection (the Burgers equation) over a 1D domain and the linear advection over a 2D domain. The computation of the PKF dynamics is performed at a symbolic level, but an automatic code generator is also introduced to perform numerical simulations. A final multivariate example illustrates the potential of SymPKF to go beyond the univariate case.

Download Full-text

SymPKF (v1.0): a symbolic and computational toolbox for the design of parametric Kalman ﬁlter dynamics

10.5194/gmd-2021-89 ◽

2021 ◽

Author(s):

Olivier Pannekoucke ◽

Philippe Arbogast

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Burgers Equation ◽

Covariance Matrices ◽

Covariance Model ◽

Local Anisotropy ◽

Code Generator ◽

Univariate Statistics ◽

Automatic Code ◽

Python Package

Abstract. Recent researches in data assimilation lead to the introduction of the parametric Kalman filter (PKF): an implementation of the Kalman filter, where the covariance matrices are approximated by a parameterized covariance model. In the PKF, the dynamics of the covariance during the forecast step relies on the prediction of the covariance parameters. Hence, the design of the parameter dynamics is crucial while it can be tedious to do this by hand. This contribution introduces a python package, SymPKF, able to compute PKF dynamics for univariate statistics and when the covariance model is parameterized from the variance and the local anisotropy of the correlations. The ability of SymPKF to produce the PKF dynamics is shown on a non-linear diffusive advection (Burgers equation) over a 1D domain and the linear advection over a 2D domain. The computation of the PKF dynamics is performed at a symbolic level, but an automatic code generator is also introduced to perform numerical simulations. A final multivariate example illustrates the potential of SymPKF to go beyond the univariate case.

Download Full-text

Biogeochemical Oceanographic Data Assimilation: Dimensionality Reduced Kalman Filter For Mediterranean Sea Forecasting

10.14293/p2199-8442.1.sop-math.vgditb.v1 ◽

2018 ◽

Author(s):

Simone K. Spada ◽

Gianpiero Cossarini ◽

Stefano Salon ◽

Stefano Maset

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Mediterranean Sea ◽

Computational Efficiency ◽

Transport Model ◽

Computational Cost ◽

Principal Component ◽

Covariance Matrices ◽

Background Error ◽

Biological Variables

Data assimilation is a key element to improve the performance of biogeochemical ocean/marine forecasting systems. Handling the very big dimension of the state vector of the system (often of the order of 10 6 ) remains an issue, also considering the computational efficiency of operational systems. Indeed, simple product operations involving the covariance matrices are too heavy to be computed for operational forecasting purposes. Various attempts have been made in literature to reduce the complexity of this task, often adding strong hypotheses to simplify the problem and decrease the computational cost. The MedBFM model system, which is responsible for monitoring and forecasting the biogeochemical state of the Mediterranean Sea within the European Copernicus Marine Services (see http://marine.copernicus.eu/ ) assimilates surface chlorophyll data through a 3D Variational algorithm, that decomposes the background error covariance matrix into sequential operators to reduce complexity. In this work, we developed a novel Kalman Filter for the MedBFM system. The novel Kalman Filter scheme starts from a SEIK approach but benefits from advanced Principal Component Analysis to reduce the dimension of covariance matrices and improve the computational efficiency. We compared the standard SEIK filter and the new Kalman filter implementations in a one dimensional transport model with 2 biological variables in terms of root mean square distance. In the vast majority of the experiments, the new Kalman filter had better performances.

Download Full-text

Uncertainties of Terrestrial Water Storage Anomalies for Global Basins – A Comparison Between Modelled and Formal Covariances

10.5194/gstm2020-24 ◽

2020 ◽

Author(s):

Eva Boergens ◽

Andreas Kvas ◽

Henryk Dobslaw ◽

Annette Eicker ◽

Christoph Dahle ◽

...

Keyword(s):

Data Assimilation ◽

Spherical Harmonic ◽

Water Storage ◽

Covariance Matrices ◽

Model Parameters ◽

Data Sets ◽

Terrestrial Water Storage ◽

Covariance Model ◽

Terrestrial Water ◽

Level 3

The application of GRACE and GRACE-FO observed gridded terrestrial water storage data (TWS) often requires realistic assumptions of the data variances and covariances. Such covariances are, e.g., needed for data assimilation in various models or combinations with other data sets. The formal variance-covariance matrices now provided with the Stokes coefficients can yield such spatial variances and covariances after variance propagating them through the various post-processing steps, including the filtering, and spherical harmonic synthesis. However, a rigorous variance propagation to the TWS grids is beyond the capabilities of most non-geodetic users. That is why we developed a new spatial covariance model for global TWS grids. This covariance model is non-stationary (time-depending), non-homogeneous (location-depending), and anisotropic (direction-depending). Additionally, it allows latitudinal wave-like correlations caused by residual striping errors. The model is tested for both GFZ RL06 Level-3 TWS data as provided via the GravIS portal (gravis.gfz-potsdam.de) and ITSG-Grace2018 GravIS-like processed Level-3 TWS data. The model parameters are fitted to empirical correlations derived from both TWS fields. Both data sets yield the same model parameters within the uncertainty of the parameter estimation. Now, the covariance model derived thereof can be used to estimate uncertainties of mean TWS time series of arbitrary regions such as river basins. Here, we use a global basin segmentation covering all continents. At the same time, such regional uncertainties can be derived from formal variance-covariance matrices as well. To this end, the formal ITSG-Grace2018 variance-covariance matrices of the spherical harmonic coefficients are used. Thus, the modelled and formal basin uncertainties can be compared against each other globally, both spatially and temporally. Further, external validation investigates the usefulness of the basin uncertainties for applications such as data assimilation into hydrological models. Our results show a high agreement between the modelled and the formal basin uncertainties proving our approach of modelled covariance to be a suitable surrogate for the formal variance-covariance matrices.

Download Full-text

Parametric covariance dynamics for the nonlinear diffusive Burgers equation

Nonlinear Processes in Geophysics ◽

10.5194/npg-25-481-2018 ◽

2018 ◽

Vol 25 (3) ◽

pp. 481-495 ◽

Cited By ~ 3

Author(s):

Olivier Pannekoucke ◽

Marc Bocquet ◽

Richard Ménard

Keyword(s):

Kalman Filter ◽

Burgers Equation ◽

Diffusion Tensor ◽

Error Variance ◽

Covariance Model ◽

Computationally Efficient ◽

Closure Model ◽

Error Covariance ◽

Efficient Alternative ◽

Linear Covariance

Abstract. The parametric Kalman filter (PKF) is a computationally efficient alternative method to the ensemble Kalman filter. The PKF relies on an approximation of the error covariance matrix by a covariance model with a space–time evolving set of parameters. This study extends the PKF to nonlinear dynamics using the diffusive Burgers equation as an application, focusing on the forecast step of the assimilation cycle. The covariance model considered is based on the diffusion equation, with the diffusion tensor and the error variance as evolving parameters. An analytical derivation of the parameter dynamics highlights a closure issue. Therefore, a closure model is proposed based on the kurtosis of the local correlation functions. Numerical experiments compare the PKF forecast with the statistics obtained from a large ensemble of nonlinear forecasts. These experiments strengthen the closure model and demonstrate the ability of the PKF to reproduce the tangent linear covariance dynamics, at a low numerical cost.

Download Full-text

A Nonvariational Consistent Hybrid Ensemble Filter

Monthly Weather Review ◽

10.1175/mwr-d-14-00391.1 ◽

2015 ◽

Vol 143 (12) ◽

pp. 5073-5090 ◽

Cited By ~ 8

Author(s):

Craig H. Bishop ◽

Bo Huang ◽

Xuguang Wang

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Forecast Error ◽

Model Space ◽

High Rank ◽

Covariance Model ◽

Ensemble Data Assimilation ◽

Error Covariance ◽

Ensemble Data ◽

Ensemble Transform Kalman Filter

Abstract A consistent hybrid ensemble filter (CHEF) for using hybrid forecast error covariance matrices that linearly combine aspects of both climatological and flow-dependent matrices within a nonvariational ensemble data assimilation scheme is described. The CHEF accommodates the ensemble data assimilation enhancements of (i) model space ensemble covariance localization for satellite data assimilation and (ii) Hodyss’s method for improving accuracy using ensemble skewness. Like the local ensemble transform Kalman filter (LETKF), the CHEF is computationally scalable because it updates local patches of the atmosphere independently of others. Like the sequential ensemble Kalman filter (EnKF), it serially assimilates batches of observations and uses perturbed observations to create ensembles of analyses. It differs from the deterministic (no perturbed observations) ensemble square root filter (ESRF) and the EnKF in that (i) its analysis correction is unaffected by the order in which observations are assimilated even when localization is required, (ii) it uses accurate high-rank solutions for the posterior error covariance matrix to serially assimilate observations, and (iii) it accommodates high-rank hybrid error covariance models. Experiments were performed to assess the effect on CHEF and ESRF analysis accuracy of these differences. In the case where both the CHEF and the ESRF used tuned localized ensemble covariances for the forecast error covariance model, the CHEF’s advantage over the ESRF increased with observational density. In the case where the CHEF used a hybrid error covariance model but the ESRF did not, the CHEF had a substantial advantage for all observational densities.

Download Full-text

Parametric covariance dynamics for the nonlinear diffusive Burgers' equation

10.5194/npg-2018-10 ◽

2018 ◽

Author(s):

Olivier Pannekoucke ◽

Marc Bocquet ◽

Richard Ménard

Keyword(s):

Kalman Filter ◽

Burgers Equation ◽

Diffusion Tensor ◽

Error Variance ◽

Covariance Model ◽

Computationally Efficient ◽

Closure Model ◽

Error Covariance ◽

Efficient Alternative ◽

Linear Covariance

Abstract. The parametric Kalman filter (PKF) is a computationally efficient alternative method to the ensemble Kalman filter (EnKF). The PKF relies on an approximation of the error covariance matrix by a covariance model with space-time evolving set of parameters. This study extends the PKF to nonlinear dynamics using the diffusive Burgers' equation as an application, focusing on the forecast step of the assimilation cycle. The covariance model considered is based on the diffusion equation, with the diffusion tensor and the error variance as evolving parameter. An analytical derivation of the parameter dynamics highlights a closure issue. Therefore, a closure model is proposed based on the so-called kurtosis of the local correlation functions. Numerical experiments compare the PKF forecast with the statistics obtained from an large ensemble of nonlinear forecasts. These experiments strengthen the closure model and demonstrate the ability of the PKF to reproduce the tangent-linear covariance dynamics, at a low numerical cost.

Download Full-text