Estimating model error covariance matrix parameters in extended Kalman filtering

Abstract. The extended Kalman filter (EKF) is a popular state estimation method for nonlinear dynamical models. The model error covariance matrix is often seen as a tuning parameter in EKF, which is often simply postulated by the user. In this paper, we study the filter likelihood technique for estimating the parameters of the model error covariance matrix. The approach is based on computing the likelihood of the covariance matrix parameters using the filtering output. We show that (a) the importance of the model error covariance matrix calibration depends on the quality of the observations, and that (b) the estimation approach yields a well-tuned EKF in terms of the accuracy of the state estimates and model predictions. For our numerical experiments, we use the two-layer quasi-geostrophic model that is often used as a benchmark model for numerical weather prediction.

Download Full-text

Ozone-sensitive channel selection over IASI full spectrum with correlated observation errors for NWP

10.5194/amt-2019-242 ◽

2019 ◽

Author(s):

Olivier Coopmann ◽

Vincent Guidard ◽

Béatrice Josse ◽

Virginie Marécal ◽

Nadia Fourrié

Keyword(s):

Covariance Matrix ◽

Weather Prediction ◽

Channel Selection ◽

Spectral Interval ◽

Observation Error ◽

Full Spectrum ◽

Error Covariance Matrix ◽

Error Covariance ◽

One Year ◽

Selection Of

Abstract. The Infrared Atmospheric Sounding Interferometer (IASI) onboard the Metop satellites provides 8461 channels in the infrared spectrum, covering the spectral interval 645–2760 cm−1 at a resolution of 0.5 cm−1. The high volume of data observation resulting from IASI presents many challenges. In current Numerical Weather Prediction (NWP) models, assimilating all channels is not feasible, due to data transmission, data storage and significant computational costs. One of the methods for reducing the data volume is the channel selection. Many NWP centres use a subset of 314 IASI channels including 15 ozone-sensitive channels. However, this channel selection has been carried out assuming uncorrelated observation errors. In addition, these ozone-sensitive channels have been selected only for ozone information. The objective of this study is to carry out a new selection of IASI ozone-sensitive channels from the full spectrum over a spectral range of 1000–1070 cm−1, in a direct radiance assimilation framework. This selection is done with a full observation error covariance matrix to take into account cross-channel error correlations. A sensitivity method based on the channel spectral sensitivity to variables and a statistical approach based on the Degrees of Freedom for Signal (DFS) have been chosen. To be representative of atmospheric variability, 345 profiles from around the world over a one-year period were selected. The new selection, is evaluated in a One-Dimensional Variational (1D-Var) analyses framework. This selection highlights a new set of 15 IASI ozone-sensitive channels. The results are very encouraging since by adding these 15 channels to 122 operational channels, temperature and humidity analyses are improved by 13.8 % and 20.9 % respectively. Obviously, these 15 channels significantly improve ozone analyses. In addition to considering inter-channel observation error correlations, the channel selection method uses a robust background error covariance matrix that takes into account temperature, humidity and ozone errors using a lagged forecast method over a one-year period. The new selection of IASI ozone-sensitive channels will be soon used in the global 4D-Var ARPEGE (Action de Recherche Petite Echelle Grande Echelle) data assimilation system.

Download Full-text

A methodology to obtain model-error covariances due to the discretization scheme from the parametric Kalman filter perspective

Nonlinear Processes in Geophysics ◽

10.5194/npg-28-1-2021 ◽

2021 ◽

Vol 28 (1) ◽

pp. 1-22

Author(s):

Olivier Pannekoucke ◽

Richard Ménard ◽

Mohammad El Aabaribaoune ◽

Matthieu Plu

Keyword(s):

Kalman Filter ◽

Numerical Model ◽

Covariance Matrix ◽

Model Error ◽

Filter Method ◽

Advection Equation ◽

Parametric Modelling ◽

Test Bed ◽

Error Covariance Matrix ◽

Error Covariance

Abstract. This contribution addresses the characterization of the model-error covariance matrix from the new theoretical perspective provided by the parametric Kalman filter method which approximates the covariance dynamics from the parametric evolution of a covariance model. The classical approach to obtain the modified equation of a dynamics is revisited to formulate a parametric modelling of the model-error covariance matrix which applies when the numerical model is dissipative compared with the true dynamics. As an illustration, the particular case of the advection equation is considered as a simple test bed. After the theoretical derivation of the predictability-error covariance matrices of both the nature and the numerical model, a numerical simulation is proposed which illustrates the properties of the resulting model-error covariance matrix.

Download Full-text

Balance and Ensemble Kalman Filter Localization Techniques

Monthly Weather Review ◽

10.1175/2010mwr3328.1 ◽

2011 ◽

Vol 139 (2) ◽

pp. 511-522 ◽

Cited By ~ 136

Author(s):

Steven J. Greybush ◽

Eugenia Kalnay ◽

Takemasa Miyoshi ◽

Kayo Ide ◽

Brian R. Hunt

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Covariance Matrix ◽

Ensemble Kalman Filter ◽

Weather Prediction ◽

Length Scale ◽

Observation Error ◽

Long Distance ◽

Error Covariance Matrix ◽

Error Covariance

Abstract In ensemble Kalman filter (EnKF) data assimilation, localization modifies the error covariance matrices to suppress the influence of distant observations, removing spurious long-distance correlations. In addition to allowing efficient parallel implementation, this takes advantage of the atmosphere’s lower dimensionality in local regions. There are two primary methods for localization. In B localization, the background error covariance matrix elements are reduced by a Schur product so that correlations between grid points that are far apart are removed. In R localization, the observation error covariance matrix is multiplied by a distance-dependent function, so that far away observations are considered to have infinite error. Successful numerical weather prediction depends upon well-balanced initial conditions to avoid spurious propagation of inertial-gravity waves. Previous studies note that B localization can disrupt the relationship between the height gradient and the wind speed of the analysis increments, resulting in an analysis that can be significantly ageostrophic. This study begins with a comparison of the accuracy and geostrophic balance of EnKF analyses using no localization, B localization, and R localization with simple one-dimensional balanced waves derived from the shallow-water equations, indicating that the optimal length scale for R localization is shorter than for B localization, and that for the same length scale R localization is more balanced. The comparison of localization techniques is then expanded to the Simplified Parameterizations, Primitive Equation Dynamics (SPEEDY) global atmospheric model. Here, natural imbalance of the slow manifold must be contrasted with undesired imbalance introduced by data assimilation. Performance of the two techniques is comparable, also with a shorter optimal localization distance for R localization than for B localization.

Download Full-text

Brief communication: An innovation-based estimation method for model error covariance in Kalman filters

10.5194/npg-2021-35 ◽

2021 ◽

Author(s):

Eviatar Bach ◽

Michael Ghil

Keyword(s):

Kalman Filter ◽

Covariance Matrix ◽

Ensemble Kalman Filter ◽

Kalman Filters ◽

Estimation Method ◽

Model Error ◽

High Accuracy ◽

Covariance Estimation ◽

Observation Error ◽

Error Covariance

Abstract. We present a simple innovation-based model error covariance estimation method for Kalman filters. The method is based on Berry and Sauer (2013) and the simplification results from assuming known observation error covariance. We carry out experiments with a prescribed model error covariance using a Lorenz (1996) model and ensemble Kalman filter. The prescribed error covariance matrix is recovered with high accuracy.

Download Full-text

Estimation of the error covariance matrix for IASI radiances and its impact on ozone analyses

10.5194/amt-2020-179 ◽

2020 ◽

Author(s):

Mohammad El Aabaribaoune ◽

Emanuele Emili ◽

Vincent Guidard

Keyword(s):

Covariance Matrix ◽

Atmospheric Chemistry ◽

Transport Model ◽

Weather Prediction ◽

Computational Cost ◽

Observation Error ◽

Chemical Transport Model ◽

Error Covariance Matrix ◽

Error Covariance ◽

Estimated Error

Abstract. In atmospheric chemistry retrievals and data assimilation systems, observation errors associated with satellite radiances are chosen empirically and generally treated as uncorrelated. In this work, we estimate inter-channel error covariances for the Infrared Atmospheric Sounding Interferometer (IASI) and evaluate their impact on ozone assimilation with the chemical transport model MOCAGE (MOdèle de Chime Atmospheric à Grand Echelle). The method used to calculate observation errors is a diagnostic based on the observation and analysis residual statistics already adopted in numerical weather prediction centers. We used a subset of 280 channels covering the spectral range between 980 and 1100 cm−1 to estimate the observation error covariance matrix. We computed hourly 3D-Var analyses and compared the resulting O3 fields against ozonesondes and the measurements provided by the Microwave Limb Sounder (MLS). The results show significant differences between using the estimated error covariance matrix with respect to the empirical diagonal matrix employed in previous studies. The validation of the analyses against independent data reports a significant improvement especially in the tropical stratosphere. The computational cost has also been reduced when the estimated covariance is employed in the assimilation system.

Download Full-text

Estimation of the error covariance matrix for IASI radiances and its impact on the assimilation of ozone in a chemistry transport model

Atmospheric Measurement Techniques ◽

10.5194/amt-14-2841-2021 ◽

2021 ◽

Vol 14 (4) ◽

pp. 2841-2856

Author(s):

Mohammad El Aabaribaoune ◽

Emanuele Emili ◽

Vincent Guidard

Keyword(s):

Covariance Matrix ◽

Spectral Range ◽

Atmospheric Chemistry ◽

Transport Model ◽

Weather Prediction ◽

Computational Cost ◽

Observation Error ◽

Chemistry Transport Model ◽

Error Covariance Matrix ◽

Error Covariance

Abstract. In atmospheric chemistry retrievals and data assimilation systems, observation errors associated with satellite radiances are chosen empirically and generally treated as uncorrelated. In this work, we estimate inter-channel error covariances for the Infrared Atmospheric Sounding Interferometer (IASI) and evaluate their impact on ozone assimilation with the chemistry transport model MOCAGE (Modèle de Chimie Atmosphérique à Grande Echelle). The method used to calculate observation errors is a diagnostic based on the observation and analysis residual statistics already adopted in many numerical weather prediction centres. We used a subset of 280 channels covering the spectral range between 980 and 1100 cm−1 to estimate the observation-error covariance matrix. This spectral range includes ozone-sensitive and atmospheric window channels. We computed hourly 3D-Var analyses and compared the resulting O3 fields against ozonesondes and the measurements provided by the Microwave Limb Sounder (MLS) and by the Ozone Monitoring Instrument (OMI). The results show significant differences between using the estimated error covariance matrix with respect to the empirical diagonal matrix employed in previous studies. The validation of the analyses against independent data reports a significant improvement, especially in the tropical stratosphere. The computational cost has also been reduced when the estimated covariance matrix is employed in the assimilation system, by reducing the number of iterations needed for the minimizer to converge.

Download Full-text

A methodology to obtain model-error covariances due to the discretization scheme from the parametric Kalman filter perspective

10.5194/npg-2020-14 ◽

2020 ◽

Author(s):

Olivier Pannekoucke ◽

Richard Ménard ◽

Mohammad El Aabaribaoune ◽

Matthieu Plu

Keyword(s):

Kalman Filter ◽

Covariance Matrix ◽

Forecast Error ◽

Theoretical Perspective ◽

Model Error ◽

Filter Method ◽

Advection Equation ◽

Test Bed ◽

Error Covariance Matrix ◽

Error Covariance

Abstract. This contribution adresses the characterization of the model-error covariance matrix from the new theoretical perspective provided by the parametric Kalman filter method which approximates the covariance dynamics from the parametric evolution of a covariance model. The classical approach to obtain the modified equation of a dynamics is revisited to formulate a parametric diagnosis of the model-error covariance matrix. As an illustration, the particular case of the advection equation is considered as a simple test bed. After the theoretical derivation of both the forecast-error and the predictability-error covariance matrices, a numerical simulation is proposed which demonstrates the skill of the parametric methodology in reproducing the model-error covariance matrix information.

Download Full-text

Including the spatial observation error correlation in data assimilation of AMSU-A radiances

10.5194/egusphere-egu21-13672 ◽

2021 ◽

Author(s):

Koji Terasaki ◽

Takemasa Miyoshi

Keyword(s):

Data Assimilation ◽

Covariance Matrix ◽

Weather Prediction ◽

Computation Time ◽

Horizontal Resolution ◽

Observation Error ◽

Correlated Errors ◽

Error Covariance Matrix ◽

Error Covariance ◽

Error Correlation

<p>Recent developments in sensing technology increased the number of observations both in space and time. It is essential to effectively utilize the information from observations to improve numerical weather prediction (NWP). It is known to have correlated errors in observations measured with a single instrument, such as satellite radiances. The observations with the horizontal error correlation are usually thinned to compensate for neglecting the error correlation in data assimilation. This study explores to explicitly include the horizontal observation error correlation of Advanced Microwave Sounding Unit-A (AMSU-A) radiances using a global atmospheric data assimilation system NICAM-LETKF, which comprises the Nonhydrostatic ICosahedral Atmospheric Model (NICAM) and the Local Ensemble Transform Kalman Filter (LETKF). This study performs the data assimilation experiments at 112-km horizontal resolution and 38 vertical layers up to 40 km and with 32 ensemble members.</p><p>In this study, we estimate the horizontal observation error correlation of AMSU-A radiances using innovation statistics. The computation cost of inverting the observation error covariance matrix will increase when non-zero off-diagonal terms are included. In this study, we assume uncorrelated observation errors between different instruments and observation variables, so that the observation error covariance matrix becomes block diagonal with only horizontal error correlations included. The computation time of the entire LETKF analysis procedure is increased only by up to 10 % compared with the case using the diagonal observation error covariance matrix. The analyses and forecasts of temperature and zonal wind in the mid- and upper-troposphere are improved by including the horizontal error correlations. We will present the most recent results at the workshop.</p>

Download Full-text

A Preliminary Study on Dimension-Reduction Algorithm for Variational Methods in Three Dimensions

10.20944/preprints201801.0293.v1 ◽

2018 ◽

Author(s):

Xuan Chen

Keyword(s):

Variational Methods ◽

Cost Function ◽

Dimension Reduction ◽

Covariance Matrix ◽

Weather Prediction ◽

Three Dimensions ◽

Matrix Analysis ◽

Reduction Algorithm ◽

Error Covariance Matrix ◽

Error Covariance

Numerical weather prediction is an initial-value problem, for determination of the initial conditions, there are many methods and one of the most classical methods is variational methods in three dimensions, or 3D-Var. In this approach, with a defined cost function proportional to the square of the distance between the analysis and both the background and the observations, one can obtain the analysis. In the cost function, the background and the observations are reshaped to vectors; within this step, the order of the background error covariance matrix and the observational error covariance matrix becomes huge, which is not convenient to one to obtain the analysis. In this paper, according to the matrix analysis approach, we put forward some possible improvements to the dimension-reduction algorithm of 3D-Var, so that provide some references for data assimilation.

Download Full-text