A Nonvariational Consistent Hybrid Ensemble Filter

Abstract A consistent hybrid ensemble filter (CHEF) for using hybrid forecast error covariance matrices that linearly combine aspects of both climatological and flow-dependent matrices within a nonvariational ensemble data assimilation scheme is described. The CHEF accommodates the ensemble data assimilation enhancements of (i) model space ensemble covariance localization for satellite data assimilation and (ii) Hodyss’s method for improving accuracy using ensemble skewness. Like the local ensemble transform Kalman filter (LETKF), the CHEF is computationally scalable because it updates local patches of the atmosphere independently of others. Like the sequential ensemble Kalman filter (EnKF), it serially assimilates batches of observations and uses perturbed observations to create ensembles of analyses. It differs from the deterministic (no perturbed observations) ensemble square root filter (ESRF) and the EnKF in that (i) its analysis correction is unaffected by the order in which observations are assimilated even when localization is required, (ii) it uses accurate high-rank solutions for the posterior error covariance matrix to serially assimilate observations, and (iii) it accommodates high-rank hybrid error covariance models. Experiments were performed to assess the effect on CHEF and ESRF analysis accuracy of these differences. In the case where both the CHEF and the ESRF used tuned localized ensemble covariances for the forecast error covariance model, the CHEF’s advantage over the ESRF increased with observational density. In the case where the CHEF used a hybrid error covariance model but the ESRF did not, the CHEF had a substantial advantage for all observational densities.

Download Full-text

Observation-Informed Generalized Hybrid Error Covariance Models

Monthly Weather Review ◽

10.1175/mwr-d-18-0016.1 ◽

2018 ◽

Vol 146 (11) ◽

pp. 3605-3622 ◽

Cited By ~ 2

Author(s):

Elizabeth A. Satterfield ◽

Daniel Hodyss ◽

David D. Kuhl ◽

Craig H. Bishop

Keyword(s):

Data Assimilation ◽

Covariance Matrix ◽

Sample Variance ◽

Calibration Function ◽

Covariance Model ◽

True Error ◽

Ensemble Data Assimilation ◽

Error Covariance ◽

Ensemble Data ◽

Prior Variance

Abstract Because of imperfections in ensemble data assimilation schemes, one cannot assume that the ensemble-derived covariance matrix is equal to the true error covariance matrix. Here, we describe a simple and intuitively compelling method to fit calibration functions of the ensemble sample variance to the mean of the distribution of true error variances, given an ensemble estimate. We demonstrate that the use of such calibration functions is consistent with theory showing that, when sampling error in the prior variance estimate is considered, the gain that minimizes the posterior error variance uses the expected true prior variance, given an ensemble sample variance. Once the calibration function has been fitted, it can be combined with ensemble-based and climatologically based error correlation information to obtain a generalized hybrid error covariance model. When the calibration function is chosen to be a linear function of the ensemble variance, the generalized hybrid error covariance model is the widely used linear hybrid consisting of a weighted sum of a climatological and an ensemble-based forecast error covariance matrix. However, when the calibration function is chosen to be, say, a cubic function of the ensemble sample variance, the generalized hybrid error covariance model is a nonlinear function of the ensemble estimate. We consider idealized univariate data assimilation and multivariate cycling ensemble data assimilation to demonstrate that the generalized hybrid error covariance model closely approximates the optimal weights found through computationally expensive tuning in the linear case and, in the nonlinear case, outperforms any plausible linear model.

Download Full-text

Maximum Likelihood Ensemble Filter: Theoretical Aspects

Monthly Weather Review ◽

10.1175/mwr2946.1 ◽

2005 ◽

Vol 133 (6) ◽

pp. 1710-1726 ◽

Cited By ~ 222

Author(s):

Milija Zupanski

Keyword(s):

Maximum Likelihood ◽

Data Assimilation ◽

Cost Function ◽

Ensemble Data Assimilation ◽

Error Covariance ◽

Ensemble Data ◽

Analysis Error ◽

Nonlinear Observation ◽

The Cost ◽

Maximum Likelihood Ensemble Filter

Abstract A new ensemble-based data assimilation method, named the maximum likelihood ensemble filter (MLEF), is presented. The analysis solution maximizes the likelihood of the posterior probability distribution, obtained by minimization of a cost function that depends on a general nonlinear observation operator. The MLEF belongs to the class of deterministic ensemble filters, since no perturbed observations are employed. As in variational and ensemble data assimilation methods, the cost function is derived using a Gaussian probability density function framework. Like other ensemble data assimilation algorithms, the MLEF produces an estimate of the analysis uncertainty (e.g., analysis error covariance). In addition to the common use of ensembles in calculation of the forecast error covariance, the ensembles in MLEF are exploited to efficiently calculate the Hessian preconditioning and the gradient of the cost function. A sufficient number of iterative minimization steps is 2–3, because of superior Hessian preconditioning. The MLEF method is well suited for use with highly nonlinear observation operators, for a small additional computational cost of minimization. The consistent treatment of nonlinear observation operators through optimization is an advantage of the MLEF over other ensemble data assimilation algorithms. The cost of MLEF is comparable to the cost of existing ensemble Kalman filter algorithms. The method is directly applicable to most complex forecast models and observation operators. In this paper, the MLEF method is applied to data assimilation with the one-dimensional Korteweg–de Vries–Burgers equation. The tested observation operator is quadratic, in order to make the assimilation problem more challenging. The results illustrate the stability of the MLEF performance, as well as the benefit of the cost function minimization. The improvement is noted in terms of the rms error, as well as the analysis error covariance. The statistics of innovation vectors (observation minus forecast) also indicate a stable performance of the MLEF algorithm. Additional experiments suggest the amplified benefit of targeted observations in ensemble data assimilation.

Download Full-text

Convection-Permitting Forecasts Initialized with Continuously Cycling Limited-Area 3DVAR, Ensemble Kalman Filter, and “Hybrid” Variational–Ensemble Data Assimilation Systems

Monthly Weather Review ◽

10.1175/mwr-d-13-00100.1 ◽

2014 ◽

Vol 142 (2) ◽

pp. 716-738 ◽

Cited By ~ 49

Author(s):

Craig S. Schwartz ◽

Zhiquan Liu

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Three Dimensional ◽

Skill Score ◽

Limited Area ◽

Computational Domain ◽

Ensemble Data Assimilation ◽

Ensemble Data ◽

Precipitation Characteristics ◽

High Precipitation

Abstract Analyses with 20-km horizontal grid spacing were produced from parallel continuously cycling three-dimensional variational (3DVAR), ensemble square root Kalman filter (EnSRF), and “hybrid” variational–ensemble data assimilation (DA) systems between 0000 UTC 6 May and 0000 UTC 21 June 2011 over a domain spanning the contiguous United States. Beginning 9 May, the 0000 UTC analyses initialized 36-h Weather Research and Forecasting Model (WRF) forecasts containing a large convection-permitting 4-km nest. These 4-km 3DVAR-, EnSRF-, and hybrid-initialized forecasts were compared to benchmark WRF forecasts initialized by interpolating 0000 UTC Global Forecast System (GFS) analyses onto the computational domain. While important differences regarding mean state characteristics of the 20-km DA systems were noted, verification efforts focused on the 4-km precipitation forecasts. The 3DVAR-, hybrid-, and EnSRF-initialized 4-km precipitation forecasts performed similarly regarding general precipitation characteristics, such as timing of the diurnal cycle, and all three forecast sets had high precipitation biases at heavier rainfall rates. However, meaningful differences emerged regarding precipitation placement as quantified by the fractions skill score. For most forecast hours, the hybrid-initialized 4-km precipitation forecasts were better than the EnSRF-, 3DVAR-, and GFS-initialized forecasts, and the improvement was often statistically significant at the 95th percentile. These results demonstrate the potential of limited-area continuously cycling hybrid DA configurations and suggest additional hybrid development is warranted.

Download Full-text

Southern High-Latitude Ensemble Data Assimilation in the Antarctic Mesoscale Prediction System

Monthly Weather Review ◽

10.1175/mwr3042.1 ◽

2005 ◽

Vol 133 (12) ◽

pp. 3431-3449 ◽

Cited By ~ 43

Author(s):

D. M. Barker

Keyword(s):

Data Assimilation ◽

Sampling Error ◽

Forecast Error ◽

Weather Prediction ◽

Forecast Errors ◽

Prediction System ◽

Spatial Covariance ◽

Ensemble Data Assimilation ◽

Ensemble Data ◽

The Antarctic

Abstract Ensemble data assimilation systems incorporate observations into numerical models via solution of the Kalman filter update equations, and estimates of forecast error covariances derived from ensembles of model integrations. In this paper, a particular algorithm, the ensemble square root filter (EnSRF), is tested in a limited-area, polar numerical weather prediction (NWP) model: the Antarctic Mesoscale Prediction System (AMPS). For application in the real-time AMPS, the number of model integrations that can be run to provide forecast error covariances is limited, resulting in an ensemble sampling error that degrades the analysis fit to observations. In this work, multivariate, climatologically plausible forecast error covariances are specified via averaged forecast difference statistics. Ensemble representations of the “true” forecast errors, created using randomized control variables of the fifth-generation Pennsylvania State University–National Center for Atmospheric Research (PSU–NCAR) Mesoscale Model (MM5) three-dimensional variational (3DVAR) data assimilation system, are then used to assess the dependence of sampling error on ensemble size, data density, and localization of covariances using simulated observation networks. Results highlight the detrimental impact of ensemble sampling error on the analysis increment structure of correlated, but unobserved fields—an issue not addressed by the spatial covariance localization techniques used to date. A 12-hourly cycling EnSRF/AMPS assimilation/forecast system is tested for a two-week period in December 2002 using real, conventional (surface, rawinsonde, satellite retrieval) observations. The dependence of forecast scores on methods used to maintain ensemble spread and the inclusion of perturbations to lateral boundary conditions are studied.

Download Full-text

Gain Form of the Ensemble Transform Kalman Filter and Its Relevance to Satellite Data Assimilation with Model Space Ensemble Covariance Localization

Monthly Weather Review ◽

10.1175/mwr-d-17-0102.1 ◽

2017 ◽

Vol 145 (11) ◽

pp. 4575-4592 ◽

Cited By ~ 15

Author(s):

Craig H. Bishop ◽

Jeffrey S. Whitaker ◽

Lili Lei

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Satellite Data ◽

Localization Length ◽

Model Space ◽

Ensemble Size ◽

Length Scales ◽

Ensemble Transform Kalman Filter ◽

Ensemble Transform ◽

Expanded Ensemble

To ameliorate suboptimality in ensemble data assimilation, methods have been introduced that involve expanding the ensemble size. Such expansions can incorporate model space covariance localization and/or estimates of climatological or model error covariances. Model space covariance localization in the vertical overcomes problematic aspects of ensemble-based satellite data assimilation. In the case of the ensemble transform Kalman filter (ETKF), the expanded ensemble size associated with vertical covariance localization would also enable the simultaneous update of entire vertical columns of model variables from hyperspectral and multispectral satellite sounders. However, if the original formulation of the ETKF were applied to an expanded ensemble, it would produce an analysis ensemble that was the same size as the expanded forecast ensemble. This article describes a variation on the ETKF called the gain ETKF (GETKF) that takes advantage of covariances from the expanded ensemble, while producing an analysis ensemble that has the required size of the unexpanded forecast ensemble. The approach also yields an inflation factor that depends on the localization length scale that causes the GETKF to perform differently to an ensemble square root filter (EnSRF) using the same expanded ensemble. Experimentation described herein shows that the GETKF outperforms a range of alternative ETKF-based solutions to the aforementioned problems. In cycling data assimilation experiments with a newly developed storm-track version of the Lorenz-96 model, the GETKF analysis root-mean-square error (RMSE) matches the EnSRF RMSE at shorter than optimal localization length scales but is superior in that it yields smaller RMSEs for longer localization length scales.

Download Full-text

Theoretical Application of Ensemble Kalman Filter to Adsorptive Solute Cr(VI) Transfer from Soil into Surface Runoff

Advanced Materials Research ◽

10.4028/www.scientific.net/amr.919-921.1257 ◽

2014 ◽

Vol 919-921 ◽

pp. 1257-1261

Author(s):

Chao Qun Tan ◽

Ju Xiu Tong ◽

Bill X. Hu ◽

Jin Zhong Yang

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Surface Runoff ◽

Amplification Factor ◽

Forecast Error ◽

Assimilation Effect ◽

Solute Transfer ◽

Error Covariance ◽

Theoretical Application

This paper mainly discusses some details when applying data assimilation method via an ensemble Kalman filter (EnKF) to improve prediction of adsorptive solute Cr(VI) transfer from soil into runoff. Based on this work, we could make better use of our theoretical model to predict adsorptive solute transfer from soil into surface runoff in practice. The results show that the ensemble number of 100 is reasonable, considering assimilation effect and efficiency after selecting its number from 25 to 225 at an interval of 25. While the initial ensemble value makes little difference to data assimilation (DA) results. Besides, DA results could be improved by multiplying an amplification factor to forecast error covariance matrix due to underestimation of forecast error.

Download Full-text

Development of a nonlinear ensemble data assimilation method with global state-space covariance localization

10.5194/egusphere-egu2020-19831 ◽

2020 ◽

Author(s):

Milija Zupanski

Keyword(s):

Data Assimilation ◽

State Space ◽

High Dimensional ◽

Ensemble Data Assimilation ◽

Error Covariance ◽

Ensemble Data ◽

Space Localization ◽

Initial Ensemble ◽

Analysis Error ◽

Observation Space

High-dimensional ensemble data assimilation applications require error covariance localization in order to address the problem of insufficient degrees of freedom, typically accomplished using the observation-space covariance localization. However, this creates a challenge for vertically integrated observations, such as satellite radiances, aerosol optical depth, etc., since the exact observation location in vertical does not exist. For nonlinear problems, there is an implied inconsistency in iterative minimization due to using observation-space localization which effectively prevents finding the optimal global minimizing solution. Using state-space localization, however, in principal resolves both issues associated with observation space localization.&#160;In this work we present a new nonlinear ensemble data assimilation method that employs covariance localization in state space and finds an optimal analysis solution. The new method resembles &#8220;modified ensembles&#8221; in the sense that ensemble size is increased in the analysis, but it differs in methodology used to create ensemble modifications, calculate the analysis error covariance, and define the initial ensemble perturbations for data assimilation cycling. From a practical point of view, the new method is considerably more efficient and potentially applicable to realistic high-dimensional data assimilation problems. A distinct characteristic of the new algorithm is that the localized error covariance and minimization are global, i.e. explicitly defined over all state points. The presentation will focus on examining feasible options for estimating the analysis error covariance and for defining the initial ensemble perturbations.

Download Full-text

A New Data Assimilation Scheme: The Space-Expanded Ensemble Localization Kalman Filter

Advances in Meteorology ◽

10.1155/2013/410812 ◽

2013 ◽

Vol 2013 ◽

pp. 1-6 ◽

Cited By ~ 4

Author(s):

Hongze Leng ◽

Junqiang Song ◽

Fengshun Lu ◽

Xiaoqun Cao

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Forecast Error ◽

Three Dimensional ◽

Full Rank ◽

Observation Error ◽

Model Framework ◽

Background Error ◽

Error Covariance ◽

Expanded Ensemble

This study considers a new hybrid three-dimensional variational (3D-Var) and ensemble Kalman filter (EnKF) data assimilation (DA) method in a non-perfect-model framework, named space-expanded ensemble localization Kalman filter (SELKF). In this method, the localization operation is directly applied to the ensemble anomalies with a Schur Product, rather than to the full error covariance of the state in the EnKF. Meanwhile, the correction space of analysis increment is expanded to a space with larger dimension, and the rank of the forecast error covariance is significantly increased. This scheme can reduce the spurious correlations in the covariance and approximate the full-rank background error covariance well. Furthermore, a deterministic scheme is used to generate the analysis anomalies. The results show that the SELKF outperforms the perturbed EnKF given a relatively small ensemble size, especially when the length scale is relatively long or the observation error covariance is relatively small.

Download Full-text

The Hybrid Local Ensemble Transform Kalman Filter

Monthly Weather Review ◽

10.1175/mwr-d-13-00131.1 ◽

2014 ◽

Vol 142 (6) ◽

pp. 2139-2149 ◽

Cited By ~ 30

Author(s):

Stephen G. Penny

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Variational Methods ◽

Hybrid Methods ◽

Dynamic Error ◽

Background Error ◽

Error Covariance ◽

Small Ensemble ◽

Ensemble Transform Kalman Filter ◽

Ensemble Transform

Abstract Hybrid data assimilation methods combine elements of ensemble Kalman filters (EnKF) and variational methods. While most approaches have focused on augmenting an operational variational system with dynamic error covariance information from an ensemble, this study takes the opposite perspective of augmenting an operational EnKF with information from a simple 3D variational data assimilation (3D-Var) method. A class of hybrid methods is introduced that combines the gain matrices of the ensemble and variational methods, rather than linearly combining the respective background error covariances. A hybrid local ensemble transform Kalman filter (Hybrid-LETKF) is presented in two forms: 1) a traditionally motivated Hybrid/Covariance-LETKF that combines the background error covariance matrices of LETKF and 3D-Var, and 2) a simple-to-implement algorithm called the Hybrid/Mean-LETKF that falls into the new class of hybrid gain methods. Both forms improve analysis errors when using small ensemble sizes and low observation coverage versus either LETKF or 3D-Var used alone. The results imply that for small ensemble sizes, allowing a solution to be found outside of the space spanned by ensemble members provides robustness in both hybrid methods compared to LETKF alone. Finally, the simplicity of the Hybrid/Mean-LETKF design implies that this algorithm can be applied operationally while requiring only minor modifications to an existing operational 3D-Var system.

Download Full-text

Parameter estimation for ocean biogeochemical component in a global model using Ensemble Kalman Filter: a twin experiment

10.5194/egusphere-egu21-9947 ◽

2021 ◽

Author(s):

Tarkeshwar Singh ◽

Francois Counillon ◽

Jerry F. Tjiputra ◽

Mohamad El Gharamti

Keyword(s):

Parameter Estimation ◽

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Earth System Model ◽

System Model ◽

Earth System ◽

Ensemble Data Assimilation ◽

Ensemble Data ◽

Parameter Values

Ocean biogeochemical (BGC) models utilize a large number of poorly-constrained global parameters to mimic unresolved processes and reproduce the observed complex spatio-temporal patterns. Large model errors stem primarily from inaccuracies in these parameters whose optimal values can vary both in space and time. This study aims to demonstrate the ability of ensemble data assimilation (DA) methods to provide high-quality and improved BGC parameters within an Earth system model in idealized twin experiment framework. &#160;We use the Norwegian Climate Prediction Model (NorCPM), which combines the Norwegian Earth System Model with the Dual-One-Step ahead smoothing-based Ensemble Kalman Filter (DOSA-EnKF). The work follows on Gharamti et al. (2017) that successfully demonstrates the approach for one-dimensional idealized ocean BGC models. We aim to estimate five spatially varying BGC parameters by assimilating Salinity and Temperature hydrographic profiles and surface BGC (Phytoplankton, Nitrate, Phosphorous, Silicate, and Oxygen) observations in a strongly coupled DA framework &#8211; i.e., jointly updating ocean and BGC state-parameters during the assimilation. The method converges quickly (less than a year), largely reducing the errors in the BGC parameters and eventually it is shown to perform nearly as well as that of the system with true parameter values. Optimal parameter values can also be recovered by assimilating climatological BGC observations and challenging sparse observational networks. The findings of this study demonstrate the applicability of the approach for tuning the system in a real framework.&#160;References:Gharamti, M. E., Tjiputra, J., Bethke, I., Samuelsen, A., Skjelvan, I., Bentsen, M., & Bertino, L. (2017). Ensemble data assimilation for ocean biogeochemical state and parameter estimation at different sites.&#160;Ocean Modelling,&#160;112, 65-89.

Download Full-text