Development of a nonlinear ensemble data assimilation method with global state-space covariance localization

Error Covariance ◽

Ensemble Data ◽

Space Localization ◽

Initial Ensemble ◽

Analysis Error ◽

Observation Space

High-dimensional ensemble data assimilation applications require error covariance localization in order to address the problem of insufficient degrees of freedom, typically accomplished using the observation-space covariance localization. However, this creates a challenge for vertically integrated observations, such as satellite radiances, aerosol optical depth, etc., since the exact observation location in vertical does not exist. For nonlinear problems, there is an implied inconsistency in iterative minimization due to using observation-space localization which effectively prevents finding the optimal global minimizing solution. Using state-space localization, however, in principal resolves both issues associated with observation space localization.&#160;In this work we present a new nonlinear ensemble data assimilation method that employs covariance localization in state space and finds an optimal analysis solution. The new method resembles &#8220;modified ensembles&#8221; in the sense that ensemble size is increased in the analysis, but it differs in methodology used to create ensemble modifications, calculate the analysis error covariance, and define the initial ensemble perturbations for data assimilation cycling. From a practical point of view, the new method is considerably more efficient and potentially applicable to realistic high-dimensional data assimilation problems. A distinct characteristic of the new algorithm is that the localized error covariance and minimization are global, i.e. explicitly defined over all state points. The presentation will focus on examining feasible options for estimating the analysis error covariance and for defining the initial ensemble perturbations.

Maximum Likelihood Ensemble Filter: Theoretical Aspects

10.1175/mwr2946.1 ◽

2005 ◽

Vol 133 (6) ◽

pp. 1710-1726 ◽

Cited By ~ 222

Author(s):

Milija Zupanski

Keyword(s):

Maximum Likelihood ◽

Data Assimilation ◽

Cost Function ◽

Maximum Likelihood Ensemble Filter

Error Covariance ◽

Ensemble Data ◽

Analysis Error ◽

Nonlinear Observation ◽

The Cost ◽

Abstract A new ensemble-based data assimilation method, named the maximum likelihood ensemble filter (MLEF), is presented. The analysis solution maximizes the likelihood of the posterior probability distribution, obtained by minimization of a cost function that depends on a general nonlinear observation operator. The MLEF belongs to the class of deterministic ensemble filters, since no perturbed observations are employed. As in variational and ensemble data assimilation methods, the cost function is derived using a Gaussian probability density function framework. Like other ensemble data assimilation algorithms, the MLEF produces an estimate of the analysis uncertainty (e.g., analysis error covariance). In addition to the common use of ensembles in calculation of the forecast error covariance, the ensembles in MLEF are exploited to efficiently calculate the Hessian preconditioning and the gradient of the cost function. A sufficient number of iterative minimization steps is 2–3, because of superior Hessian preconditioning. The MLEF method is well suited for use with highly nonlinear observation operators, for a small additional computational cost of minimization. The consistent treatment of nonlinear observation operators through optimization is an advantage of the MLEF over other ensemble data assimilation algorithms. The cost of MLEF is comparable to the cost of existing ensemble Kalman filter algorithms. The method is directly applicable to most complex forecast models and observation operators. In this paper, the MLEF method is applied to data assimilation with the one-dimensional Korteweg–de Vries–Burgers equation. The tested observation operator is quadratic, in order to make the assimilation problem more challenging. The results illustrate the stability of the MLEF performance, as well as the benefit of the cost function minimization. The improvement is noted in terms of the rms error, as well as the analysis error covariance. The statistics of innovation vectors (observation minus forecast) also indicate a stable performance of the MLEF algorithm. Additional experiments suggest the amplified benefit of targeted observations in ensemble data assimilation.

Observation-Informed Generalized Hybrid Error Covariance Models

10.1175/mwr-d-18-0016.1 ◽

2018 ◽

Vol 146 (11) ◽

pp. 3605-3622 ◽

Cited By ~ 2

Author(s):

Elizabeth A. Satterfield ◽

Daniel Hodyss ◽

David D. Kuhl ◽

Craig H. Bishop

Keyword(s):

Data Assimilation ◽

Covariance Matrix ◽

Sample Variance ◽

Calibration Function ◽

Covariance Model ◽

True Error ◽

Error Covariance ◽

Ensemble Data ◽

Prior Variance

Abstract Because of imperfections in ensemble data assimilation schemes, one cannot assume that the ensemble-derived covariance matrix is equal to the true error covariance matrix. Here, we describe a simple and intuitively compelling method to fit calibration functions of the ensemble sample variance to the mean of the distribution of true error variances, given an ensemble estimate. We demonstrate that the use of such calibration functions is consistent with theory showing that, when sampling error in the prior variance estimate is considered, the gain that minimizes the posterior error variance uses the expected true prior variance, given an ensemble sample variance. Once the calibration function has been fitted, it can be combined with ensemble-based and climatologically based error correlation information to obtain a generalized hybrid error covariance model. When the calibration function is chosen to be a linear function of the ensemble variance, the generalized hybrid error covariance model is the widely used linear hybrid consisting of a weighted sum of a climatological and an ensemble-based forecast error covariance matrix. However, when the calibration function is chosen to be, say, a cubic function of the ensemble sample variance, the generalized hybrid error covariance model is a nonlinear function of the ensemble estimate. We consider idealized univariate data assimilation and multivariate cycling ensemble data assimilation to demonstrate that the generalized hybrid error covariance model closely approximates the optimal weights found through computationally expensive tuning in the linear case and, in the nonlinear case, outperforms any plausible linear model.

A Nonvariational Consistent Hybrid Ensemble Filter

10.1175/mwr-d-14-00391.1 ◽

2015 ◽

Vol 143 (12) ◽

pp. 5073-5090 ◽

Cited By ~ 8

Author(s):

Craig H. Bishop ◽

Bo Huang ◽

Xuguang Wang

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Forecast Error ◽

Model Space ◽

High Rank ◽

Covariance Model ◽

Error Covariance ◽

Ensemble Data ◽

Ensemble Transform Kalman Filter

Abstract A consistent hybrid ensemble filter (CHEF) for using hybrid forecast error covariance matrices that linearly combine aspects of both climatological and flow-dependent matrices within a nonvariational ensemble data assimilation scheme is described. The CHEF accommodates the ensemble data assimilation enhancements of (i) model space ensemble covariance localization for satellite data assimilation and (ii) Hodyss’s method for improving accuracy using ensemble skewness. Like the local ensemble transform Kalman filter (LETKF), the CHEF is computationally scalable because it updates local patches of the atmosphere independently of others. Like the sequential ensemble Kalman filter (EnKF), it serially assimilates batches of observations and uses perturbed observations to create ensembles of analyses. It differs from the deterministic (no perturbed observations) ensemble square root filter (ESRF) and the EnKF in that (i) its analysis correction is unaffected by the order in which observations are assimilated even when localization is required, (ii) it uses accurate high-rank solutions for the posterior error covariance matrix to serially assimilate observations, and (iii) it accommodates high-rank hybrid error covariance models. Experiments were performed to assess the effect on CHEF and ESRF analysis accuracy of these differences. In the case where both the CHEF and the ESRF used tuned localized ensemble covariances for the forecast error covariance model, the CHEF’s advantage over the ESRF increased with observational density. In the case where the CHEF used a hybrid error covariance model but the ESRF did not, the CHEF had a substantial advantage for all observational densities.

Initiation of ensemble data assimilation

Tellus A Dynamic Meteorology and Oceanography ◽

10.3402/tellusa.v58i2.14766 ◽

2006 ◽

Cited By ~ 1

Author(s):

M. Zupanski ◽

S. J. Fletcher ◽

I. M. Navon ◽

B. Uzunoglu ◽

R. P. Heikes ◽

...

Keyword(s):

Data Assimilation ◽

Ensemble Data

Using a machine learning proxy for localization in ensemble data assimilation

Computational Geosciences ◽

10.1007/s10596-020-10031-0 ◽

2021 ◽

Vol 25 (3) ◽

pp. 931-944

Author(s):

Johann M. Lacerda ◽

Alexandre A. Emerick ◽

Adolfo P. Pires

Keyword(s):

Machine Learning ◽

Data Assimilation ◽

Ensemble Data

An adaptive covariance relaxation method for ensemble data assimilation

Quarterly Journal of the Royal Meteorological Society ◽

10.1002/qj.2576 ◽

2015 ◽

Vol 141 (692) ◽

pp. 2898-2906 ◽

Cited By ~ 24

Author(s):

Yue Ying ◽

Fuqing Zhang

Keyword(s):

Data Assimilation ◽

Relaxation Method ◽

Ensemble Data

Convection-Permitting Forecasts Initialized with Continuously Cycling Limited-Area 3DVAR, Ensemble Kalman Filter, and “Hybrid” Variational–Ensemble Data Assimilation Systems

10.1175/mwr-d-13-00100.1 ◽

2014 ◽

Vol 142 (2) ◽

pp. 716-738 ◽

Cited By ~ 49

Author(s):

Craig S. Schwartz ◽

Zhiquan Liu

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Three Dimensional ◽

Skill Score ◽

Limited Area ◽

Computational Domain ◽

Ensemble Data ◽

Precipitation Characteristics ◽

High Precipitation

Abstract Analyses with 20-km horizontal grid spacing were produced from parallel continuously cycling three-dimensional variational (3DVAR), ensemble square root Kalman filter (EnSRF), and “hybrid” variational–ensemble data assimilation (DA) systems between 0000 UTC 6 May and 0000 UTC 21 June 2011 over a domain spanning the contiguous United States. Beginning 9 May, the 0000 UTC analyses initialized 36-h Weather Research and Forecasting Model (WRF) forecasts containing a large convection-permitting 4-km nest. These 4-km 3DVAR-, EnSRF-, and hybrid-initialized forecasts were compared to benchmark WRF forecasts initialized by interpolating 0000 UTC Global Forecast System (GFS) analyses onto the computational domain. While important differences regarding mean state characteristics of the 20-km DA systems were noted, verification efforts focused on the 4-km precipitation forecasts. The 3DVAR-, hybrid-, and EnSRF-initialized 4-km precipitation forecasts performed similarly regarding general precipitation characteristics, such as timing of the diurnal cycle, and all three forecast sets had high precipitation biases at heavier rainfall rates. However, meaningful differences emerged regarding precipitation placement as quantified by the fractions skill score. For most forecast hours, the hybrid-initialized 4-km precipitation forecasts were better than the EnSRF-, 3DVAR-, and GFS-initialized forecasts, and the improvement was often statistically significant at the 95th percentile. These results demonstrate the potential of limited-area continuously cycling hybrid DA configurations and suggest additional hybrid development is warranted.

Southern High-Latitude Ensemble Data Assimilation in the Antarctic Mesoscale Prediction System

10.1175/mwr3042.1 ◽

2005 ◽

Vol 133 (12) ◽

pp. 3431-3449 ◽

Cited By ~ 43

Author(s):

D. M. Barker

Keyword(s):

Data Assimilation ◽

Sampling Error ◽

Forecast Error ◽

Weather Prediction ◽

Forecast Errors ◽

Prediction System ◽

Spatial Covariance ◽

Ensemble Data ◽

The Antarctic

Abstract Ensemble data assimilation systems incorporate observations into numerical models via solution of the Kalman filter update equations, and estimates of forecast error covariances derived from ensembles of model integrations. In this paper, a particular algorithm, the ensemble square root filter (EnSRF), is tested in a limited-area, polar numerical weather prediction (NWP) model: the Antarctic Mesoscale Prediction System (AMPS). For application in the real-time AMPS, the number of model integrations that can be run to provide forecast error covariances is limited, resulting in an ensemble sampling error that degrades the analysis fit to observations. In this work, multivariate, climatologically plausible forecast error covariances are specified via averaged forecast difference statistics. Ensemble representations of the “true” forecast errors, created using randomized control variables of the fifth-generation Pennsylvania State University–National Center for Atmospheric Research (PSU–NCAR) Mesoscale Model (MM5) three-dimensional variational (3DVAR) data assimilation system, are then used to assess the dependence of sampling error on ensemble size, data density, and localization of covariances using simulated observation networks. Results highlight the detrimental impact of ensemble sampling error on the analysis increment structure of correlated, but unobserved fields—an issue not addressed by the spatial covariance localization techniques used to date. A 12-hourly cycling EnSRF/AMPS assimilation/forecast system is tested for a two-week period in December 2002 using real, conventional (surface, rawinsonde, satellite retrieval) observations. The dependence of forecast scores on methods used to maintain ensemble spread and the inclusion of perturbations to lateral boundary conditions are studied.