Intercomparison of Variational Data Assimilation and the Ensemble Kalman Filter for Global Deterministic NWP. Part I: Description and Single-Observation Experiments

Abstract An intercomparison of the Environment Canada variational and ensemble Kalman filter (EnKF) data assimilation systems is presented in the context of global deterministic NWP. In an EnKF experiment having the same spatial resolution as the inner loop in the four-dimensional variational data assimilation system (4D-Var), the mean of each analysis ensemble is used to initialize the higher-resolution deterministic forecasts. Five different variational data assimilation experiments are also conducted. These include both 4D-Var and 3D-Var (with first guess at appropriate time) experiments using either (i) prescribed background-error covariances similar to those used operationally, which are static in time and include horizontally homogeneous and isotropic correlations; or (ii) flow-dependent covariances computed from the EnKF background ensembles with spatial covariance localization applied. The fifth variational data assimilation experiment is a new approach called the Ensemble-4D-Var (En-4D-Var). This approach uses 4D flow-dependent background-error covariances estimated from EnKF ensembles to produce a 4D analysis without the need for tangent-linear or adjoint versions of the forecast model. In this first part of a two-part paper, results from a series of idealized assimilation experiments are presented. In these experiments, only a single observation or vertical profile of observations is assimilated to explore the impact of various fundamental differences among the EnKF and the various variational data assimilation approaches considered. In particular, differences in the application of covariance localization in the EnKF and variational approaches are shown to have a significant impact on the assimilation of satellite radiance observations. The results also demonstrate that 4D-Var and the EnKF can both produce similar 4D background-error covariances within a 6-h assimilation window. In the second part, results from medium-range deterministic forecasts for the study period of February 2007 are presented for the EnKF and the five variational data assimilation approaches considered.

Download Full-text

Intercomparison of Variational Data Assimilation and the Ensemble Kalman Filter for Global Deterministic NWP. Part II: One-Month Experiments with Real Observations

Monthly Weather Review ◽

10.1175/2009mwr3158.1 ◽

2010 ◽

Vol 138 (5) ◽

pp. 1567-1586 ◽

Cited By ~ 146

Author(s):

Mark Buehner ◽

P. L. Houtekamer ◽

Cecilien Charette ◽

Herschel L. Mitchell ◽

Bin He

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Forecast Model ◽

Variational Data Assimilation ◽

Background Error ◽

Single Observation ◽

Operational System ◽

Medium Range ◽

Forecast Quality

Abstract An intercomparison of the Environment Canada variational and ensemble Kalman filter (EnKF) data assimilation systems is presented in the context of producing global deterministic numerical weather forecasts. Five different variational data assimilation approaches are considered including four-dimensional variational data assimilation (4D-Var) and three-dimensional variational data assimilation (3D-Var) with first guess at the appropriate time (3D-FGAT). Also included among these is a new approach, called Ensemble-4D-Var (En-4D-Var), that uses 4D ensemble background-error covariances from the EnKF. A description of the experimental configurations and results from single-observation experiments are presented in the first part of this two-part paper. The present paper focuses on results from medium-range deterministic forecasts initialized with analyses from the EnKF and the five variational data assimilation approaches for the period of February 2007. All experiments assimilate exactly the same full set of meteorological observations and use the same configuration of the forecast model to produce global deterministic medium-range forecasts. The quality of forecasts in the short (medium) range obtained by using the EnKF ensemble mean analysis is slightly degraded (improved) in the extratropics relative to using the 4D-Var analysis with background-error covariances similar to those used operationally. The use of the EnKF flow-dependent error covariances in the variational system (4D-Var or 3D-FGAT) leads to large (modest) forecast improvements in the southern extratropics (tropics) as compared with using covariances similar to the operational system (a gain of up to 9 h at day 5). The En-4D-Var approach leads to (i) either improved or similar forecast quality when compared with the 4D-Var experiment similar to the currently operational system, (ii) slightly worse forecast quality when compared with the 4D-Var experiment with EnKF error covariances, and (iii) generally similar forecast quality when compared with the EnKF experiment.

Download Full-text

An Examination of the Incremental Balance in a Global Ensemble-Based 3D-Var Data Assimilation System

Monthly Weather Review ◽

10.1175/2010mwr3339.1 ◽

2010 ◽

Vol 138 (10) ◽

pp. 3946-3966 ◽

Cited By ~ 2

Author(s):

Jean-François Caron ◽

Luc Fillion

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Three Dimensional ◽

Vertical Motion ◽

Variational Data Assimilation ◽

Spatial Covariance ◽

Background Error ◽

Localization Technique ◽

Vertical Localization

Abstract This study examines the modification to the balance properties of the analysis increments in a global three-dimensional variational data assimilation scheme when using flow-dependent background-error covariances derived from an operational ensemble Kalman filter instead of static homogenous and isotropic background-error covariances based on lagged forecast differences. It is shown that the degree of balance in the analysis increments is degraded when the former method is used. This change can be attributed in part to the reduced degree of rotational balance found in short-term ensemble Kalman filter perturbations as compared to lagged forecast differences based on longer-range forecasts. However, the use of a horizontal and vertical localization technique to increase the rank of the ensemble-based covariances are found to have a significant deleterious effect on the rotational balance with the largest detrimental impact coming from the vertical localization and affecting particularly the upper levels. The examination of the vertical motion part of the analysis increments revealed that the spatial covariance localization technique also produces unrealistic vertical structure of vertical motion increments with abnormally large increments near the surface. A comparison between the analysis increments from the ensemble Kalman filter and from the ensemble-based three-dimensional variational data assimilation (3D-Var) scheme showed that the balance characteristics of the analysis increments resulting from the two systems are very similar.

Download Full-text

Singular-Vector-Based Covariance Propagation in a Quasigeostrophic Assimilation System

Monthly Weather Review ◽

10.1175/mwr2909.1 ◽

2005 ◽

Vol 133 (5) ◽

pp. 1295-1310 ◽

Cited By ~ 13

Author(s):

Alexander Beck ◽

Martin Ehrendorfer

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Variational Data Assimilation ◽

Dynamic Information ◽

Dynamic Background ◽

Background Error ◽

Singular Vectors ◽

Reduced Rank ◽

The Impact ◽

Static Background

Abstract Variational data assimilation systems require the specification of the covariances of background and observation errors. Although the specification of the background-error covariances has been the subject of intense research, current operational data assimilation systems still rely on essentially static and thus flow-independent background-error covariances. At least theoretically, it is possible to use flow-dependent background-error covariances in four-dimensional variational data assimilation (4DVAR) through exploiting the connection between variational data assimilation and estimation theory. This paper reports on investigations concerning the impact of flow-dependent background-error covariances in an idealized 4DVAR system that, based on quasigeostrophic dynamics, assimilates artificial observations. The main emphasis is placed on quantifying the improvement in analysis quality that is achievable in 4DVAR through the use of flow-dependent background-error covariances. Flow dependence is achieved through dynamical error-covariance evolution based on singular vectors in a reduced-rank approach, referred to as reduced-rank Kalman filter (RRKF). The RRKF yields partly dynamic background-error covariances through blending static and dynamic information, where the dynamic information is obtained from error evolution in a subspace of dimension k (defined here through the singular vectors) that may be small compared to the dimension of the model’s phase space n, which is equal to 1449 in the system investigated here. The results show that the use of flow-dependent background-error covariances based on the RRKF leads to improved analyses compared to a system using static background-error statistics. That latter system uses static background-error covariances that are carefully tuned given the model dynamics and the observational information available. It is also shown that the performance of the RRKF approaches the performance of the extended Kalman filter, as k approaches n. Results therefore support the hypothesis that significant analysis improvement is possible through the use of flow-dependent background-error covariances given that a sufficiently large number (here on the order of n/10) of singular vectors is used.

Download Full-text

An Ensemble Kalman Filter for Numerical Weather Prediction Based on Variational Data Assimilation: VarEnKF

Monthly Weather Review ◽

10.1175/mwr-d-16-0106.1 ◽

2017 ◽

Vol 145 (2) ◽

pp. 617-635 ◽

Cited By ~ 8

Author(s):

Mark Buehner ◽

Ron McTaggart-Cowan ◽

Sylvain Heilliette

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Variational Approach ◽

Variational Data Assimilation ◽

Ensemble Prediction ◽

Deterministic System ◽

Ensemble Forecasts ◽

Background Error ◽

Ensemble Mean

Several NWP centers currently employ a variational data assimilation approach for initializing deterministic forecasts and a separate ensemble Kalman filter (EnKF) system both for initializing ensemble forecasts and for providing ensemble background error covariances for the deterministic system. This study describes a new approach for performing the data assimilation step within a perturbed-observation EnKF. In this approach, called VarEnKF, the analysis increment is computed with a variational data assimilation approach both for the ensemble mean and for all of the ensemble perturbations. To obtain a computationally efficient algorithm, a much simpler configuration is used for the ensemble perturbations, whereas the configuration used for the ensemble mean is similar to that used for the deterministic system. Numerous practical benefits may be realized by using a variational approach for both deterministic and ensemble prediction, including improved efficiency for the development and maintenance of the computer code. Also, the use of essentially the same data assimilation algorithm would likely reduce the amount of numerical experimentation required when making system changes, since their impacts in the two systems would be very similar. The variational approach enables the use of hybrid background error covariances and may also allow the assimilation of a larger volume of observations. Preliminary tests with the Canadian global 256-member system produced significantly improved ensemble forecasts with VarEnKF as compared with the current EnKF and at a comparable computational cost. These improvements resulted entirely from changes to the ensemble mean analysis increment calculation. Moreover, because each ensemble perturbation is updated independently, VarEnKF scales perfectly up to a very large number of processors.

Download Full-text

Preemptive Forecasts Using an Ensemble Kalman Filter

Monthly Weather Review ◽

10.1175/mwr3480.1 ◽

2007 ◽

Vol 135 (10) ◽

pp. 3484-3495 ◽

Cited By ~ 9

Author(s):

Brian J. Etherton

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Three Dimensional ◽

Forecast Model ◽

Observation Time ◽

Model Integration ◽

Model Forecast ◽

Error Statistics ◽

Background Error

Abstract An ensemble Kalman filter (EnKF) estimates the error statistics of a model forecast using an ensemble of model forecasts. One use of an EnKF is data assimilation, resulting in the creation of an increment to the first-guess field at the observation time. Another use of an EnKF is to propagate error statistics of a model forecast forward in time, such as is done for optimizing the location of adaptive observations. Combining these two uses of an ensemble Kalman filter, a “preemptive forecast” can be generated. In a preemptive forecast, the increment to the first-guess field is, using ensembles, propagated to some future time and added to the future control forecast, resulting in a new forecast. This new forecast requires no more time to produce than the time needed to run a data assimilation scheme, as no model integration is necessary. In an observing system simulation experiment (OSSE), a barotropic vorticity model was run to produce a 300-day “nature run.” The same model, run with a different vorticity forcing scheme, served as the forecast model. The model produced 24- and 48-h forecasts for each of the 300 days. The model was initialized every 24 h by assimilating observations of the nature run using a hybrid ensemble Kalman filter–three-dimensional variational data assimilation (3DVAR) scheme. In addition to the control forecast, a 64-member forecast ensemble was generated for each of the 300 days. Every 24 h, given a set of observations, the 64-member ensemble, and the control run, an EnKF was used to create 24-h preemptive forecasts. The preemptive forecasts were more accurate than the unmodified, original 48-h forecasts, though not quite as accurate as the 24-h forecast obtained from a new model integration initialized by assimilating the same observations as were used in the preemptive forecasts. The accuracy of the preemptive forecasts improved significantly when 1) the ensemble-based error statistics used by the EnKF were localized using a Schur product and 2) a model error term was included in the background error covariance matrices.

Download Full-text

A New Approach for Estimating the Observation Impact in Ensemble–Variational Data Assimilation

Monthly Weather Review ◽

10.1175/mwr-d-17-0252.1 ◽

2018 ◽

Vol 146 (2) ◽

pp. 447-465 ◽

Cited By ~ 12

Author(s):

Mark Buehner ◽

Ping Du ◽

Joël Bédard

Keyword(s):

Data Assimilation ◽

Forecast Error ◽

Forecast Model ◽

Variational Data Assimilation ◽

Observation Error ◽

New Approach ◽

Regional Prediction ◽

Observation Impact ◽

Adjoint Approach ◽

The Impact

Abstract Two types of approaches are commonly used for estimating the impact of arbitrary subsets of observations on short-range forecast error. The first was developed for variational data assimilation systems and requires the adjoint of the forecast model. Comparable approaches were developed for use with the ensemble Kalman filter and rely on ensembles of forecasts. In this study, a new approach for computing observation impact is proposed for ensemble–variational data assimilation (EnVar). Like standard adjoint approaches, the adjoint of the data assimilation procedure is implemented through the iterative minimization of a modified cost function. However, like ensemble approaches, the adjoint of the forecast step is obtained by using an ensemble of forecasts. Numerical experiments were performed to compare the new approach with the standard adjoint approach in the context of operational deterministic NWP. Generally similar results are obtained with both approaches, especially when the new approach uses covariance localization that is horizontally advected between analysis and forecast times. However, large differences in estimated impacts are obtained for some surface observations. Vertical propagation of the observation impact is noticeably restricted with the new approach because of vertical covariance localization. The new approach is used to evaluate changes in observation impact as a result of the use of interchannel observation error correlations for radiance observations. The estimated observation impact in similarly configured global and regional prediction systems is also compared. Overall, the new approach should provide useful estimates of observation impact for data assimilation systems based on EnVar when an adjoint model is not available.

Download Full-text

Evaluation of Surface Analyses and Forecasts with a Multiscale Ensemble Kalman Filter in Regions of Complex Terrain

Monthly Weather Review ◽

10.1175/2010mwr3612.1 ◽

2011 ◽

Vol 139 (6) ◽

pp. 2008-2024 ◽

Cited By ~ 24

Author(s):

Brian C. Ancell ◽

Clifford F. Mass ◽

Gregory J. Hakim

Keyword(s):

Kalman Filter ◽

High Resolution ◽

Data Assimilation ◽

Ensemble Kalman Filter ◽

Complex Terrain ◽

Error Variance ◽

Small Scale ◽

Observation Error ◽

Grid Spacing ◽

Background Error

Abstract Previous research suggests that an ensemble Kalman filter (EnKF) data assimilation and modeling system can produce accurate atmospheric analyses and forecasts at 30–50-km grid spacing. This study examines the ability of a mesoscale EnKF system using multiscale (36/12 km) Weather Research and Forecasting (WRF) model simulations to produce high-resolution, accurate, regional surface analyses, and 6-h forecasts. This study takes place over the complex terrain of the Pacific Northwest, where the small-scale features of the near-surface flow field make the region particularly attractive for testing an EnKF and its flow-dependent background error covariances. A variety of EnKF experiments are performed over a 5-week period to test the impact of decreasing the grid spacing from 36 to 12 km and to evaluate new approaches for dealing with representativeness error, lack of surface background variance, and low-level bias. All verification in this study is performed with independent, unassimilated observations. Significant surface analysis and 6-h forecast improvements are found when EnKF grid spacing is reduced from 36 to 12 km. Forecast improvements appear to be a consequence of increased resolution during model integration, whereas analysis improvements also benefit from high-resolution ensemble covariances during data assimilation. On the 12-km domain, additional analysis improvements are found by reducing observation error variance in order to address representativeness error. Removing model surface biases prior to assimilation significantly enhances the analysis. Inflating surface wind and temperature background error variance has large impacts on analyses, but only produces small improvements in analysis RMS errors. Both surface and upper-air 6-h forecasts are nearly unchanged in the 12-km experiments. Last, 12-km WRF EnKF surface analyses and 6-h forecasts are shown to generally outperform those of the Global Forecast System (GFS), North American Model (NAM), and the Rapid Update Cycle (RUC) by about 10%–30%, although these improvements do not extend above the surface. Based on these results, future improvements in multiscale EnKF are suggested.

Download Full-text

Simulation of Monsoon Depressions Using WRF-VAR: Impact of Different Background Error Statistics and Lateral Boundary Conditions

Monthly Weather Review ◽

10.1175/mwr-d-13-00285.1 ◽

2014 ◽

Vol 142 (10) ◽

pp. 3586-3613 ◽

Cited By ~ 20

Author(s):

A. Routray ◽

S. C. Kar ◽

P. Mali ◽

K. Sowjanya

Keyword(s):

Data Assimilation ◽

Boundary Conditions ◽

Initial Conditions ◽

Variational Data Assimilation ◽

Error Statistics ◽

Background Error ◽

Single Observation ◽

Monsoon Depressions ◽

Data Assimilation System ◽

Assimilation System

Abstract In a variational data assimilation system, background error statistics (BES) spread the influence of the observations in space and filter analysis increments through dynamic balance or statistical relationships. In a data-sparse region such as the Bay of Bengal, BES play an important role in defining the location and structure of monsoon depressions (MDs). In this study, the Indian-region-specific BES have been computed for the Weather Research and Forecasting (WRF) three-dimensional variational data assimilation system. A comparative study using single observation tests is carried out using the computed BES and global BES within the WRF system. Both sets of BES are used in the assimilation cycles and forecast runs for simulating the meteorological features associated with the MDs. Numerical experiments have been conducted to assess the relative impact of various BES in the analysis and simulations of the MDs. The results show that use of regional BES in the assimilation cycle has a positive impact on the prediction of the location, propagation, and development of rainbands associated with the MDs. The track errors of MDs are smaller when domain-specific BES are used in the assimilation cycle. Additional experiments have been conducted using data from the Interim European Centre for Medium-Range Weather Forecasts Re-Analysis (ERA-Interim) as initial and boundary conditions (IBCs) in the assimilation cycle. The results indicate that the use of domain-dependent BES and high-resolution ERA-I data as IBCs further improved the initial conditions for the model leading to better forecasts of the MDs.

Download Full-text

An Ensemble Kalman Filter for severe dust storm data assimilation over China

Atmospheric Chemistry and Physics ◽

10.5194/acp-8-2975-2008 ◽

2008 ◽

Vol 8 (11) ◽

pp. 2975-2983 ◽

Cited By ~ 45

Author(s):

C. Lin ◽

Z. Wang ◽

J. Zhu

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Dust Storm ◽

Ensemble Kalman Filter ◽

Deposition Velocity ◽

Dust Storms ◽

Initial Time ◽

Model Errors ◽

The Impact ◽

Severe Dust Storm

Abstract. An Ensemble Kalman Filter (EnKF) data assimilation system was developed for a regional dust transport model. This paper applied the EnKF method to investigate modeling of severe dust storm episodes occurring in March 2002 over China based on surface observations of dust concentrations to explore the impact of the EnKF data assimilation systems on forecast improvement. A series of sensitivity experiments using our system demonstrates the ability of the advanced EnKF assimilation method using surface observed PM10 in North China to correct initial conditions, which leads to improved forecasts of dust storms. However, large errors in the forecast may arise from model errors (uncertainties in meteorological fields, dust emissions, dry deposition velocity, etc.). This result illustrates that the EnKF requires identification and correction model errors during the assimilation procedure in order to significantly improve forecasts. Results also show that the EnKF should use a large inflation parameter to obtain better model performance and forecast potential. Furthermore, the ensemble perturbations generated at the initial time should include enough ensemble spreads to represent the background error after several assimilation cycles.

Download Full-text

E4DVar: Coupling an Ensemble Kalman Filter with Four-Dimensional Variational Data Assimilation in a Limited-Area Weather Prediction Model

Monthly Weather Review ◽

10.1175/mwr-d-11-00023.1 ◽

2012 ◽

Vol 140 (2) ◽

pp. 587-600 ◽

Cited By ~ 69

Author(s):

Meng Zhang ◽

Fuqing Zhang

Keyword(s):

Kalman Filter ◽

Data Assimilation ◽

Prediction Model ◽

Ensemble Kalman Filter ◽

Weather Prediction ◽

Coupled System ◽

Limited Area ◽

Ensemble Forecasts ◽

Background Error ◽

Hybrid Data Assimilation

A hybrid data assimilation approach that couples the ensemble Kalman filter (EnKF) and four-dimensional variational (4DVar) methods is implemented for the first time in a limited-area weather prediction model. In this coupled system, denoted E4DVar, the EnKF and 4DVar systems run in parallel while feeding into each other. The multivariate, flow-dependent background error covariance estimated from the EnKF ensemble is used in the 4DVar minimization and the ensemble mean in the EnKF analysis is replaced by the 4DVar analysis, while updating the analysis perturbations for the next cycle of ensemble forecasts with the EnKF. Therefore, the E4DVar can obtain flow-dependent information from both the explicit covariance matrix derived from ensemble forecasts, as well as implicitly from the 4DVar trajectory. The performance of an E4DVar system is compared with the uncoupled 4DVar and EnKF for a limited-area model by assimilating various conventional observations over the contiguous United States for June 2003. After verifying the forecasts from each analysis against standard sounding observations, it is found that the E4DVar substantially outperforms both the EnKF and 4DVar during this active summer month, which featured several episodes of severe convective weather. On average, the forecasts produced from E4DVar analyses have considerably smaller errors than both of the stand-alone EnKF and 4DVar systems for forecast lead times up to 60 h.

Download Full-text