scholarly journals Investigating the Use of Ensemble Variance to Predict Observation Error of Representation

2017 ◽  
Vol 145 (2) ◽  
pp. 653-667 ◽  
Author(s):  
Elizabeth Satterfield ◽  
Daniel Hodyss ◽  
David D. Kuhl ◽  
Craig H. Bishop
Keyword(s):  
Data Assimilation ◽  
Forecast Model ◽  
Error Variance ◽  
Observation Error ◽  
Coarse Resolution ◽  
Error Covariance ◽  
Flow Dependence ◽  
Representation Error ◽  
Resolution Model ◽  
Predictive Relationship

Data assimilation schemes combine observational data with a short-term model forecast to produce an analysis. However, many characteristics of the atmospheric states described by the observations and the model differ. Observations often measure a higher-resolution state than coarse-resolution model grids can describe. Hence, the observations may measure aspects of gradients or unresolved eddies that are poorly resolved by the filtered version of reality represented by the model. This inconsistency, known as observation representation error, must be accounted for in data assimilation schemes. In this paper the ability of the ensemble to predict the variance of the observation error of representation is explored, arguing that the portion of representation error being detected by the ensemble variance is that portion correlated to the smoothed features that the coarse-resolution forecast model is able to predict. This predictive relationship is explored using differences between model states and their spectrally truncated form, as well as commonly used statistical methods to estimate observation error variances. It is demonstrated that the ensemble variance is a useful predictor of the observation error variance of representation and that it could be used to account for flow dependence in the observation error covariance matrix.

scholarly journals EnKF and 4D-Var data assimilation with chemical transport model BASCOE (version 05.06)

2016 ◽  
Vol 9 (8) ◽  
pp. 2893-2908 ◽  
Author(s):  
Sergey Skachko ◽  
Richard Ménard ◽  
Quentin Errera ◽  
Yves Christophe ◽  
Simon Chabrillat
Keyword(s):  
Data Assimilation ◽  
Transport Model ◽  
Chemical Transport ◽  
Error Variance ◽  
Model Error ◽  
Chemical Processes ◽  
Observation Error ◽  
Chemical Transport Model ◽  
Error Covariance ◽  
Chemical Tracer

Abstract. We compare two optimized chemical data assimilation systems, one based on the ensemble Kalman filter (EnKF) and the other based on four-dimensional variational (4D-Var) data assimilation, using a comprehensive stratospheric chemistry transport model (CTM). This work is an extension of the Belgian Assimilation System for Chemical ObsErvations (BASCOE), initially designed to work with a 4D-Var data assimilation. A strict comparison of both methods in the case of chemical tracer transport was done in a previous study and indicated that both methods provide essentially similar results. In the present work, we assimilate observations of ozone, HCl, HNO3, H2O and N2O from EOS Aura-MLS data into the BASCOE CTM with a full description of stratospheric chemistry. Two new issues related to the use of the full chemistry model with EnKF are taken into account. One issue is a large number of error variance parameters that need to be optimized. We estimate an observation error variance parameter as a function of pressure level for each observed species using the Desroziers method. For comparison purposes, we apply the same estimate procedure in the 4D-Var data assimilation, where both scale factors of the background and observation error covariance matrices are estimated using the Desroziers method. However, in EnKF the background error covariance is modelled using the full chemistry model and a model error term which is tuned using an adjustable parameter. We found that it is adequate to have the same value of this parameter based on the chemical tracer formulation that is applied for all observed species. This is an indication that the main source of model error in chemical transport model is due to the transport. The second issue in EnKF with comprehensive atmospheric chemistry models is the noise in the cross-covariance between species that occurs when species are weakly chemically related at the same location. These errors need to be filtered out in addition to a localization based on distance. The performance of two data assimilation methods was assessed through an 8-month long assimilation of limb sounding observations from EOS Aura MLS. This paper discusses the differences in results and their relation to stratospheric chemical processes. Generally speaking, EnKF and 4D-Var provide results of comparable quality but differ substantially in the presence of model error or observation biases. If the erroneous chemical modelling is associated with moderately fast chemical processes, but whose lifetimes are longer than the model time step, then EnKF performs better, while 4D-Var develops spurious increments in the chemically related species. If, however, the observation biases are significant, then 4D-Var is more robust and is able to reject erroneous observations while EnKF does not.

scholarly journals Formulations for Estimating Spatial Variations of Analysis Error Variance to Improve Multiscale and Multistep Variational Data Assimilation

2018 ◽  
Vol 2018 ◽  
pp. 1-17
Author(s):  
Qin Xu ◽  
Li Wei
Keyword(s):  
Data Assimilation ◽  
Spatial Variation ◽  
Variational Analysis ◽  
Variance Reduction ◽  
Error Variance ◽  
Spatial Variations ◽  
Variational Data Assimilation ◽  
Coarse Resolution ◽  
Error Covariance ◽  
Analysis Error

When the coarse-resolution observations used in the first step of multiscale and multistep variational data assimilation become increasingly nonuniform and/or sparse, the error variance of the first-step analysis tends to have increasingly large spatial variations. However, the analysis error variance computed from the previously developed spectral formulations is constant and thus limited to represent only the spatially averaged error variance. To overcome this limitation, analytic formulations are constructed to efficiently estimate the spatial variation of analysis error variance and associated spatial variation in analysis error covariance. First, a suite of formulations is constructed to efficiently estimate the error variance reduction produced by analyzing the coarse-resolution observations in one- and two-dimensional spaces with increased complexity and generality (from uniformly distributed observations with periodic extension to nonuniformly distributed observations without periodic extension). Then, three different formulations are constructed for using the estimated analysis error variance to modify the analysis error covariance computed from the spectral formulations. The successively improved accuracies of these three formulations and their increasingly positive impacts on the two-step variational analysis (or multistep variational analysis in first two steps) are demonstrated by idealized experiments.

scholarly journals Accounting for Correlated Observation Error in a Dual-Formulation 4D Variational Data Assimilation System

2017 ◽  
Vol 145 (3) ◽  
pp. 1019-1032 ◽  
Author(s):  
William F. Campbell ◽  
Elizabeth A. Satterfield ◽  
Benjamin Ruston ◽  
Nancy L. Baker
Keyword(s):  
Data Assimilation ◽  
Positive Impact ◽  
Error Variance ◽  
Variational Data Assimilation ◽  
Advanced Technology ◽  
Radiosonde Data ◽  
Observation Error ◽  
Error Covariance ◽  
Data Assimilation System ◽  
Assimilation System

Appropriate specification of the error statistics for both observational data and short-term forecasts is necessary to produce an optimal analysis. Observation error stems from instrument error, forward model error, and error of representation. All sources of observation error, particularly error of representation, can lead to nonzero correlations. While correlated forecast error has been accounted for since the early days of atmospheric data assimilation, observation error has typically been treated as uncorrelated until relatively recently. Thinning, averaging, and/or inflation of the assigned observation error variance have been employed to compensate for unaccounted error correlations, especially for high-resolution satellite data. In this study, the benefits of accounting for nonzero vertical (interchannel) correlation for both the Advanced Technology Microwave Satellite (ATMS) and Infrared Atmospheric Sounding Interferometer (IASI) in the NRL Atmospheric Variational Data Assimilation System-Accelerated Representer (NAVDAS-AR) are assessed. The vertical observation error covariance matrix for the ATMS and IASI instruments was estimated using the Desroziers method. The results suggest lowering the assigned error variance and introducing strong correlations, especially in the moisture-sensitive channels. Strong positive impact on forecast skill (verified against both the ECMWF analyses and high-quality radiosonde data) is shown in both the ATMS and IASI instruments. Additionally, the convergence of the iterative solver in NAVDAS-AR can be improved by small modifications to the observation error covariance matrices, resulting in further reduction in RMS error.

A New Approach for Estimating the Observation Impact in Ensemble–Variational Data Assimilation

2018 ◽  
Vol 146 (2) ◽  
pp. 447-465 ◽  
Author(s):  
Mark Buehner ◽  
Ping Du ◽  
Joël Bédard
Keyword(s):  
Data Assimilation ◽  
Forecast Error ◽  
Forecast Model ◽  
Variational Data Assimilation ◽  
Observation Error ◽  
New Approach ◽  
Regional Prediction ◽  
Observation Impact ◽  
Adjoint Approach ◽  
The Impact

Abstract Two types of approaches are commonly used for estimating the impact of arbitrary subsets of observations on short-range forecast error. The first was developed for variational data assimilation systems and requires the adjoint of the forecast model. Comparable approaches were developed for use with the ensemble Kalman filter and rely on ensembles of forecasts. In this study, a new approach for computing observation impact is proposed for ensemble–variational data assimilation (EnVar). Like standard adjoint approaches, the adjoint of the data assimilation procedure is implemented through the iterative minimization of a modified cost function. However, like ensemble approaches, the adjoint of the forecast step is obtained by using an ensemble of forecasts. Numerical experiments were performed to compare the new approach with the standard adjoint approach in the context of operational deterministic NWP. Generally similar results are obtained with both approaches, especially when the new approach uses covariance localization that is horizontally advected between analysis and forecast times. However, large differences in estimated impacts are obtained for some surface observations. Vertical propagation of the observation impact is noticeably restricted with the new approach because of vertical covariance localization. The new approach is used to evaluate changes in observation impact as a result of the use of interchannel observation error correlations for radiance observations. The estimated observation impact in similarly configured global and regional prediction systems is also compared. Overall, the new approach should provide useful estimates of observation impact for data assimilation systems based on EnVar when an adjoint model is not available.

Evaluation of Surface Analyses and Forecasts with a Multiscale Ensemble Kalman Filter in Regions of Complex Terrain

2011 ◽  
Vol 139 (6) ◽  
pp. 2008-2024 ◽  
Author(s):  
Brian C. Ancell ◽  
Clifford F. Mass ◽  
Gregory J. Hakim
Keyword(s):  
Kalman Filter ◽  
High Resolution ◽  
Data Assimilation ◽  
Ensemble Kalman Filter ◽  
Complex Terrain ◽  
Error Variance ◽  
Small Scale ◽  
Observation Error ◽  
Grid Spacing ◽  
Background Error

Abstract Previous research suggests that an ensemble Kalman filter (EnKF) data assimilation and modeling system can produce accurate atmospheric analyses and forecasts at 30–50-km grid spacing. This study examines the ability of a mesoscale EnKF system using multiscale (36/12 km) Weather Research and Forecasting (WRF) model simulations to produce high-resolution, accurate, regional surface analyses, and 6-h forecasts. This study takes place over the complex terrain of the Pacific Northwest, where the small-scale features of the near-surface flow field make the region particularly attractive for testing an EnKF and its flow-dependent background error covariances. A variety of EnKF experiments are performed over a 5-week period to test the impact of decreasing the grid spacing from 36 to 12 km and to evaluate new approaches for dealing with representativeness error, lack of surface background variance, and low-level bias. All verification in this study is performed with independent, unassimilated observations. Significant surface analysis and 6-h forecast improvements are found when EnKF grid spacing is reduced from 36 to 12 km. Forecast improvements appear to be a consequence of increased resolution during model integration, whereas analysis improvements also benefit from high-resolution ensemble covariances during data assimilation. On the 12-km domain, additional analysis improvements are found by reducing observation error variance in order to address representativeness error. Removing model surface biases prior to assimilation significantly enhances the analysis. Inflating surface wind and temperature background error variance has large impacts on analyses, but only produces small improvements in analysis RMS errors. Both surface and upper-air 6-h forecasts are nearly unchanged in the 12-km experiments. Last, 12-km WRF EnKF surface analyses and 6-h forecasts are shown to generally outperform those of the Global Forecast System (GFS), North American Model (NAM), and the Rapid Update Cycle (RUC) by about 10%–30%, although these improvements do not extend above the surface. Based on these results, future improvements in multiscale EnKF are suggested.

scholarly journals Multiscale Representation of Observation Error Statistics in Data Assimilation

Sensors ◽  
2020 ◽  
Vol 20 (5) ◽  
pp. 1460
Author(s):  
Vincent Chabot ◽  
Maëlle Nodet ◽  
Arthur Vidard
Keyword(s):  
Data Assimilation ◽  
Real Life ◽  
Covariance Matrices ◽  
Multiscale Modelling ◽  
Observation Error ◽  
Error Statistics ◽  
Life Data ◽  
Promising Tool ◽  
Error Covariance ◽  
Real Life Data

Accounting for realistic observation errors is a known bottleneck in data assimilation, because dealing with error correlations is complex. Following a previous study on this subject, we propose to use multiscale modelling, more precisely wavelet transform, to address this question. This study aims to investigate the problem further by addressing two issues arising in real-life data assimilation: how to deal with partially missing data (e.g., concealed by an obstacle between the sensor and the observed system), and how to solve convergence issues associated with complex observation error covariance matrices? Two adjustments relying on wavelets modelling are proposed to deal with those, and offer significant improvements. The first one consists of adjusting the variance coefficients in the frequency domain to account for masked information. The second one consists of a gradual assimilation of frequencies. Both of these fully rely on the multiscale properties associated with wavelet covariance modelling. Numerical results on twin experiments show that multiscale modelling is a promising tool to account for correlations in observation errors in realistic applications.

scholarly journals Assimilation of Stratospheric Temperature and Ozone with an Ensemble Kalman Filter in a Chemistry–Climate Model

2011 ◽  
Vol 139 (11) ◽  
pp. 3389-3404 ◽  
Author(s):  
Thomas Milewski ◽  
Michel S. Bourqui
Keyword(s):  
Kalman Filter ◽  
Data Assimilation ◽  
Covariance Matrix ◽  
Ensemble Kalman Filter ◽  
Climate Model ◽  
Observation Error ◽  
Background Error ◽  
Error Covariance Matrix ◽  
Error Covariance ◽  
Analysis Error

Abstract A new stratospheric chemical–dynamical data assimilation system was developed, based upon an ensemble Kalman filter coupled with a Chemistry–Climate Model [i.e., the intermediate-complexity general circulation model Fast Stratospheric Ozone Chemistry (IGCM-FASTOC)], with the aim to explore the potential of chemical–dynamical coupling in stratospheric data assimilation. The system is introduced here in a context of a perfect-model, Observing System Simulation Experiment. The system is found to be sensitive to localization parameters, and in the case of temperature (ozone), assimilation yields its best performance with horizontal and vertical decorrelation lengths of 14 000 km (5600 km) and 70 km (14 km). With these localization parameters, the observation space background-error covariance matrix is underinflated by only 5.9% (overinflated by 2.1%) and the observation-error covariance matrix by only 1.6% (0.5%), which makes artificial inflation unnecessary. Using optimal localization parameters, the skills of the system in constraining the ensemble-average analysis error with respect to the true state is tested when assimilating synthetic Michelson Interferometer for Passive Atmospheric Sounding (MIPAS) retrievals of temperature alone and ozone alone. It is found that in most cases background-error covariances produced from ensemble statistics are able to usefully propagate information from the observed variable to other ones. Chemical–dynamical covariances, and in particular ozone–wind covariances, are essential in constraining the dynamical fields when assimilating ozone only, as the radiation in the stratosphere is too slow to transfer ozone analysis increments to the temperature field over the 24-h forecast window. Conversely, when assimilating temperature, the chemical–dynamical covariances are also found to help constrain the ozone field, though to a much lower extent. The uncertainty in forecast/analysis, as defined by the variability in the ensemble, is large compared to the analysis error, which likely indicates some amount of noise in the covariance terms, while also reducing the risk of filter divergence.

scholarly journals Estimating Observation Error Statistics Using a Robust Filter Method for Data Assimilation

2020 ◽  
Author(s):  
Yue Wang ◽  
Yulong Bai
Keyword(s):  
Data Assimilation ◽  
Error Estimation ◽  
Filter Method ◽  
Robust Filtering ◽  
Observation Error ◽  
Error Statistics ◽  
H Infinity ◽  
Error Covariance ◽  
Lorenz 96 ◽  
Data Assimilation System

For the data assimilation algorithms, the observation error covariance plays an important role, because they control the weight that is given to the model forecast and to the observation in the solution, i.e., the analysis. In order to easily calculate, we often assume observation to be a diagonal matrix, however, the observation errors are correlated to the state and have a certain dependence on time, such as certain observing types which are remotely sensed. In this work, we obtain the time-dependent and correlated observation error by the method of observation error estimation in the data assimilation system. We combine the ensemble time-local H-infinity filter (EnTLHF) with an estimate of observation error covariance matrix, named ensemble time-local H-infinity filter with observation error covariance estimation (EnTLHF-R). In the experiment, a classical nonlinear Lorenz-96 model to evaluate the performance of new method is used. The results show that the robust filtering with observation error estimation is more accurate, more robust, and the filtering is more stable.

scholarly journals A Review of Innovation-Based Methods to Jointly Estimate Model and Observation Error Covariance Matrices in Ensemble Data Assimilation

2020 ◽  
Vol 148 (10) ◽  
pp. 3973-3994 ◽  
Author(s):  
Pierre Tandeo ◽  
Pierre Ailliot ◽  
Marc Bocquet ◽  
Alberto Carrassi ◽  
Takemasa Miyoshi ◽  
...  
Keyword(s):  
Data Assimilation ◽  
Current Data ◽  
Covariance Matrices ◽  
Estimation Methods ◽  
Observation Error ◽  
Correlated Noise ◽  
Unified Framework ◽  
Error Covariance ◽  
Error Terms ◽  
Observation Space

AbstractData assimilation combines forecasts from a numerical model with observations. Most of the current data assimilation algorithms consider the model and observation error terms as additive Gaussian noise, specified by their covariance matrices and , respectively. These error covariances, and specifically their respective amplitudes, determine the weights given to the background (i.e., the model forecasts) and to the observations in the solution of data assimilation algorithms (i.e., the analysis). Consequently, and matrices significantly impact the accuracy of the analysis. This review aims to present and to discuss, with a unified framework, different methods to jointly estimate the and matrices using ensemble-based data assimilation techniques. Most of the methods developed to date use the innovations, defined as differences between the observations and the projection of the forecasts onto the observation space. These methods are based on two main statistical criteria: 1) the method of moments, in which the theoretical and empirical moments of the innovations are assumed to be equal, and 2) methods that use the likelihood of the observations, themselves contained in the innovations. The reviewed methods assume that innovations are Gaussian random variables, although extension to other distributions is possible for likelihood-based methods. The methods also show some differences in terms of levels of complexity and applicability to high-dimensional systems. The conclusion of the review discusses the key challenges to further develop estimation methods for and . These challenges include taking into account time-varying error covariances, using limited observational coverage, estimating additional deterministic error terms, or accounting for correlated noise.

scholarly journals Observation error covariance specification in dynamical systems for data assimilation using recurrent neural networks

2021 ◽  
Author(s):  
Sibo Cheng ◽  
Mingming Qiu
Keyword(s):  
Neural Networks ◽  
Time Series ◽  
Dynamical Systems ◽  
Data Assimilation ◽  
Recurrent Neural Networks ◽  
Time Series Data ◽  
Series Data ◽  
Observation Data ◽  
Observation Error ◽  
Error Covariance

AbstractData assimilation techniques are widely used to predict complex dynamical systems with uncertainties, based on time-series observation data. Error covariance matrices modeling is an important element in data assimilation algorithms which can considerably impact the forecasting accuracy. The estimation of these covariances, which usually relies on empirical assumptions and physical constraints, is often imprecise and computationally expensive, especially for systems of large dimensions. In this work, we propose a data-driven approach based on long short term memory (LSTM) recurrent neural networks (RNN) to improve both the accuracy and the efficiency of observation covariance specification in data assimilation for dynamical systems. Learning the covariance matrix from observed/simulated time-series data, the proposed approach does not require any knowledge or assumption about prior error distribution, unlike classical posterior tuning methods. We have compared the novel approach with two state-of-the-art covariance tuning algorithms, namely DI01 and D05, first in a Lorenz dynamical system and then in a 2D shallow water twin experiments framework with different covariance parameterization using ensemble assimilation. This novel method shows significant advantages in observation covariance specification, assimilation accuracy, and computational efficiency.

Sign in / Sign up

Export Citation Format

Share Document