scholarly journals A Probabilistic Approach to Adaptive Covariance Localization for Serial Ensemble Square Root Filters

2014 ◽  
Vol 142 (12) ◽  
pp. 4499-4518 ◽  
Author(s):  
Yicun Zhen ◽  
Fuqing Zhang
Keyword(s):  
Probabilistic Approach ◽  
Filter Method ◽  
Square Root ◽  
Ensemble Size ◽  
Sampling Errors ◽  
Jeffreys Prior ◽  
Ensemble Kalman Filters ◽  
Localization Radius ◽  
Lorenz 96 ◽  
The Impact

Abstract This study proposes a variational approach to adaptively determine the optimum radius of influence for ensemble covariance localization when uncorrelated observations are assimilated sequentially. The covariance localization is commonly used by various ensemble Kalman filters to limit the impact of covariance sampling errors when the ensemble size is small relative to the dimension of the state. The probabilistic approach is based on the premise of finding an optimum localization radius that minimizes the distance between the Kalman update using the localized sampling covariance versus using the true covariance, when the sequential ensemble Kalman square root filter method is used. The authors first examine the effectiveness of the proposed method for the cases when the true covariance is known or can be approximated by a sufficiently large ensemble size. Not surprisingly, it is found that the smaller the true covariance distance or the smaller the ensemble, the smaller the localization radius that is needed. The authors further generalize the method to the more usual scenario that the true covariance is unknown but can be represented or estimated probabilistically based on the ensemble sampling covariance. The mathematical formula for this probabilistic and adaptive approach with the use of the Jeffreys prior is derived. Promising results and limitations of this new method are discussed through experiments using the Lorenz-96 system.

scholarly journals On the Selection of Localization Radius in Ensemble Filtering for Multiscale Quasigeostrophic Dynamics

2018 ◽  
Vol 146 (2) ◽  
pp. 543-560 ◽  
Author(s):  
Yue Ying ◽  
Fuqing Zhang ◽  
Jeffrey L. Anderson
Keyword(s):  
Correlation Length ◽  
Large Scale ◽  
Observation Error ◽  
State Variables ◽  
Model Resolution ◽  
Ensemble Size ◽  
Sampling Errors ◽  
Localization Radius ◽  
Spectral Variance ◽  
Best Localization

Covariance localization remedies sampling errors due to limited ensemble size in ensemble data assimilation. Previous studies suggest that the optimal localization radius depends on ensemble size, observation density and accuracy, as well as the correlation length scale determined by model dynamics. A comprehensive localization theory for multiscale dynamical systems with varying observation density remains an active area of research. Using a two-layer quasigeostrophic (QG) model, this study systematically evaluates the sensitivity of the best Gaspari–Cohn localization radius to changes in model resolution, ensemble size, and observing networks. Numerical experiment results show that the best localization radius is smaller for smaller-scale components of a QG flow, indicating its scale dependency. The best localization radius is rather insensitive to changes in model resolution, as long as the key dynamical processes are reasonably well represented by the low-resolution model with inflation methods that account for representation errors. As ensemble size decreases, the best localization radius shifts to smaller values. However, for nonlocal correlations between an observation and state variables that peak at a certain distance, decreasing localization radii further within this distance does not reduce analysis errors. Increasing the density of an observing network has two effects that both reduce the best localization radius. First, the reduced observation error spectral variance further constrains prior ensembles at large scales. Less large-scale contribution results in a shorter overall correlation length, which favors a smaller localization radius. Second, a denser network provides more independent pieces of information, thus a smaller localization radius still allows the same number of observations to constrain each state variable.

scholarly journals Mitigating Observation Perturbation Sampling Errors in the Stochastic EnKF

2015 ◽  
Vol 143 (7) ◽  
pp. 2918-2936 ◽  
Author(s):  
I. Hoteit ◽  
D.-T. Pham ◽  
M. E. Gharamti ◽  
X. Luo
Keyword(s):  
Ensemble Kalman Filter ◽  
Forecast Error ◽  
Sampling Errors ◽  
Observational Error ◽  
Error Covariance ◽  
Atmospheric Data ◽  
Observational Errors ◽  
Analysis Error ◽  
Lorenz 96 ◽  
The Impact

Abstract The stochastic ensemble Kalman filter (EnKF) updates its ensemble members with observations perturbed with noise sampled from the distribution of the observational errors. This was shown to introduce noise into the system and may become pronounced when the ensemble size is smaller than the rank of the observational error covariance, which is often the case in real oceanic and atmospheric data assimilation applications. This work introduces an efficient serial scheme to mitigate the impact of observations’ perturbations sampling in the analysis step of the EnKF, which should provide more accurate ensemble estimates of the analysis error covariance matrices. The new scheme is simple to implement within the serial EnKF algorithm, requiring only the approximation of the EnKF sample forecast error covariance matrix by a matrix with one rank less. The new EnKF scheme is implemented and tested with the Lorenz-96 model. Results from numerical experiments are conducted to compare its performance with the EnKF and two standard deterministic EnKFs. This study shows that the new scheme enhances the behavior of the EnKF and may lead to better performance than the deterministic EnKFs even when implemented with relatively small ensembles.

scholarly journals Hybrid Ensemble–Variational Filter: A Spatially and Temporally Varying Adaptive Algorithm to Estimate Relative Weighting

2021 ◽  
Vol 149 (1) ◽  
pp. 65-76
Author(s):  
Mohamad El Gharamti
Keyword(s):  
Adaptive Algorithm ◽  
Random Variable ◽  
Weighting Coefficient ◽  
Model Errors ◽  
Sampling Errors ◽  
Ensemble Kalman Filters ◽  
Constant Weight ◽  
Open Question ◽  
Spatially Varying ◽  
Lorenz 96

AbstractModel errors and sampling errors produce inaccurate sample covariances that limit the performance of ensemble Kalman filters. Linearly hybridizing the flow-dependent ensemble-based covariance with a time-invariant background covariance matrix gives a better estimate of the true error covariance. Previous studies have shown this, both in theory and in practice. How to choose the weight for each covariance remains an open question especially in the presence of model biases. This study assumes the weighting coefficient to be a random variable and then introduces a Bayesian scheme to estimate it using the available data. The scheme takes into account the discrepancy between the ensemble mean and the observations, the ensemble variance, the static background variance, and the uncertainties in the observations. The proposed algorithm is first derived for a spatially constant weight and then this assumption is relaxed by estimating a unique scalar weight for each state variable. Using twin experiments with the 40-variable Lorenz 96 system, it is shown that the proposed scheme is able to produce quality forecasts even in the presence of severe sampling errors. The adaptive algorithm allows the hybrid filter to switch between an EnKF and a simple EnOI depending on the statistics of the ensemble. In the presence of model errors, the adaptive scheme demonstrates additional improvements compared with standard enhancements alone, such as inflation and localization. Finally, the potential of the spatially varying variant to accommodate challenging sparse observation networks is demonstrated. The computational efficiency and storage of the proposed scheme, which remain an obstacle, are discussed.

scholarly journals Comparing Adaptive Prior and Posterior Inflation for Ensemble Filters Using an Atmospheric General Circulation Model

2019 ◽  
Vol 147 (7) ◽  
pp. 2535-2553 ◽  
Author(s):  
Mohamad El Gharamti ◽  
Kevin Raeder ◽  
Jeffrey Anderson ◽  
Xuguang Wang
Keyword(s):  
General Circulation ◽  
Atmospheric General Circulation Model ◽  
Circulation Model ◽  
Model Errors ◽  
Ensemble Size ◽  
Sampling Errors ◽  
Computationally Efficient ◽  
Ensemble Kalman Filters ◽  
Community Atmosphere Model ◽  
Slight Advantage

Abstract Sampling errors and model errors are major drawbacks from which ensemble Kalman filters suffer. Sampling errors arise because of the use of a limited ensemble size, while model errors are deficiencies in the dynamics and underlying parameterizations that may yield biases in the model’s prediction. In this study, we propose a new time-adaptive posterior inflation algorithm in which the analyzed ensemble anomalies are locally inflated. The proposed inflation strategy is computationally efficient and is aimed at restoring enough spread in the analysis ensemble after assimilating the observations. The performance of this scheme is tested against the relaxation to prior spread (RTPS) and adaptive prior inflation. For this purpose, two model are used: the three-variable Lorenz 63 system and the Community Atmosphere Model (CAM). In CAM, global refractivity, temperature, and wind observations from several sources are incorporated to perform a set of assimilation experiments using the Data Assimilation Research Testbed (DART). The proposed scheme is shown to yield better quality forecasts than the RTPS. Assimilation results further suggest that when model errors are small, both prior and posterior inflation are able to mitigate sampling errors with a slight advantage to posterior inflation. When large model errors, such as wind and temperature biases, are present, prior inflation is shown to be more accurate than posterior inflation. Densely observed regions as in the Northern Hemisphere present numerous challenges to the posterior inflation algorithm. A compelling enhancement to the performance of the filter is achieved by combining both adaptive inflation schemes.

scholarly journals Empirical Localization of Observation Impact in Ensemble Kalman Filters

2013 ◽  
Vol 141 (11) ◽  
pp. 4140-4153 ◽  
Author(s):  
Jeffrey Anderson ◽  
Lili Lei
Keyword(s):  
Kalman Filters ◽  
Horizontal Distance ◽  
State Variables ◽  
Sampling Errors ◽  
Ensemble Kalman Filters ◽  
Filter Performance ◽  
State Variable ◽  
The Impact ◽  
Mean Square Errors ◽  
Low Order

Abstract Localization is a method for reducing the impact of sampling errors in ensemble Kalman filters. Here, the regression coefficient, or gain, relating ensemble increments for observed quantity y to increments for state variable x is multiplied by a real number α defined as a localization. Localization of the impact of observations on model state variables is required for good performance when applying ensemble data assimilation to large atmospheric and oceanic problems. Localization also improves performance in idealized low-order ensemble assimilation applications. An algorithm that computes localization from the output of an ensemble observing system simulation experiment (OSSE) is described. The algorithm produces localizations for sets of pairs of observations and state variables: for instance, all state variables that are between 300- and 400-km horizontal distance from an observation. The algorithm is applied in a low-order model to produce localizations from the output of an OSSE and the computed localizations are then used in a new OSSE. Results are compared to assimilations using tuned localizations that are approximately Gaussian functions of the distance between an observation and a state variable. In most cases, the empirically computed localizations produce the lowest root-mean-square errors in subsequent OSSEs. Localizations derived from OSSE output can provide guidance for localization in real assimilation experiments. Applying the algorithm in large geophysical applications may help to tune localization for improved ensemble filter performance.

scholarly journals Accounting for Model Error from Unresolved Scales in Ensemble Kalman Filters by Stochastic Parameterization

2017 ◽  
Vol 145 (9) ◽  
pp. 3709-3723 ◽  
Author(s):  
Fei Lu ◽  
Xuemin Tu ◽  
Alexandre J. Chorin
Keyword(s):  
Kalman Filters ◽  
Numerical Experiments ◽  
Sampling Error ◽  
Forecast Model ◽  
Model Error ◽  
Ensemble Size ◽  
Ensemble Kalman Filters ◽  
Filter Performance ◽  
Performance Results ◽  
Lorenz 96

The use of discrete-time stochastic parameterization to account for model error due to unresolved scales in ensemble Kalman filters is investigated by numerical experiments. The parameterization quantifies the model error and produces an improved non-Markovian forecast model, which generates high quality forecast ensembles and improves filter performance. Results are compared with the methods of dealing with model error through covariance inflation and localization (IL), using as an example the two-layer Lorenz-96 system. The numerical results show that when the ensemble size is sufficiently large, the parameterization is more effective in accounting for the model error than IL; if the ensemble size is small, IL is needed to reduce sampling error, but the parameterization further improves the performance of the filter. This suggests that in real applications where the ensemble size is relatively small, the filter can achieve better performance than pure IL if stochastic parameterization methods are combined with IL.

scholarly journals The diffuse ensemble filter

2009 ◽  
Vol 16 (4) ◽  
pp. 475-486 ◽  
Author(s):  
X. Yang ◽  
T. DelSole
Keyword(s):  
Kalman Filter ◽  
Infinite Variance ◽  
Forecast Errors ◽  
Implicit Assumption ◽  
Square Root ◽  
Ensemble Size ◽  
Ensemble Kalman Filters ◽  
Small Ensemble ◽  
Model Dimension ◽  
Square Root Filter

Abstract. A new class of ensemble filters, called the Diffuse Ensemble Filter (DEnF), is proposed in this paper. The DEnF assumes that the forecast errors orthogonal to the first guess ensemble are uncorrelated with the latter ensemble and have infinite variance. The assumption of infinite variance corresponds to the limit of "complete lack of knowledge" and differs dramatically from the implicit assumption made in most other ensemble filters, which is that the forecast errors orthogonal to the first guess ensemble have vanishing errors. The DEnF is independent of the detailed covariances assumed in the space orthogonal to the ensemble space, and reduces to conventional ensemble square root filters when the number of ensembles exceeds the model dimension. The DEnF is well defined only in data rich regimes and involves the inversion of relatively large matrices, although this barrier might be circumvented by variational methods. Two algorithms for solving the DEnF, namely the Diffuse Ensemble Kalman Filter (DEnKF) and the Diffuse Ensemble Transform Kalman Filter (DETKF), are proposed and found to give comparable results. These filters generally converge to the traditional EnKF and ETKF, respectively, when the ensemble size exceeds the model dimension. Numerical experiments demonstrate that the DEnF eliminates filter collapse, which occurs in ensemble Kalman filters for small ensemble sizes. Also, the use of the DEnF to initialize a conventional square root filter dramatically accelerates the spin-up time for convergence. However, in a perfect model scenario, the DEnF produces larger errors than ensemble square root filters that have covariance localization and inflation. For imperfect forecast models, the DEnF produces smaller errors than the ensemble square root filter with inflation. These experiments suggest that the DEnF has some advantages relative to the ensemble square root filters in the regime of small ensemble size, imperfect model, and copious observations.

scholarly journals The Square Root Information Increment Ensemble Filter

2016 ◽  
Vol 144 (12) ◽  
pp. 4667-4686
Author(s):  
Mark L. Psiaki
Keyword(s):  
Kalman Filters ◽  
A Priori ◽  
Full Rank ◽  
Forecast Model ◽  
Model Error ◽  
The State ◽  
Low Rank ◽  
Square Root ◽  
Ensemble Kalman Filters ◽  
Lorenz 96

Abstract A new type of ensemble filter is developed, one that stores and updates its state information in an efficient square root information filter form. It addresses two shortcomings of conventional ensemble Kalman filters: the coarse characterization of random forecast model error effects and the overly optimistic approximation of the estimation error statistics. The new filter uses an assumed a priori covariance approximation that is full rank but sparse, possibly with a dense low-rank increment. This matrix can be used to develop a nominal square root information equation for the system state and uncertainty. The measurements are used to develop an additional low-rank square root information equation. New algorithms provide forecasts and analyses of these increments at a computational cost comparable to that of existing ensemble Kalman filters. Model error effects are implicit in the a priori covariance time history, thereby obviating one of the reasons for including an inflation operation. The use of an a priori full-rank covariance allows the analysis operations to improve the state estimate without the need for a localization adjustment. This new filter exhibited worse performance than a typical covariance square root ensemble Kalman filter when operating on the Lorenz-96 problem in a chaotic regime. It excelled on a version of the Lorenz-96 problem where nonlinearities in the forecast model were weak, where the state vector uncertainty lay predominantly in a small subspace, and where the observations were spatially sparse. Such a problem might be representative of ionospheric space weather data assimilation where forcing variability can dominate the state uncertainty and where remote sensing data coverage can be sparse.

scholarly journals Probabilistic assessment of pile capacity based on CPTu probing including random pile foundation depth

2018 ◽  
Vol 196 ◽  
pp. 01058 ◽  
Author(s):  
Marek Wyjadłowski ◽  
Irena Bagińska ◽  
Jakub Reiner
Keyword(s):  
Bearing Capacity ◽  
Objective Assessment ◽  
Probabilistic Approach ◽  
Measurement Data ◽  
Direct Methods ◽  
Side Surface ◽  
Design Stage ◽  
Direct Data ◽  
Design Calculations ◽  
The Impact

The modern recognition of subsoil with the use of CPTu static probes allows to obtain detailed information necessary for the designing. Registered basic two quantities, i.e. cone resistance qc and friction on the sleeve fs, often become direct data, which allow to estimate bearing capacity of the base and the side surface of the pile. Direct methods use similarity of the pile work and piezo-cone work during the examination. An important design stage is the appropriate development of measurement data prior to the commencement of the procedure of determining the pile bearing capacity. Algorithms generated on the basis of empirical experiments are often applied with the simultaneous use of test loads. The probabilistic approach is also significant, because it enables objective assessment of the reliability level of performed design calculations. This work contains an analysis of the impact on the estimated bearing capacity and reliability of a pile of variable random depth of the pile base. It also includes the determination of probabilities of obtaining the assumed safety index for the designed solution at random foundation depth.

scholarly journals Perturbation analysis of a variable M/M/1 queue: a probabilistic approach

2006 ◽  
Vol 38 (01) ◽  
pp. 263-283 ◽  
Author(s):  
Nelson Antunes ◽  
Christine Fricker ◽  
Fabrice Guillemin ◽  
Philippe Robert
Keyword(s):  
Power Series ◽  
Busy Period ◽  
Stationary Process ◽  
Probabilistic Approach ◽  
Power Series Expansion ◽  
Mean Value ◽  
Service Rate ◽  
The Mean ◽  
The Impact ◽  
Period Duration

In this paper, motivated by the problem of the coexistence on transmission links of telecommunications networks of elastic and unresponsive traffic, we study the impact on the busy period of an M/M/1 queue of a small perturbation in the service rate. The perturbation depends upon an independent stationary process (X(t)) and is quantified by means of a parameter ε ≪ 1. We specifically compute the two first terms of the power series expansion in ε of the mean value of the busy period duration. This allows us to study the validity of the reduced service rate approximation, which consists in comparing the perturbed M/M/1 queue with the M/M/1 queue whose service rate is constant and equal to the mean value of the perturbation. For the first term of the expansion, the two systems are equivalent. For the second term, the situation is more complex and it is shown that the correlations of the environment process (X(t)) play a key role.

Sign in / Sign up

Export Citation Format

Share Document