Data assimilation challenges posed by nonlinear operators: A comparative study of ensemble and variational filters and smoothers

2021 ◽  
Author(s):  
Kenta Kurosawa ◽  
Jonathan Poterjoy
Keyword(s):  
Data Assimilation ◽  
Weather Prediction ◽  
Ensemble Methods ◽  
Current Data ◽  
Bayesian Filtering ◽  
Posterior Density ◽  
Nonlinear Operators ◽  
Linear System Of Equations ◽  
Multiple Data ◽  
Non Gaussian

AbstractThe ensemble Kalman Filter (EnKF) and the 4D variational method (4DVar) are the most commonly used filters and smoothers in atmospheric science. These methods typically approximate prior densities using a Gaussian and solve a linear system of equations for the posterior mean and covariance. Therefore, strongly nonlinear model dynamics and measurement operators can lead to bias in posterior estimates. To improve the performance in nonlinear regimes, minimization of the 4DVar cost function typically follows multiple sets of iterations, known as an “outer loop”, which helps reduce bias caused by linear assumptions. Alternatively, “iterative ensemble methods” follow a similar strategy of periodically re-linearizing model and measurement operators. These methods come with different, possibly more appropriate, assumptions for drawing samples from the posterior density, but have seen little attention in numerical weather prediction (NWP) communities. Lastly, particle filters (PFs) present a purely Bayesian filtering approach for state estimation, which avoids many of the assumptions made by the above methods. Several strategies for applying localized PFs for NWP have been proposed very recently. The current study investigates intrinsic limitations of current data assimilation methodology for applications that require nonlinear measurement operators. In doing so, it targets a specific problem that is relevant to the assimilation of remotely-sensed measurements, such as radar reflectivity and all-sky radiances, which pose challenges for Gaussian-based data assimilation systems. This comparison includes multiple data assimilation approaches designed recently for nonlinear/non-Gaussian applications, as well as those currently used for NWP.

scholarly journals Scale-Dependent Representation of the Information Content of Observations in the Global Ensemble Kalman Filter Data Assimilation

2016 ◽  
Vol 144 (8) ◽  
pp. 2927-2945
Author(s):  
Nedjeljka Žagar ◽  
Jeffrey Anderson ◽  
Nancy Collins ◽  
Timothy Hoar ◽  
Kevin Raeder ◽  
...  
Keyword(s):  
Kalman Filter ◽  
Data Assimilation ◽  
Information Content ◽  
Ensemble Kalman Filter ◽  
Large Scale ◽  
Variance Reduction ◽  
Weather Prediction ◽  
Current Data ◽  
Model Framework ◽  
The Tropics

Abstract Global data assimilation systems for numerical weather prediction (NWP) are characterized by significant uncertainties in tropical analysis fields. Furthermore, the largest spread of global ensemble forecasts in the short range on all scales is in the tropics. The presented results suggest that these properties hold even in the perfect-model framework and the ensemble Kalman filter data assimilation with a globally homogeneous network of wind and temperature profiles. The reasons for this are discussed by using the modal analysis, which provides information about the scale dependency of analysis and forecast uncertainties and information about the efficiency of data assimilation to reduce the prior uncertainties in the balanced and inertio-gravity dynamics. The scale-dependent representation of variance reduction of the prior ensemble by the data assimilation shows that the peak efficiency of data assimilation is on the synoptic scales in the midlatitudes that are associated with quasigeostrophic dynamics. In contrast, the variance associated with the inertia–gravity modes is less successfully reduced on all scales. A smaller information content of observations on planetary scales with respect to the synoptic scales is discussed in relation to the large-scale tropical uncertainties that current data assimilation methodologies do not address successfully. In addition, it is shown that a smaller reduction of the large-scale uncertainties in the prior state for NWP in the tropics than in the midlatitudes is influenced by the applied radius for the covariance localization.

scholarly journals Convective-Scale Data Assimilation for the Weather Research and Forecasting Model Using the Local Particle Filter

2017 ◽  
Vol 145 (5) ◽  
pp. 1897-1918 ◽  
Author(s):  
Jonathan Poterjoy ◽  
Ryan A. Sobash ◽  
Jeffrey L. Anderson
Keyword(s):  
Kalman Filter ◽  
Data Assimilation ◽  
Ensemble Kalman Filter ◽  
Weather Prediction ◽  
Squall Line ◽  
Monte Carlo Data ◽  
Mixing Ratios ◽  
Convective Scale ◽  
Non Gaussian ◽  
Mean Square Errors

Abstract Particle filters (PFs) are Monte Carlo data assimilation techniques that operate with no parametric assumptions for prior and posterior errors. A data assimilation method introduced recently, called the local PF, approximates the PF solution within neighborhoods of observations, thus allowing for its use in high-dimensional systems. The current study explores the potential of the local PF for atmospheric data assimilation through cloud-permitting numerical experiments performed for an idealized squall line. Using only 100 ensemble members, experiments using the local PF to assimilate simulated radar measurements demonstrate that the method provides accurate analyses at a cost comparable to ensemble filters currently used in weather models. Comparisons between the local PF and an ensemble Kalman filter demonstrate benefits of the local PF for producing probabilistic analyses of non-Gaussian variables, such as hydrometeor mixing ratios. The local PF also provides more accurate forecasts than the ensemble Kalman filter, despite yielding higher posterior root-mean-square errors. A major advantage of the local PF comes from its ability to produce more physically consistent posterior members than the ensemble Kalman filter, which leads to fewer spurious model adjustments during forecasts. This manuscript presents the first successful application of the local PF in a weather prediction model and discusses implications for real applications where nonlinear measurement operators and nonlinear model processes limit the effectiveness of current Gaussian data assimilation techniques.

A Non-Gaussian Ensemble Filter Update for Data Assimilation

2010 ◽  
Vol 138 (11) ◽  
pp. 4186-4198 ◽  
Author(s):  
Jeffrey L. Anderson
Keyword(s):  
Kalman Filter ◽  
Data Assimilation ◽  
Ensemble Kalman Filter ◽  
Sampling Error ◽  
Weather Prediction ◽  
Large Ensembles ◽  
Stochastic Filter ◽  
Pathological Behavior ◽  
Non Gaussian ◽  
Observation Space

Abstract A deterministic square root ensemble Kalman filter and a stochastic perturbed observation ensemble Kalman filter are used for data assimilation in both linear and nonlinear single variable dynamical systems. For the linear system, the deterministic filter is simply a method for computing the Kalman filter and is optimal while the stochastic filter has suboptimal performance due to sampling error. For the nonlinear system, the deterministic filter has increasing error as ensemble size increases because all ensemble members but one become tightly clustered. In this case, the stochastic filter performs better for sufficiently large ensembles. A new method for computing ensemble increments in observation space is proposed that does not suffer from the pathological behavior of the deterministic filter while avoiding much of the sampling error of the stochastic filter. This filter uses the order statistics of the prior observation space ensemble to create an approximate continuous prior probability distribution in a fashion analogous to the use of rank histograms for ensemble forecast evaluation. This rank histogram filter can represent non-Gaussian observation space priors and posteriors and is shown to be competitive with existing filters for problems as large as global numerical weather prediction. The ability to represent non-Gaussian distributions is useful for a variety of applications such as convective-scale assimilation and assimilation of bounded quantities such as relative humidity.

scholarly journals Linking big models to big data: efficient ecosystem model calibration through Bayesian model emulation

2018 ◽  
Author(s):  
Istem Fer ◽  
Ryan Kelly ◽  
Paul R. Moorcroft ◽  
Andrew D. Richardson ◽  
Elizabeth M. Cowdery ◽  
...  
Keyword(s):  
Data Assimilation ◽  
Bayesian Model ◽  
Performance Metrics ◽  
Critical Role ◽  
Ecosystem Dynamics ◽  
Model Assessment ◽  
Posterior Density ◽  
True Parameter ◽  
Multiple Data ◽  
Data Constraints

Abstract. Data-model integration plays a critical role in assessing and improving our capacity to predict ecosystem dynamics. Similarly, the ability to attach quantitative statements of uncertainty around model forecasts is crucial for model assessment and interpretation and for setting field research priorities. Bayesian methods provide a rigorous data assimilation framework for these applications, especially for problems with multiple data constraints. However, the Markov Chain Monte Carlo (MCMC) techniques underlying most Bayesian calibration can be prohibitive for computationally-demanding models and large data sets. We describe an alternative method, Bayesian model emulation of sufficient statistics, that can approximate the full joint posterior density, is more amenable to parallelization, and provides an estimate of parameter sensitivity. Analysis involved informative priors constructed from a meta-analysis of the primary literature, and introduced novel approaches to the specification of both model and data uncertainties, including bias and autocorrelation corrections on multiple data streams. We report the integration of this method within an ecological workflow management software, Predictive Ecosystem Analyzer (PEcAn), and its application and validation with two process-based terrestrial ecosystem models: SIPNET and ED2. In a test against a synthetic dataset, the emulator was able to retrieve the true parameter values. A comparison of the emulator approach to standard "bruteforce" MCMC involving multiple data constraints showed that the emulator method was able to constrain the faster and simpler SIPNET model’s parameters with comparable performance to the bruteforce approach, but reduced computation time by more than two orders of magnitude. The emulator was then applied to calibration of the ED2 model, whose complexity precludes standard (bruteforce) Bayesian data assimilation techniques. Both models are constrained after assimilation of the observational data with the emulator method, reducing the uncertainty around their predictions. Performance metrics showed increased agreement between model predictions and data. Our study furthers efforts toward reducing model uncertainties showing that the emulator method makes it possible to efficiently calibrate complex models. This efficient data assimilation method allows us to conduct more calibration experiments in relatively much shorter times, enabling constraining of numerous models using the expanding amount and types of data.

The Spire TEC Environment Assimilation Model (STEAM)

2020 ◽  
Author(s):  
Matthew Angling ◽  
Francois-Xavier Bocquet ◽  
German Olivares-Pulido ◽  
Sanita Vetra-Carvalho ◽  
Karl Nordstrom ◽  
...  
Keyword(s):  
Data Assimilation ◽  
Weather Prediction ◽  
Grid Point ◽  
Ensemble Methods ◽  
Electron Content ◽  
Combined System ◽  
Background Error ◽  
Total Electron ◽  
Wide Range ◽  
Assimilation Model

<p>The ionosphere can affect a wide range of radio frequency (RF) systems operating below 2 GHz. One option for mitigating these effects is to produce assimilative models of the ionospheric density from which products can be derived for specific systems. Such models aim to optimally combine a background model of the ionospheric state with measurements of the ionosphere. This approach is analogous to the use of numerical weather prediction in the meteorological community, and has been evolving for ionospheric use for the last 10 to 15 years.</p><p>Published research has demonstrated the utility of this approach. However, obstacles to providing effective data products remain due to the sparseness of ionospheric data over large parts of the world and the timeliness with which data are available. Spire is working to overcome these issues through the use of its large constellation of satellites that can measure Total Electron Content (TEC) data in both zenith looking and radio occultation (RO) geometries and its large ground station network that will allow low data latency.</p><p>Spire data will be combined with an innovative data assimilation model (the Spire TEC Environment Assimilation Model, STEAM) to provide accurate and actionable ionospheric products. Data assimilation is required to overcome the limitations and assumptions of the traditional Abel Transform analysis of RO data (i.e., spherical symmetry; transmitter and receiver in free space and the same plane) and to effectively combine RO data, topside data, ground-based GNSS data, and other sources of ionospheric information (i.e., ionosondes).</p><p>STEAM uses a 4D Local ensemble transform Kalman Filter (LETKF). As with other ensemble methods, the LETKF uses an ensemble of models to approximate the background error covariance matrix. However, the LETKF provides a more efficient way to solve the ensemble equations. Furthermore, 4D operation permits the use of data with varying latency. Localisation means that grid points are only modified by data within a local volume; this restricts spurious long-range spatial correlations and means that the ensemble only has to span the space locally. The LETKF transforms the problem into ensemble space which makes each grid point independent, resulting in an algorithm that is easily parallelised.</p><p>This paper will describe the data collection and processing chain, the data assimilation model, and plans for the ongoing development of the combined system. </p>

A new procedure for generating data covariance inflation factors for ensemble smoother with multiple data assimilation

2021 ◽  
Vol 150 ◽  
pp. 104722
Author(s):  
Thiago M.D. Silva ◽  
Sinesio Pesco ◽  
Abelardo Barreto Jr. ◽  
Mustafa Onur
Keyword(s):  
Data Assimilation ◽  
Multiple Data ◽  
Ensemble Smoother

scholarly journals Use of Microwave Radiances from Metop-C and Fengyun-3 C/D Satellites for a Northern European Limited-area Data Assimilation System

2021 ◽  
Author(s):  
Magnus Lindskog ◽  
Adam Dybbroe ◽  
Roger Randriamampianina
Keyword(s):  
Data Assimilation ◽  
Positive Impact ◽  
Weather Prediction ◽  
Implementation Process ◽  
Satellite System ◽  
Limited Area ◽  
Global Navigation Satellite ◽  
Microwave Radiances ◽  
Data Assimilation System ◽  
Assimilation System

AbstractMetCoOp is a Nordic collaboration on operational Numerical Weather Prediction based on a common limited-area km-scale ensemble system. The initial states are produced using a 3-dimensional variational data assimilation scheme utilizing a large amount of observations from conventional in-situ measurements, weather radars, global navigation satellite system, advanced scatterometer data and satellite radiances from various satellite platforms. A version of the forecasting system which is aimed for future operations has been prepared for an enhanced assimilation of microwave radiances. This enhanced data assimilation system will use radiances from the Microwave Humidity Sounder, the Advanced Microwave Sounding Unit-A and the Micro-Wave Humidity Sounder-2 instruments on-board the Metop-C and Fengyun-3 C/D polar orbiting satellites. The implementation process includes channel selection, set-up of an adaptive bias correction procedure, and careful monitoring of data usage and quality control of observations. The benefit of the additional microwave observations in terms of data coverage and impact on analyses, as derived using the degree of freedom of signal approach, is demonstrated. A positive impact on forecast quality is shown, and the effect on the precipitation for a case study is examined. Finally, the role of enhanced data assimilation techniques and adaptions towards nowcasting are discussed.

scholarly journals Evaluation of Operational Monsoon Moisture Surveillance and Severe Weather Prediction Utilizing COSMIC-2/FORMOSAT-7 Radio Occultation Observations

2021 ◽  
Vol 13 (15) ◽  
pp. 2979
Author(s):  
Yu-Chun Chen ◽  
Chih-Chien Tsai ◽  
Yi-Chao Wu ◽  
An-Hsiang Wang ◽  
Chieh-Ju Wang ◽  
...  
Keyword(s):  
Numerical Model ◽  
Data Assimilation ◽  
Water Resource Management ◽  
Radio Occultation ◽  
Positive Impact ◽  
Weather Prediction ◽  
Severe Weather ◽  
Model Experiments ◽  
Typhoon Structure ◽  
Numerical Model Experiments

Operational monsoon moisture surveillance and severe weather prediction is essential for timely water resource management and disaster risk reduction. For these purposes, this study suggests a moisture indicator using the COSMIC-2/FORMOSAT-7 radio occultation (RO) observations and evaluates numerical model experiments with RO data assimilation. The RO data quality is validated by a comparison between sampled RO profiles and nearby radiosonde profiles around Taiwan prior to the experiments. The suggested moisture indicator accurately monitors daily moisture variations in the South China Sea and the Bay of Bengal throughout the 2020 monsoon rainy season. For the numerical model experiments, the statistics of 152 moisture and rainfall forecasts for the 2020 Meiyu season in Taiwan show a neutral to slightly positive impact brought by RO data assimilation. A forecast sample with the most significant improvement reveals that both thermodynamic and dynamic fields are appropriately adjusted by model integration posterior to data assimilation. The statistics of 17 track forecasts for typhoon Hagupit (2020) also show the positive effect of RO data assimilation. A forecast sample reveals that the member with RO data assimilation simulates better typhoon structure and intensity than the member without, and the effect can be larger and faster via multi-cycle RO data assimilation.

scholarly journals Efficient Dimensionality Reduction Methods in Reservoir History Matching

Energies ◽  
2021 ◽  
Vol 14 (11) ◽  
pp. 3137
Author(s):  
Amine Tadjer ◽  
Reider B. Bratvold ◽  
Remus G. Hanea
Keyword(s):  
Data Assimilation ◽  
Dimensionality Reduction ◽  
Gaussian Process ◽  
Latent Variable ◽  
History Matching ◽  
Production Performance ◽  
Latent Variable Model ◽  
Variable Model ◽  
Multiple Data ◽  
Ensemble Smoother

Production forecasting is the basis for decision making in the oil and gas industry, and can be quite challenging, especially in terms of complex geological modeling of the subsurface. To help solve this problem, assisted history matching built on ensemble-based analysis such as the ensemble smoother and ensemble Kalman filter is useful in estimating models that preserve geological realism and have predictive capabilities. These methods tend, however, to be computationally demanding, as they require a large ensemble size for stable convergence. In this paper, we propose a novel method of uncertainty quantification and reservoir model calibration with much-reduced computation time. This approach is based on a sequential combination of nonlinear dimensionality reduction techniques: t-distributed stochastic neighbor embedding or the Gaussian process latent variable model and clustering K-means, along with the data assimilation method ensemble smoother with multiple data assimilation. The cluster analysis with t-distributed stochastic neighbor embedding and Gaussian process latent variable model is used to reduce the number of initial geostatistical realizations and select a set of optimal reservoir models that have similar production performance to the reference model. We then apply ensemble smoother with multiple data assimilation for providing reliable assimilation results. Experimental results based on the Brugge field case data verify the efficiency of the proposed approach.

Sign in / Sign up

Export Citation Format

Share Document