On variational data assimilation for estimating the model initial conditions and emission fluxes for short-term forecasting of SOx concentrations

<p>This paper addresses representation learning for the resolution of inverse problems&#160; with geophysical dynamics. Among others, examples of inverse problems of interest include space-time interpolation, short-term forecasting, conditional simulation w.r.t. available observations, downscaling problems&#8230; From a methodological point of view, we rely on a variational data assimilation framework. Data assimilation (DA) aims to reconstruct the time evolution of some state given a series of&#160; observations, possibly noisy and irregularly-sampled. Here, we investigate DA from a machine learning point of view backed by an underlying variational representation.&#160; Using automatic differentiation tools embedded in deep learning frameworks, we introduce end-to-end neural network architectures for variational data assimilation. It comprises two key components: a variational model and a gradient-based solver both implemented as neural networks. A key feature of the proposed end-to-end learning architecture is that we may train the neural networks models using both supervised and unsupervised strategies. We first illustrate applications to the reconstruction of Lorenz-63 and Lorenz-96 systems from partial and noisy observations. Whereas the gain issued from the supervised learning setting emphasizes the relevance of groundtruthed observation dataset for real-world case-studies, these results also suggest new means to design data assimilation models from data. Especially, they suggest that learning task-oriented representations of the underlying dynamics may be beneficial. We further discuss applications to short-term forecasting and sampling design along with preliminary results for the reconstruction of sea surface currents from satellite altimetry data.&#160;</p><p>This abstract is supported by a preprint available online: https://arxiv.org/abs/2007.12941</p>

Download Full-text

Implementation of a 3-D-Var system for atmospheric profiling data assimilation into the RAMS model: initial results

Atmospheric Measurement Techniques Discussions ◽

10.5194/amtd-6-3581-2013 ◽

2013 ◽

Vol 6 (2) ◽

pp. 3581-3610

Author(s):

S. Federico

Keyword(s):

Relative Humidity ◽

Data Assimilation ◽

Initial Conditions ◽

Weather Prediction ◽

Wind Profiler ◽

Short Term ◽

Recursive Filters ◽

Data Assimilation System ◽

Rams Model ◽

Assimilation System

Abstract. This paper presents the current status of development of a three-dimensional variational data assimilation system. The system can be used with different numerical weather prediction models, but it is mainly designed to be coupled with the Regional Atmospheric Modelling System (RAMS). Analyses are given for the following parameters: zonal and meridional wind components, temperature, relative humidity, and geopotential height. Important features of the data assimilation system are the use of incremental formulation of the cost-function, and the use of an analysis space represented by recursive filters and eigenmodes of the vertical background error matrix. This matrix and the length-scale of the recursive filters are estimated by the National Meteorological Center (NMC) method. The data assimilation and forecasting system is applied to the real context of atmospheric profiling data assimilation, and in particular to the short-term wind prediction. The analyses are produced at 20 km horizontal resolution over central Europe and extend over the whole troposphere. Assimilated data are vertical soundings of wind, temperature, and relative humidity from radiosondes, and wind measurements of the European wind profiler network. Results show the validity of the analysis solutions because they are closer to the observations (lower RMSE) compared to the background (higher RMSE), and the differences of the RMSEs are consistent with the data assimilation settings. To quantify the impact of improved initial conditions on the short-term forecast, the analyses are used as initial conditions of a three-hours forecast of the RAMS model. In particular two sets of forecasts are produced: (a) the first uses the ECMWF analysis/forecast cycle as initial and boundary conditions; (b) the second uses the analyses produced by the 3-D-Var scheme as initial conditions, then is driven by the ECMWF forecast. The improvement is quantified by considering the horizontal components of the wind, which are measured at a-synoptic times by the European wind profiler network. The results show that the RMSE is effectively reduced at the short range (1–2 h). The results are in agreement with the set-up of the numerical experiment.

Download Full-text

Using adjoint sensitivity as a local structure function in variational data assimilation

Nonlinear Processes in Geophysics ◽

10.5194/npg-8-347-2001 ◽

2001 ◽

Vol 8 (6) ◽

pp. 347-355 ◽

Cited By ~ 10

Author(s):

G. Hello ◽

F. Bouttier

Keyword(s):

Data Assimilation ◽

Structure Function ◽

Local Structure ◽

Initial Conditions ◽

Storm Track ◽

Variational Data Assimilation ◽

Sensitive Area ◽

Adjoint Sensitivity ◽

Weather Events ◽

Gradient Fields

Abstract. One approach recently proposed in order to improve the forecast of weather events, such as cyclogenesis, is to increase the number of observations in areas depending on the flow configuration. These areas are obtained using, for example, the sensitivity to initial conditions of a selected predicted cyclone. An alternative or complementary way is proposed here. The idea is to employ such an adjoint sensitivity field as a local structure function within variational data assimilation, 3D-Var in this instance. Away from the sensitive area, observation increments project on the initial fields with the usual climatological (or weakly flow-dependent, in the case of 4D-Var) structure functions. Within the sensitive area, the gradient fields are projected using all the available data in the zone, conventional or extra, if any. The formulation of the technique is given and the approach is further explained by using a simple 1D scheme. The technique is implemented in the ARPEGE/IFS code and applied to 11 FASTEX (Fronts and Atlantic Storm-Track Experiment) cyclone cases, together with the targeted observations performed at the time of the campaign. The new approach is shown to allow for the desired stronger impact of the available observations and to systematically improve the forecasts of the FASTEX cyclones, unlike the standard 3D-Var.

Download Full-text

Impact of Lightning Data Assimilation on Forecasts of a Leeward Slope Precipitation Event in the Western Margin of the Junggar Basin

Remote Sensing ◽

10.3390/rs13183584 ◽

2021 ◽

Vol 13 (18) ◽

pp. 3584

Author(s):

Peng Liu ◽

Yi Yang ◽

Yu Xin ◽

Chenghai Wang

Keyword(s):

Water Vapor ◽

Data Assimilation ◽

Three Dimensional ◽

Junggar Basin ◽

Precipitation Event ◽

Short Term ◽

Western Margin ◽

Precipitation Prediction ◽

Analysis Experiment ◽

Short Term Forecasting

A moderate precipitation event occurring in northern Xinjiang, a region with a continental climate with little rainfall, and in leeward slope areas influenced by topography is important but rarely studied. In this study, the performance of lightning data assimilation is evaluated in the short-term forecasting of a moderate precipitation event along the western margin of the Junggar Basin and eastern Jayer Mountain. Pseudo-water vapor observations driven by lightning data are assimilated in both single and cycling analysis experiments of the Weather Research and Forecast (WRF) three-dimensional variational (3DVAR) system. Lightning data assimilation yields a larger increment in the relative humidity in the analysis field at the observed lightning locations, and the largest increment is obtained in the cycling analysis experiment. Due to the increase in water vapor content in the analysis field, more suitable thermal and dynamic conditions for moderate precipitation are obtained on the leeward slope, and the ice-phase and raindrop particle contents increase in the forecast field. Lightning data assimilation significantly improves the short-term leeward slope moderate precipitation prediction along the western margin of the Junggar Basin and provides the best forecast skill in cycling analysis experiments.

Download Full-text

Simulation of Monsoon Depressions Using WRF-VAR: Impact of Different Background Error Statistics and Lateral Boundary Conditions

Monthly Weather Review ◽

10.1175/mwr-d-13-00285.1 ◽

2014 ◽

Vol 142 (10) ◽

pp. 3586-3613 ◽

Cited By ~ 20

Author(s):

A. Routray ◽

S. C. Kar ◽

P. Mali ◽

K. Sowjanya

Keyword(s):

Data Assimilation ◽

Boundary Conditions ◽

Initial Conditions ◽

Variational Data Assimilation ◽

Error Statistics ◽

Background Error ◽

Single Observation ◽

Monsoon Depressions ◽

Data Assimilation System ◽

Assimilation System

Abstract In a variational data assimilation system, background error statistics (BES) spread the influence of the observations in space and filter analysis increments through dynamic balance or statistical relationships. In a data-sparse region such as the Bay of Bengal, BES play an important role in defining the location and structure of monsoon depressions (MDs). In this study, the Indian-region-specific BES have been computed for the Weather Research and Forecasting (WRF) three-dimensional variational data assimilation system. A comparative study using single observation tests is carried out using the computed BES and global BES within the WRF system. Both sets of BES are used in the assimilation cycles and forecast runs for simulating the meteorological features associated with the MDs. Numerical experiments have been conducted to assess the relative impact of various BES in the analysis and simulations of the MDs. The results show that use of regional BES in the assimilation cycle has a positive impact on the prediction of the location, propagation, and development of rainbands associated with the MDs. The track errors of MDs are smaller when domain-specific BES are used in the assimilation cycle. Additional experiments have been conducted using data from the Interim European Centre for Medium-Range Weather Forecasts Re-Analysis (ERA-Interim) as initial and boundary conditions (IBCs) in the assimilation cycle. The results indicate that the use of domain-dependent BES and high-resolution ERA-I data as IBCs further improved the initial conditions for the model leading to better forecasts of the MDs.

Download Full-text

CIRA/CSU Four-Dimensional Variational Data Assimilation System

Monthly Weather Review ◽

10.1175/mwr2891.1 ◽

2005 ◽

Vol 133 (4) ◽

pp. 829-843 ◽

Cited By ~ 39

Author(s):

Milija Zupanski ◽

Dusanka Zupanski ◽

Tomislava Vukicevic ◽

Kenneth Eis ◽

Thomas Vonder Haar

Keyword(s):

Data Assimilation ◽

Initial Conditions ◽

Control Variable ◽

Three Dimensional ◽

Atmospheric Modeling ◽

Variational Data Assimilation ◽

Regional Atmospheric Modeling ◽

The Impact ◽

Data Assimilation System ◽

Assimilation System

A new four-dimensional variational data assimilation (4DVAR) system is developed at the Cooperative Institute for Research in the Atmosphere (CIRA)/Colorado State University (CSU). The system is also called the Regional Atmospheric Modeling Data Assimilation System (RAMDAS). In its present form, the 4DVAR system is employing the CSU/Regional Atmospheric Modeling System (RAMS) nonhydrostatic primitive equation model. The Weather Research and Forecasting (WRF) observation operator is used to access the observations, adopted from the WRF three-dimensional variational data assimilation (3DVAR) algorithm. In addition to the initial conditions adjustment, the RAMDAS includes the adjustment of model error (bias) and lateral boundary conditions through an augmented control variable definition. Also, the control variable is defined in terms of the velocity potential and streamfunction instead of the horizontal winds. The RAMDAS is developed after the National Centers for Environmental Prediction (NCEP) Eta 4DVAR system, however with added improvements addressing its use in a research environment. Preliminary results with RAMDAS are presented, focusing on the minimization performance and the impact of vertical correlations in error covariance modeling. A three-dimensional formulation of the background error correlation is introduced and evaluated. The Hessian preconditioning is revisited, and an alternate algebraic formulation is presented. The results indicate a robust minimization performance.

Download Full-text

Strong Solvability of a Variational Data Assimilation Problem for the Primitive Equations of Large-Scale Atmosphere and Ocean Dynamics

Journal of Nonlinear Science ◽

10.1007/s00332-021-09707-3 ◽

2021 ◽

Vol 31 (3) ◽

Author(s):

Peter Korn

Keyword(s):

Data Assimilation ◽

Large Scale ◽

Initial Conditions ◽

Strong Solutions ◽

Variational Data Assimilation ◽

Ocean Dynamics ◽

Primitive Equations ◽

Cost Functional ◽

Strong Solvability ◽

Gradient Based

AbstractFor the primitive equations of large-scale atmosphere and ocean dynamics, we study the problem of determining by means of a variational data assimilation algorithm initial conditions that generate strong solutions which minimize the distance to a given set of time-distributed observations. We suggest a modification of the adjoint algorithm whose novel elements is to use norms in the variational cost functional that reflects the $$H^1$$ H 1 -regularity of strong solutions of the primitive equations. For such a cost functional, we prove the existence of minima and a first-order adjoint condition for strong solutions that provides the basis for computing these minima. We prove the local convergence of a gradient-based descent algorithm to optimal initial conditions using the second-order adjoint primitive equations. The algorithmic modifications due to the $$H^1$$ H 1 -norms are straightforwardly to implement into a variational algorithm that employs the standard $$L^2$$ L 2 -metrics.

Download Full-text

Hybridizing Bayesian and variational data assimilation for robust high-resolution hydrologic forecasting

10.5194/hess-2017-431 ◽

2017 ◽

Author(s):

Felipe Hernández ◽

Xu Liang

Keyword(s):

High Resolution ◽

Data Assimilation ◽

Predictive Accuracy ◽

Initial Conditions ◽

Time Estimation ◽

Variational Data Assimilation ◽

Streamflow Forecasting ◽

Infiltration Capacity ◽

Computational Overhead ◽

Capacity Model

Abstract. The success of real-time estimation and forecasting applications based on geophysical models has been possible thanks to the two main frameworks for the determination of the models’ initial conditions: Bayesian data assimilation and variational data assimilation. However, while there have been efforts to unify these two paradigms, existing attempts struggle to fully leverage the advantages of both in order to face the challenges posed by modern high-resolution models – mainly related to model indeterminacy and steep computational requirements. In this article we introduce a hybrid algorithm called OPTIMISTS (Optimized PareTo Inverse Modeling through Integrated STochastic Search) which is targeted at non-linear high-resolution problems and that brings together ideas from particle filters, 4-dimensional variational methods, evolutionary Pareto optimization, and kernel density estimation in a unique way. Streamflow forecasting experiments were conducted to test which specific parameterizations of OPTIMISTS led to higher predictive accuracy. The experiments analysed two watersheds, one with a low resolution using the VIC (Variable Infiltration Capacity) model and one with a high-resolution using the DHSVM (Distributed Hydrology Soil Vegetation Model). By selecting kernel-based non-parametric sampling, non-sequential evaluation of candidate particles, and through the multi-objective minimization of departures from the streamflow observations and from the background states, OPTIMISTS was shown to outperform a particle filter and a 4D variational method. Moreover, the experiments demonstrated that OPTIMISTS scales well in high-resolution cases without imposing a significant computational overhead and that it was successful in mitigating the harmful effects of overfitting. With these combined advantages, the algorithm shows the potential to increase the accuracy and efficiency of operational prediction systems for the improved management of natural resources.

Download Full-text

Implementation of a 3D-Var system for atmospheric profiling data assimilation into the RAMS model: initial results

Atmospheric Measurement Techniques ◽

10.5194/amt-6-3563-2013 ◽

2013 ◽

Vol 6 (12) ◽

pp. 3563-3576 ◽

Cited By ~ 8

Author(s):

S. Federico

Keyword(s):

Relative Humidity ◽

Data Assimilation ◽

Initial Conditions ◽

Weather Prediction ◽

Wind Profiler ◽

Short Term ◽

Background Error ◽

Data Assimilation System ◽

Rams Model ◽

Assimilation System

Abstract. This paper presents the current status of development of a three-dimensional variational data assimilation system (3D-Var). The system can be used with different numerical weather prediction models, but it is mainly designed to be coupled with the Regional Atmospheric Modelling System (RAMS). Analyses are given for the following parameters: zonal and meridional wind components, temperature, relative humidity, and geopotential height. Important features of the data assimilation system are the use of incremental formulation of the cost function, and the representation of the background error by recursive filters and the eigenmodes of the vertical component of the background error covariance matrix. This matrix is estimated by the National Meteorological Center (NMC) method. The data assimilation and forecasting system is applied to the real context of atmospheric profiling data assimilation, and in particular to the short-term wind prediction. The analyses are produced at 20 km horizontal resolution over central Europe and extend over the whole troposphere. Assimilated data are vertical soundings of wind, temperature, and relative humidity from radiosondes, and wind measurements of the European wind profiler network. Results show the validity of the analyses because they are closer to the observations (lower root mean square error (RMSE)) compared to the background (higher RMSE), and the differences of the RMSEs are in agreement with the data assimilation settings. To quantify the impact of improved initial conditions on the short-term forecast, the analyses are used as initial conditions of three-hours forecasts of the RAMS model. In particular two sets of forecasts are produced: (a) the first uses the ECMWF analysis/forecast cycle as initial and boundary conditions; (b) the second uses the analyses produced by the 3D-Var as initial conditions, then it is driven by the ECMWF forecast. The improvement is quantified by considering the horizontal components of the wind, which are measured at asynoptic times by the European wind profiler network. The results show that the RMSE is effectively reduced at the short range. The results are in agreement with the set-up of the numerical experiment.

Download Full-text

Hybridizing Bayesian and variational data assimilation for high-resolution hydrologic forecasting

Hydrology and Earth System Sciences ◽

10.5194/hess-22-5759-2018 ◽

2018 ◽

Vol 22 (11) ◽

pp. 5759-5779 ◽

Cited By ~ 5

Author(s):

Felipe Hernández ◽

Xu Liang

Keyword(s):

High Resolution ◽

Data Assimilation ◽

Predictive Accuracy ◽

Initial Conditions ◽

Time Estimation ◽

Variational Data Assimilation ◽

Streamflow Forecasting ◽

Infiltration Capacity ◽

Probabilistic Forecasts ◽

Non Linear

Abstract. The success of real-time estimation and forecasting applications based on geophysical models has been possible thanks to the two main existing frameworks for the determination of the models' initial conditions: Bayesian data assimilation and variational data assimilation. However, while there have been efforts to unify these two paradigms, existing attempts struggle to fully leverage the advantages of both in order to face the challenges posed by modern high-resolution models – mainly related to model indeterminacy and steep computational requirements. In this article we introduce a hybrid algorithm called OPTIMISTS (Optimized PareTo Inverse Modeling through Integrated STochastic Search) which is targeted at non-linear high-resolution problems and that brings together ideas from particle filters (PFs), four-dimensional variational methods (4D-Var), evolutionary Pareto optimization, and kernel density estimation in a unique way. Streamflow forecasting experiments were conducted to test which specific configurations of OPTIMISTS led to higher predictive accuracy. The experiments were conducted on two watersheds: the Blue River (low resolution) using the VIC (Variable Infiltration Capacity) model and the Indiantown Run (high resolution) using the DHSVM (Distributed Hydrology Soil Vegetation Model). By selecting kernel-based non-parametric sampling, non-sequential evaluation of candidate particles, and through the multi-objective minimization of departures from the streamflow observations and from the background states, OPTIMISTS was shown to efficiently produce probabilistic forecasts with comparable accuracy to that obtained from using a particle filter. Moreover, the experiments demonstrated that OPTIMISTS scales well in high-resolution cases without imposing a significant computational overhead. With the combined advantages of allowing for fast, non-Gaussian, non-linear, high-resolution prediction, the algorithm shows the potential to increase the efficiency of operational prediction systems.

Download Full-text