DATA ASSIMILATION IN A SOLAR DYNAMO MODEL USING ENSEMBLE KALMAN FILTERS: SENSITIVITY AND ROBUSTNESS IN RECONSTRUCTION OF MERIDIONAL FLOW SPEED

Abstract. The high computational resources and the time-consuming IO (Input/Output) are major issues in offline ensemble- based high-dimentional data assimilation systems. Bearing these in mind, this study proposes a sophisticated dynamically running job scheme as well as an innovative parallel IO algorithm to reduce the time-to-solution of an offline framework for high-dimensional ensemble Kalman filters. The dynamically running job scheme runs as many tasks as possible within a single job to reduce the queuing time and minimize the overhead of starting/ending a job. The parallel IO algorithm reads or writes non-overlapping segments of multiple files with an identical structure to reduce the IO times by minimizing the IO competitions and maximizing the overlapping of the MPI (Message Passing Interface) communications with the IO operations. Results based on sensitive experiments shown that the proposed parallel IO algorithm can significantly reduce the IO times and has a very good scalability, too. Based on these two advanced techniques, the offline and online modes of ensemble Kalman filters are built based on PDAF (Parallel Data Assimilation Framework) to comprehensively assess their efficiencies. It can be seen from the comparisons between the offline and online modes that the IO time only accounts for a small fraction of the total time with the proposed parallel IO algorithm. The queuing time might be less than the running time in a low-loaded supercomputer such as in an operational context but the offline mode can be nearly as fast as, if not faster than, the online mode in terms of time-to-solution. However, the queuing time is dominant and several times larger than the running time in a high-loaded supercomputer. Thus, the offline mode is substantially faster than the online mode in terms of time-to-solution, especially for large-scale assimilation problems. From this point of view, it suggests that an offline ensemble Kalman filter with an efficient implementation and a high performance parallel file system should be preferred over its online counterpart for the intermittent data assimilation in many situations.

Download Full-text

Efficient Assimilation of Simulated Observations in a High-Dimensional Geophysical System Using a Localized Particle Filter

Monthly Weather Review ◽

10.1175/mwr-d-15-0322.1 ◽

2016 ◽

Vol 144 (5) ◽

pp. 2007-2020 ◽

Cited By ~ 18

Author(s):

Jonathan Poterjoy ◽

Jeffrey L. Anderson

Keyword(s):

Data Assimilation ◽

Particle Filter ◽

Kalman Filters ◽

General Circulation ◽

Prediction Models ◽

Weather Prediction ◽

Circulation Model ◽

High Dimensional ◽

Ensemble Kalman Filters ◽

Gaussian Filters

This study presents the first application of a localized particle filter (PF) for data assimilation in a high-dimensional geophysical model. Particle filters form Monte Carlo approximations of model probability densities conditioned on observations, while making no assumptions about the underlying error distribution. Unlike standard PFs, the local PF uses a localization function to reduce the influence of distant observations on state variables, which significantly decreases the number of particles required to maintain the filter’s stability. Because the local PF operates effectively using small numbers of particles, it provides a possible alternative to Gaussian filters, such as ensemble Kalman filters, for large geophysical models. In the current study, the local PF is compared with stochastic and deterministic ensemble Kalman filters using a simplified atmospheric general circulation model. The local PF is found to provide stable filtering results over yearlong data assimilation experiments using only 25 particles. The local PF also outperforms the Gaussian filters when observation networks include measurements that have non-Gaussian errors or relate nonlinearly to the model state, like remotely sensed data used frequently in atmospheric analyses. Results from this study encourage further testing of the local PF on more complex geophysical systems, such as weather prediction models.

Download Full-text

Towards Estimating the Solar Meridional Flow and Predicting the 11-yr Cycle Using Advanced Variational Data Assimilation Techniques

Proceedings of the International Astronomical Union ◽

10.1017/s1743921317010754 ◽

2017 ◽

Vol 13 (S335) ◽

pp. 183-186

Author(s):

Ching Pui Hung ◽

Allan Sacha Brun ◽

Alexandre Fournier ◽

Laurène Jouve ◽

Olivier Talagrand ◽

...

Keyword(s):

Data Assimilation ◽

Meridional Circulation ◽

Mean Field ◽

Magnetic Surface ◽

Meridional Flow ◽

Variational Data Assimilation ◽

Dynamo Model ◽

Cyclic Activity ◽

Surface Data ◽

Quasi Newton

AbstractWe present in this work the development of a solar data assimilation method based on an axisymmetric mean field dynamo model and magnetic surface data. Our mid-term goal is to predict the solar quasi cyclic activity. We focus on the ability of our variational data assimilation algorithm to constrain the deep meridional circulation of the Sun based on solar magnetic observations. Within a given assimilation window, the assimilation procedure minimizes the differences between data and the forecast from the model, by finding an optimal meridional circulation in the convection zone, and an optimal initial magnetic field, via a quasi-Newton algorithm. We demonstrate the capability of the technique to estimate the meridional flow by a closed-loop experiment involving 40 years of synthetic, solar-like data. We show that the method is robust in estimating a (stochastic) time-varying flow fluctuating 30% about the average, and that the horizon of predictability of the method is ~ 1 cycle length.

Download Full-text

An offline framework for high-dimensional ensemble Kalman filters to reduce the time to solution

Geoscientific Model Development ◽

10.5194/gmd-13-3607-2020 ◽

2020 ◽

Vol 13 (8) ◽

pp. 3607-3625

Author(s):

Yongjun Zheng ◽

Clément Albergel ◽

Simon Munier ◽

Bertrand Bonan ◽

Jean-Christophe Calvet

Keyword(s):

Data Assimilation ◽

Kalman Filters ◽

High Performance ◽

Large Scale ◽

Message Passing Interface ◽

High Dimensional ◽

Ensemble Kalman Filters ◽

Running Time ◽

Online Mode ◽

Parallel Io

Abstract. The high computational resources and the time-consuming IO (input/output) are major issues in offline ensemble-based high-dimensional data assimilation systems. Bearing these in mind, this study proposes a sophisticated dynamically running job scheme as well as an innovative parallel IO algorithm to reduce the time to solution of an offline framework for high-dimensional ensemble Kalman filters. The dynamically running job scheme runs as many tasks as possible within a single job to reduce the queuing time and minimize the overhead of starting and/or ending a job. The parallel IO algorithm reads or writes non-overlapping segments of multiple files with an identical structure to reduce the IO times by minimizing the IO competitions and maximizing the overlapping of the MPI (Message Passing Interface) communications with the IO operations. Results based on sensitive experiments show that the proposed parallel IO algorithm can significantly reduce the IO times and have a very good scalability, too. Based on these two advanced techniques, the offline and online modes of ensemble Kalman filters are built based on PDAF (Parallel Data Assimilation Framework) to comprehensively assess their efficiencies. It can be seen from the comparisons between the offline and online modes that the IO time only accounts for a small fraction of the total time with the proposed parallel IO algorithm. The queuing time might be less than the running time in a low-loaded supercomputer such as in an operational context, but the offline mode can be nearly as fast as, if not faster than, the online mode in terms of time to solution. However, the queuing time is dominant and several times larger than the running time in a high-loaded supercomputer. Thus, the offline mode is substantially faster than the online mode in terms of time to solution, especially for large-scale assimilation problems. From this point of view, results suggest that an offline ensemble Kalman filter with an efficient implementation and a high-performance parallel file system should be preferred over its online counterpart for intermittent data assimilation in many situations.

Download Full-text

Comparison of extended and ensemble Kalman filters for data assimilation in coastal area modelling

International Journal for Numerical Methods in Fluids ◽

10.1002/(sici)1097-0363(19991130)31:6<961::aid-fld907>3.0.co;2-0 ◽

1999 ◽

Vol 31 (6) ◽

pp. 961-981 ◽

Cited By ~ 44

Author(s):

H. Madsen ◽

R. Cañizares

Keyword(s):

Data Assimilation ◽

Coastal Area ◽

Kalman Filters ◽

Ensemble Kalman Filters

Download Full-text

Mixed-Resolution Ensemble Data Assimilation

Monthly Weather Review ◽

10.1175/mwr-d-12-00234.1 ◽

2013 ◽

Vol 141 (9) ◽

pp. 3007-3021 ◽

Cited By ~ 13

Author(s):

Sabrina Rainwater ◽

Brian Hunt

Keyword(s):

High Resolution ◽

Data Assimilation ◽

Kalman Filters ◽

Computation Time ◽

Computational Time ◽

Ensemble Kalman Filters ◽

Mixed Resolution ◽

Resolution Analysis ◽

Ensemble Transform Kalman Filter ◽

Similar Accuracy

Abstract Ensemble Kalman filters perform data assimilation by forming a background covariance matrix from an ensemble forecast. Most of the literature on ensemble Kalman filters assumes that all ensemble members come from the same model. This article presents and tests a modified local ensemble transform Kalman filter (LETKF) that takes its background covariance from a combination of a high-resolution ensemble and a low-resolution ensemble. The computational time and the accuracy of this mixed-resolution LETKF are explored and compared to the standard LETKF on a high-resolution ensemble, using simulated observation experiments with the Lorenz models II and III (more complex versions of the Lorenz-96 model). In a variety of scenarios, mixed-resolution analysis can obtain higher accuracy with similar computation time (or similar accuracy with a reduced computation time) compared to single-resolution analysis.

Download Full-text