scholarly journals AIEADA 1.0: Efficient high-dimensional variational data assimilation with machine-learned reduced-order models

2022 ◽  
Author(s):  
Romit Maulik ◽  
Vishwas Rao ◽  
Jiali Wang ◽  
Gianmarco Mengaldo ◽  
Emil Constantinescu ◽  
...  
Keyword(s):  
Data Assimilation ◽  
Objective Function ◽  
High Performance ◽  
Automatic Differentiation ◽  
Weather Forecasting ◽  
Initial Conditions ◽  
Numerical Models ◽  
Weather Prediction ◽  
Forward Model ◽  
Decision Variables

Abstract. Data assimilation (DA) in the geophysical sciences remains the cornerstone of robust forecasts from numerical models. Indeed, DA plays a crucial role in the quality of numerical weather prediction, and is a crucial building block that has allowed dramatic improvements in weather forecasting over the past few decades. DA is commonly framed in a variational setting, where one solves an optimization problem within a Bayesian formulation using raw model forecasts as a prior, and observations as likelihood. This leads to a DA objective function that needs to be minimized, where the decision variables are the initial conditions specified to the model. In traditional DA, the forward model is numerically and computationally expensive. Here we replace the forward model with a low-dimensional, data-driven, and differentiable emulator. Consequently, gradients of our DA objective function with respect to the decision variables are obtained rapidly via automatic differentiation. We demonstrate our approach by performing an emulator-assisted DA forecast of geopotential height. Our results indicate that emulator-assisted DA is faster than traditional equation-based DA forecasts by four orders of magnitude, allowing computations to be performed on a workstation rather than a dedicated high-performance computer. In addition, we describe accuracy benefits of emulator-assisted DA when compared to simply using the emulator for forecasting (i.e., without DA). Our overall formulation is denoted AIAEDA (Artificial Intelligence Emulator Assisted Data Assimilation).

scholarly journals Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving spatial transformers in a case study with ERA5

2021 ◽  
Author(s):  
Ashesh Chattopadhyay ◽  
Mustafa Mustafa ◽  
Pedram Hassanzadeh ◽  
Eviatar Bach ◽  
Karthik Kashinath
Keyword(s):  
Data Assimilation ◽  
Weather Forecasting ◽  
Initial Conditions ◽  
Weather Prediction ◽  
Forecast Accuracy ◽  
Forward Model ◽  
Data Driven ◽  
Average Error ◽  
Time Step ◽  
Sigma Point

Abstract. There is growing interest in data-driven weather prediction (DDWP), for example using convolutional neural networks such as U-NETs that are trained on data from models or reanalysis. Here, we propose 3 components to integrate with commonly used DDWP models in order to improve their physical consistency and forecast accuracy. These components are 1) a deep spatial transformer added to the latent space of the U-NETs to preserve a property called equivariance, which is related to correctly capturing rotations and scalings of features in spatio-temporal data, 2) a data-assimilation (DA) algorithm to ingest noisy observations and improve the initial conditions for next forecasts, and 3) a multi-time-step algorithm, which combines forecasts from DDWP models with different time steps through DA, improving the accuracy of forecasts at short intervals. To show the benefit/feasibility of each component, we use geopotential height at 500 hPa (Z500) from ERA5 reanalysis and examine the short-term forecast accuracy of specific setups of the DDWP framework. Results show that the equivariance-preserving networks (U-STNs) clearly outperform the U-NETs, for example improving the forecast skill by 45 %. Using a sigma-point ensemble Kalman (SPEnKF) algorithm for DA and U-STN as the forward model, we show that stable, accurate DA cycles are achieved even with high observation noise. The DDWP+DA framework substantially benefits from large (O(1000)) ensembles that are inexpensively generated with the data-driven forward model in each DA cycle. The multi-time-step DDWP+DA framework also shows promises, e.g., it reduces the average error by factors of 2–3. These results show the benefits/feasibilities of these 3 components, which are flexible and can be used in a variety of DDWP setups. Furthermore, while here we focus on weather forecasting, the 3 components can be readily adopted for other parts of the Earth system, such as ocean and land, for which there is a rapid growth of data and need for forecast/assimilation.

Big Data Assimilation: Real-time Workflow for 30-second-update Forecasting and Perspectives toward DA-AI Integration

2020 ◽  
Author(s):  
Takemasa Miyoshi ◽  
Takmi Honda ◽  
Shigenori Otsuka ◽  
Arata Amemiya ◽  
Yasumitsu Maejima ◽  
...  
Keyword(s):  
Big Data ◽  
Deep Learning ◽  
Data Assimilation ◽  
Real Time ◽  
High Performance ◽  
Short Term Memory ◽  
Weather Forecasting ◽  
Learning Algorithm ◽  
Weather Prediction ◽  
Scale Model

<p>The Japan’s Big Data Assimilation (BDA) project started in October 2013 and ended its 5.5-year period in March 2019. The direct follow-on project was accepted and started in April 2019 under the Japan Science and Technology Agency (JST) AIP (Advanced Intelligence Project) Acceleration Research, with emphases on the connection with AI technologies, in particular, an integration of DA and AI with high-performance computation (HPC). The BDA project aimed to fully take advantage of “big data” from advanced sensors such as the phased array weather radar (PAWR) and Himawari-8 geostationary satellite, which provide two orders of magnitude more data than the previous sensors. We have achieved successful case studies with newly-developed 30-second-update, 100-m-mesh numerical weather prediction (NWP) system based on the RIKEN’s SCALE model and local ensemble transform Kalman filter (LETKF) to assimilate PAWR in Osaka and Kobe. We have been actively developing the workflow for real-time weather forecasting in Tokyo in summer 2020. In addition, we developed two precipitation nowcasting systems with the every-30-second PAWR data: one with an optical-flow-based system, the other with a deep-learning-based system. We chose the convolutional Long Short Term Memory (Conv-LSTM) as a deep learning algorithm, and found it effective for precipitation nowcasting. The use of Conv-LSTM would lead to an integration of DA and AI with HPC. This presentation will include an overview of the BDA project toward a DA-AI-HPC integration under the new AIP Acceleration Research scheme, and recent progress of the project.</p>

scholarly journals Fengyun Meteorological Satellite Products for Earth System Science Applications

2021 ◽  
Author(s):  
Di Xian ◽  
Peng Zhang ◽  
Ling Gao ◽  
Ruijing Sun ◽  
Haizhen Zhang ◽  
...  
Keyword(s):  
Data Sharing ◽  
Satellite Data ◽  
Prediction Models ◽  
Weather Forecasting ◽  
Numerical Models ◽  
Weather Prediction ◽  
Vegetation Indices ◽  
Open Data ◽  
Earth System ◽  
Inversion Algorithm

AbstractFollowing the progress of satellite data assimilation in the 1990s, the combination of meteorological satellites and numerical models has changed the way scientists understand the earth. With the evolution of numerical weather prediction models and earth system models, meteorological satellites will play a more important role in earth sciences in the future. As part of the space-based infrastructure, the Fengyun (FY) meteorological satellites have contributed to earth science sustainability studies through an open data policy and stable data quality since the first launch of the FY-1A satellite in 1988. The capability of earth system monitoring was greatly enhanced after the second-generation polar orbiting FY-3 satellites and geostationary orbiting FY-4 satellites were developed. Meanwhile, the quality of the products generated from the FY-3 and FY-4 satellites is comparable to the well-known MODIS products. FY satellite data has been utilized broadly in weather forecasting, climate and climate change investigations, environmental disaster monitoring, etc. This article reviews the instruments mounted on the FY satellites. Sensor-dependent level 1 products (radiance data) and inversion algorithm-dependent level 2 products (geophysical parameters) are introduced. As an example, some typical geophysical parameters, such as wildfires, lightning, vegetation indices, aerosol products, soil moisture, and precipitation estimation have been demonstrated and validated by in-situ observations and other well-known satellite products. To help users access the FY products, a set of data sharing systems has been developed and operated. The newly developed data sharing system based on cloud technology has been illustrated to improve the efficiency of data delivery.

scholarly journals Implementation of a 3-D-Var system for atmospheric profiling data assimilation into the RAMS model: initial results

2013 ◽  
Vol 6 (2) ◽  
pp. 3581-3610
Author(s):  
S. Federico
Keyword(s):  
Relative Humidity ◽  
Data Assimilation ◽  
Initial Conditions ◽  
Weather Prediction ◽  
Wind Profiler ◽  
Short Term ◽  
Recursive Filters ◽  
Data Assimilation System ◽  
Rams Model ◽  
Assimilation System

Abstract. This paper presents the current status of development of a three-dimensional variational data assimilation system. The system can be used with different numerical weather prediction models, but it is mainly designed to be coupled with the Regional Atmospheric Modelling System (RAMS). Analyses are given for the following parameters: zonal and meridional wind components, temperature, relative humidity, and geopotential height. Important features of the data assimilation system are the use of incremental formulation of the cost-function, and the use of an analysis space represented by recursive filters and eigenmodes of the vertical background error matrix. This matrix and the length-scale of the recursive filters are estimated by the National Meteorological Center (NMC) method. The data assimilation and forecasting system is applied to the real context of atmospheric profiling data assimilation, and in particular to the short-term wind prediction. The analyses are produced at 20 km horizontal resolution over central Europe and extend over the whole troposphere. Assimilated data are vertical soundings of wind, temperature, and relative humidity from radiosondes, and wind measurements of the European wind profiler network. Results show the validity of the analysis solutions because they are closer to the observations (lower RMSE) compared to the background (higher RMSE), and the differences of the RMSEs are consistent with the data assimilation settings. To quantify the impact of improved initial conditions on the short-term forecast, the analyses are used as initial conditions of a three-hours forecast of the RAMS model. In particular two sets of forecasts are produced: (a) the first uses the ECMWF analysis/forecast cycle as initial and boundary conditions; (b) the second uses the analyses produced by the 3-D-Var scheme as initial conditions, then is driven by the ECMWF forecast. The improvement is quantified by considering the horizontal components of the wind, which are measured at a-synoptic times by the European wind profiler network. The results show that the RMSE is effectively reduced at the short range (1–2 h). The results are in agreement with the set-up of the numerical experiment.

Ensemble-based data assimilation of volcanic aerosols using FALL3D+PDAF

2021 ◽  
Author(s):  
Leonardo Mingari ◽  
Andrew Prata ◽  
Federica Pardini
Keyword(s):  
Data Assimilation ◽  
Information Transfer ◽  
High Performance ◽  
Volcanic Ash ◽  
Numerical Models ◽  
Volcanic Eruptions ◽  
Column Height ◽  
Eruption Column ◽  
Operational Environment ◽  
Assimilation System

<p>Modelling atmospheric dispersion and deposition of volcanic ash is becoming increasingly valuable for understanding the potential impacts of explosive volcanic eruptions on infrastructures, air quality and aviation. The generation of high-resolution forecasts depends on the accuracy and reliability of the input data for models. Uncertainties in key parameters such as eruption column height injection, physical properties of particles or meteorological fields, represent a major source of error in forecasting airborne volcanic ash. The availability of nearly real time geostationary satellite observations with high spatial and temporal resolutions provides the opportunity to improve forecasts in an operational context. Data assimilation (DA) is one of the most effective ways to reduce the error associated with the forecasts through the incorporation of available observations into numerical models. Here we present a new implementation of an ensemble-based data assimilation system based on the coupling between the FALL3D dispersal model and the Parallel Data Assimilation Framework (PDAF). The implementation is based on the last version release of FALL3D (versions 8.x) tailored to the extreme-scale computing requirements, which has been redesigned and rewritten from scratch in the framework of the EU Center of Excellence for Exascale in Solid Earth (ChEESE). The proposed methodology can be efficiently implemented in an operational environment by exploiting high-performance computing (HPC) resources. The FALL3D+PDAF system can be run in parallel and supports online-coupled DA, which allows an efficient information transfer through parallel communication. Satellite-retrieved data from recent volcanic eruptions were considered as input observations for the assimilation system.</p>

scholarly journals Model error in weather forecasting

2001 ◽  
Vol 8 (6) ◽  
pp. 357-371 ◽  
Author(s):  
D. Orrell ◽  
L. Smith ◽  
J. Barkmeijer ◽  
T. N. Palmer
Keyword(s):  
Prediction Models ◽  
Weather Forecasting ◽  
Initial Conditions ◽  
Forecast Error ◽  
Weather Prediction ◽  
Model Error ◽  
Local Model ◽  
State Dependent ◽  
Rapid Divergence ◽  
Dependent Model

Abstract. Operational forecasting is hampered both by the rapid divergence of nearby initial conditions and by error in the underlying model. Interest in chaos has fuelled much work on the first of these two issues; this paper focuses on the second. A new approach to quantifying state-dependent model error, the local model drift, is derived and deployed both in examples and in operational numerical weather prediction models. A simple law is derived to relate model error to likely shadowing performance (how long the model can stay close to the observations). Imperfect model experiments are used to contrast the performance of truncated models relative to a high resolution run, and the operational model relative to the analysis. In both cases the component of forecast error due to state-dependent model error tends to grow as the square-root of forecast time, and provides a major source of error out to three days. These initial results suggest that model error plays a major role and calls for further research in quantifying both the local model drift and expected shadowing times.

scholarly journals Data assimilation method in flood forecasting for Red river system using high performent computer

2015 ◽  
Vol 37 (1) ◽  
pp. 29-42
Author(s):  
Nguyen Thanh Don ◽  
Nguyen Van Que ◽  
Tran Quang Hung ◽  
Nguyen Hong Phong
Keyword(s):  
Data Assimilation ◽  
Flow Rate ◽  
New Technologies ◽  
Weather Forecasting ◽  
Initial Conditions ◽  
River System ◽  
Red River ◽  
Time Dependent ◽  
Complex Case ◽  
Flood Model

Around the world, the data assimilation framework has been reported to be of great interest for weather forecasting, oceanography modeling and for shallow water flows particularly for flood model. For flood model this method is a power full tool to identify time-independent parameters (e.g. Manning coefficients and initial conditions) and time-dependent parameters (e.g. inflow). This paper demonstrates the efficiency of the method to identify time-dependent parameter: inflow discharge with a real complex case Red River. Firstly, we briefly discuss about current methods for determining flow rate which encompasses the new technologies, then present the ability to recover flow rate of this method. For the case of very long time series, a temporal strategy with time overlapping is suggested to decrease the amount of memory required. In addition, some different aspects of data assimilation are covered from this case.

scholarly journals Assessment of ocean analysis and forecast from an atmosphere–ocean coupled data assimilation operational system

Ocean Science ◽  
2019 ◽  
Vol 15 (5) ◽  
pp. 1307-1326 ◽  
Author(s):  
Catherine Guiavarc'h ◽  
Jonah Roberts-Jones ◽  
Chris Harris ◽  
Daniel J. Lea ◽  
Andrew Ryan ◽  
...  
Keyword(s):  
Data Assimilation ◽  
Weather Forecasting ◽  
Current System ◽  
Weather Prediction ◽  
Surface Ocean ◽  
Weakly Coupled ◽  
Ocean Prediction ◽  
Coupled Data Assimilation ◽  
Ocean Forecasting ◽  
Prediction Systems

Abstract. The development of coupled atmosphere–ocean prediction systems with utility on short-range numerical weather prediction (NWP) and ocean forecasting timescales has accelerated over the last decade. This builds on a body of evidence showing the benefit, particularly for weather forecasting, of more correctly representing the feedbacks between the surface ocean and atmosphere. It prepares the way for more unified prediction systems with the capability of providing consistent surface meteorology, wave and surface ocean products to users for whom this is important. Here we describe a coupled ocean–atmosphere system, with weakly coupled data assimilation, which was operationalised at the Met Office as part of the Copernicus Marine Environment Service (CMEMS). We compare the ocean performance to that of an equivalent ocean-only system run at the Met Office and other CMEMS products. Sea surface temperatures in particular are shown to verify better than in the ocean-only systems, although other aspects including temperature profiles and surface currents are slightly degraded. We then discuss the plans to improve the current system in future as part of the development of a “coupled NWP” system at the Met Office.

scholarly journals Advancing from Convection-Allowing NWP to Warn-on-Forecast: Evidence of Progress

2018 ◽  
Vol 33 (2) ◽  
pp. 599-607 ◽  
Author(s):  
John R. Lawson ◽  
John S. Kain ◽  
Nusrat Yussouf ◽  
David C. Dowell ◽  
Dustan M. Wheatley ◽  
...  
Keyword(s):  
Data Assimilation ◽  
Real Time ◽  
Initial Conditions ◽  
Spatial Scales ◽  
Weather Prediction ◽  
Radar Data ◽  
Model Systems ◽  
Deterministic Systems ◽  
Forecast Lead Time ◽  
The Impact

Abstract The Warn-on-Forecast (WoF) program, driven by advanced data assimilation and ensemble design of numerical weather prediction (NWP) systems, seeks to advance 0–3-h NWP to aid National Weather Service warnings for thunderstorm-induced hazards. An early prototype of the WoF prediction system is the National Severe Storms Laboratory (NSSL) Experimental WoF System for ensembles (NEWSe), which comprises 36 ensemble members with varied initial conditions and parameterization suites. In the present study, real-time 3-h quantitative precipitation forecasts (QPFs) during spring 2016 from NEWSe members are compared against those from two real-time deterministic systems: the operational High Resolution Rapid Refresh (HRRR, version 1) and an upgraded, experimental configuration of the HRRR. All three model systems were run at 3-km horizontal grid spacing and differ in initialization, particularly in the radar data assimilation methods. It is the impact of this difference that is evaluated herein using both traditional and scale-aware verification schemes. NEWSe, evaluated deterministically for each member, shows marked improvement over the two HRRR versions for 0–3-h QPFs, especially at higher thresholds and smaller spatial scales. This improvement diminishes with forecast lead time. The experimental HRRR model, which became operational as HRRR version 2 in August 2016, also provides added skill over HRRR version 1.

scholarly journals Development of a hybrid variational-ensemble data assimilation technique for observed lightning tested in a mesoscale model

2014 ◽  
Vol 21 (5) ◽  
pp. 1027-1041 ◽  
Author(s):  
K. Apodaca ◽  
M. Zupanski ◽  
M. DeMaria ◽  
J. A. Knaff ◽  
L. D. Grasso
Keyword(s):  
Data Assimilation ◽  
Prediction Models ◽  
Mesoscale Model ◽  
Initial Conditions ◽  
Positive Impact ◽  
Weather Prediction ◽  
New Information ◽  
Forecasting System ◽  
Maximum Likelihood Ensemble Filter ◽  
The Impact

Abstract. Lightning measurements from the Geostationary Lightning Mapper (GLM) that will be aboard the Geostationary Operational Environmental Satellite – R Series will bring new information that can have the potential for improving the initialization of numerical weather prediction models by assisting in the detection of clouds and convection through data assimilation. In this study we focus on investigating the utility of lightning observations in mesoscale and regional applications suitable for current operational environments, in which convection cannot be explicitly resolved. Therefore, we examine the impact of lightning observations on storm environment. Preliminary steps in developing a lightning data assimilation capability suitable for mesoscale modeling are presented in this paper. World Wide Lightning Location Network (WWLLN) data was utilized as a proxy for GLM measurements and was assimilated with the Maximum Likelihood Ensemble Filter, interfaced with the Nonhydrostatic Mesoscale Model core of the Weather Research and Forecasting system (WRF-NMM). In order to test this methodology, regional data assimilation experiments were conducted. Results indicate that lightning data assimilation had a positive impact on the following: information content, influencing several dynamical variables in the model (e.g., moisture, temperature, and winds), and improving initial conditions during several data assimilation cycles. However, the 6 h forecast after the assimilation did not show a clear improvement in terms of root mean square (RMS) errors.

Sign in / Sign up

Export Citation Format

Share Document