The performance of LSTM models from basin to continental scales

2020 ◽  
Author(s):  
Frederik Kratzert ◽  
Daniel Klotz ◽  
Günter Klambauer ◽  
Grey Nearing ◽  
Sepp Hochreiter
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
Short Term Memory ◽  
Regional Scale ◽  
Data Driven ◽  
Hydrological Models ◽  
Short Term ◽  
Term Memory ◽  
Basin Scale ◽  
Long Short Term Memory

<p>Simulation accuracy among traditional hydrological models usually degrades significantly when going from single basin to regional scale. Hydrological models perform best when calibrated for specific basins, and do worse when a regional calibration scheme is used. </p><p>One reason for this is that these models do not (have to) learn hydrological processes from data. Rather, they have a predefined model structure and only a handful of parameters adapt to specific basins. This often yields less-than-optimal parameter values when the loss is not determined by a single basin, but by many through regional calibration.</p><p>The opposite is true for data driven approaches where models tend to get better with more and diverse training data. We examine whether this holds true when modeling rainfall-runoff processes with deep learning, or if, like their process-based counterparts, data-driven hydrological models degrade when going from basin to regional scale.</p><p>Recently, Kratzert et al. (2018) showed that the Long Short-Term Memory network (LSTM), a special type of recurrent neural network, achieves comparable performance to the SAC-SMA at basin scale. In follow up work Kratzert et al. (2019a) trained a single LSTM for hundreds of basins in the continental US, which outperformed a set of hydrological models significantly, even compared to basin-calibrated hydrological models. On average, a single LSTM is even better in out-of-sample predictions (ungauged) compared to the SAC-SMA in-sample (gauged) or US National Water Model (Kratzert et al. 2019b).</p><p>LSTM-based approaches usually involve tuning a large number of hyperparameters, such as the number of neurons, number of layers, and learning rate, that are critical for the predictive performance. Therefore, large-scale hyperparameter search has to be performed to obtain a proficient LSTM network.  </p><p>However, in the abovementioned studies, hyperparameter optimization was not conducted at large scale and e.g. in Kratzert et al. (2018) the same network hyperparameters were used in all basins, instead of tuning hyperparameters for each basin separately. It is yet unclear whether LSTMs follow the same trend of traditional hydrological models to degrade performance from basin to regional scale. </p><p>In the current study, we performed a computational expensive, basin-specific hyperparameter search to explore how site-specific LSTMs differ in performance compared to regionally calibrated LSTMs. We compared our results to the mHM and VIC models, once calibrated per-basin and once using an MPR regionalization scheme. These benchmark models were calibrated individual research groups, to eliminate bias in our study. We analyse whether differences in basin-specific vs regional model performance can be linked to basin attributes or data set characteristics.</p><p>References:</p><p>Kratzert, F., Klotz, D., Brenner, C., Schulz, K., and Herrnegger, M.: Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks, Hydrol. Earth Syst. Sci., 22, 6005–6022, https://doi.org/10.5194/hess-22-6005-2018, 2018. </p><p>Kratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., and Nearing, G.: Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets, Hydrol. Earth Syst. Sci., 23, 5089–5110, https://doi.org/10.5194/hess-23-5089-2019, 2019a. </p><p>Kratzert, F., Klotz, D., Herrnegger, M., Sampson, A. K., Hochreiter, S., & Nearing, G. S.: Toward improved predictions in ungauged basins: Exploiting the power of machine learning. Water Resources Research, 55. https://doi.org/10.1029/2019WR026065, 2019b.</p>

scholarly journals Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network

2020 ◽  
Vol 27 (3) ◽  
pp. 373-389 ◽  
Author(s):  
Ashesh Chattopadhyay ◽  
Pedram Hassanzadeh ◽  
Devika Subramanian
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Short Term Memory ◽  
Data Driven ◽  
Reservoir Computing ◽  
Short Term ◽  
Term Memory ◽  
Machine Learning Methods ◽  
Long Short Term Memory ◽  
Lorenz 96

Abstract. In this paper, the performance of three machine-learning methods for predicting short-term evolution and for reproducing the long-term statistics of a multiscale spatiotemporal Lorenz 96 system is examined. The methods are an echo state network (ESN, which is a type of reservoir computing; hereafter RC–ESN), a deep feed-forward artificial neural network (ANN), and a recurrent neural network (RNN) with long short-term memory (LSTM; hereafter RNN–LSTM). This Lorenz 96 system has three tiers of nonlinearly interacting variables representing slow/large-scale (X), intermediate (Y), and fast/small-scale (Z) processes. For training or testing, only X is available; Y and Z are never known or used. We show that RC–ESN substantially outperforms ANN and RNN–LSTM for short-term predictions, e.g., accurately forecasting the chaotic trajectories for hundreds of numerical solver's time steps equivalent to several Lyapunov timescales. The RNN–LSTM outperforms ANN, and both methods show some prediction skills too. Furthermore, even after losing the trajectory, data predicted by RC–ESN and RNN–LSTM have probability density functions (pdf's) that closely match the true pdf – even at the tails. The pdf of the data predicted using ANN, however, deviates from the true pdf. Implications, caveats, and applications to data-driven and data-assisted surrogate modeling of complex nonlinear dynamical systems, such as weather and climate, are discussed.

scholarly journals Rainfall-Runoff modelling using Long-Short-Term-Memory (LSTM) networks

2018 ◽  
Author(s):  
Frederik Kratzert ◽  
Daniel Klotz ◽  
Claire Brenner ◽  
Karsten Schulz ◽  
Mathew Herrnegger
Keyword(s):  
Short Term Memory ◽  
Regional Scale ◽  
Model Performance ◽  
Data Driven ◽  
Rainfall Runoff ◽  
Short Term ◽  
Data Set ◽  
Term Memory ◽  
Soil Moisture Accounting ◽  
Long Short Term Memory

Abstract. Rainfall-runoff modelling is one of the key challenges in the field of hydrology. Various approaches exist, ranging from physically based over conceptual to fully data driven models. In this paper, we propose a novel data driven approach, using the Long-Short-Term-Memory (LSTM) network, a special type of recurrent neural networks. The advantage of the LSTM is its ability to learn long-term dependencies between the provided input and output of the network, which are essential for modelling storage effects in e.g. catchments with snow influence. We use 241 catchments of the freely available CAMELS data set to test our approach and also compare the results to the well-known Sacramento Soil Moisture Accounting Model (SAC-SMA) coupled with the Snow-17 snow routine. We also show the potential of the LSTM as a regional hydrological model, in which one model predicts the discharge for a variety of catchments. In our last experiment, we show the possibility to transfer process understanding, learned at regional scale, to individual catchments and thereby increasing model performance when compared to a LSTM trained only on the data of single catchments. Using this approach, we were able to achieve better model performance as the SAC-SMA + Snow-17, which underlines the potential of the LSTM for hydrological modelling applications.

scholarly journals Benchmarking a Catchment-Aware Long Short-Term Memory Network (LSTM) for Large-Scale Hydrological Modeling

2019 ◽  
Author(s):  
Frederik Kratzert ◽  
Daniel Klotz ◽  
Guy Shalev ◽  
Günter Klambauer ◽  
Sepp Hochreiter ◽  
...  
Keyword(s):  
Large Scale ◽  
Hydrological Modeling ◽  
Time Series Data ◽  
Short Term Memory ◽  
Series Data ◽  
Hydrological Models ◽  
Short Term ◽  
Data Set ◽  
Term Memory ◽  
Long Short Term Memory

Abstract. Regional rainfall-runoff modeling is an old but still mostly out-standing problem in Hydrological Sciences. The problem currently is that traditional hydrological models degrade significantly in performance when calibrated for multiple basins together instead of for a single basin alone. In this paper, we propose a novel, data-driven approach using Long Short-Term Memory networks (LSTMs), and demonstrate that under a big data paradigm, this is not necessarily the case. By training a single LSTM model on 531 basins from the CAMELS data set using meteorological time series data and static catchment attributes, we were able to significantly improve performance compared to a set of several different hydrological benchmark models. Our proposed approach not only significantly outperforms hydrological models that were calibrated regionally but also achieves better performance than hydrological models that were calibrated for each basin individually. Furthermore, we propose an adaption to the standard LSTM architecture, which we call an Entity-Aware-LSTM (EA-LSTM), that allows for learning, and embedding as a feature layer in a deep learning model, catchment similarities. We show that this learned catchment similarity corresponds well with what we would expect from prior hydrological understanding.

scholarly journals Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks

2018 ◽  
Vol 22 (11) ◽  
pp. 6005-6022 ◽  
Author(s):  
Frederik Kratzert ◽  
Daniel Klotz ◽  
Claire Brenner ◽  
Karsten Schulz ◽  
Mathew Herrnegger
Keyword(s):  
Short Term Memory ◽  
Regional Scale ◽  
Model Performance ◽  
Data Driven ◽  
Rainfall Runoff ◽  
Short Term ◽  
Data Set ◽  
Term Memory ◽  
Soil Moisture Accounting ◽  
Long Short Term Memory

Abstract. Rainfall–runoff modelling is one of the key challenges in the field of hydrology. Various approaches exist, ranging from physically based over conceptual to fully data-driven models. In this paper, we propose a novel data-driven approach, using the Long Short-Term Memory (LSTM) network, a special type of recurrent neural network. The advantage of the LSTM is its ability to learn long-term dependencies between the provided input and output of the network, which are essential for modelling storage effects in e.g. catchments with snow influence. We use 241 catchments of the freely available CAMELS data set to test our approach and also compare the results to the well-known Sacramento Soil Moisture Accounting Model (SAC-SMA) coupled with the Snow-17 snow routine. We also show the potential of the LSTM as a regional hydrological model in which one model predicts the discharge for a variety of catchments. In our last experiment, we show the possibility to transfer process understanding, learned at regional scale, to individual catchments and thereby increasing model performance when compared to a LSTM trained only on the data of single catchments. Using this approach, we were able to achieve better model performance as the SAC-SMA + Snow-17, which underlines the potential of the LSTM for hydrological modelling applications.

How Google's Flood Forecasting Initiative Leverages Deep Learning Hydrologic Models

2020 ◽  
Author(s):  
Asher Metzger ◽  
Zach Moshe ◽  
Guy Shalev ◽  
Ofir Reich ◽  
Zvika Ben-Haim ◽  
...  
Keyword(s):  
Machine Learning ◽  
Hydrologic Modeling ◽  
Short Term Memory ◽  
State Of The Art ◽  
Flood Forecasting ◽  
Data Driven ◽  
Hydrologic Models ◽  
Short Term ◽  
Term Memory ◽  
Long Short Term Memory

<p>One of the major natural disasters is flooding, which causes thousands of fatalities, affects the lives of hundreds of millions, and results in huge economic damages annually. Google’s Flood Forecasting Initiative aims at providing high-resolution flood forecasts and timely warnings around the globe, while focusing first on developing countries where most of the fatalities occur. The high level structure of Google’s flood forecasting framework follows the natural hydrologic-hydraulic coupling, where the hydrologic modeling predicts discharge (or other proxies for discharge) based on rainfall-runoff relationships, and the hydraulic model produces high resolution inundation maps based on those discharge predictions.  Within this general partition, both the hydraulic and hydrologic modules benefit by the use of advanced machine learning techniques allowing for precision and global scale.</p><p>Classical conceptual hydrologic models such as the Sacramento Soil Moisture Accounting Model explicitly model the dynamics of water volumes based on explicit measurements and estimates of the variables (parameters) involved. These models are, however, inherently challenged by the lack of accurate estimates of model parameters and by inaccurate/incomplete description of the complex non-linear rules that govern the underlying dynamics. In contrast, machine learning models, driven by data alone, are potentially capable of describing complex functional dynamics without explicit modelling.  Both the hydrologic and hydraulic models employed by Google rely on data-driven machine learning technologies to achieve superior and scalable performance. In this presentation we focus on describing one of the deep neural hydrologic models proposed by Google. </p><p>As was already shown in a recent work by Kratzert et al. (2018, 2019)[1], a deep neural model can achieve high performance hydrologic forecasts using deep recurrent models such as long short-term memory networks (LSTMs). Moreover, it was shown by Shalev et al. (2019)[2] that a single globally shared LSTM can achieve state-of-the-art performance by utilizing a data-driven learned embedding without the need for geographical-specific attributes.  While the need for explicit rules in pure conceptual modeling is likely to impede the creation of scalable and accurate hydrologic models, an agnostic approach that ignores reliable and available physical properties of water networks is also likely to be sub-optimal. HydroNet is one of Google’s hydrologic models that leverages the known water network structure as well as deep neural technology to create a scalable and reliable hydrologic model. HydroNet builds a globally shared model together with regional adaptation sub-models at each site by utilizing the tree structure of river flow network, and is shown to achieve state-of-the-art scalable hydrologic modeling in several large basins in India and the USA. </p><p> </p><p>[1] Kratzert, Frederik, Daniel Klotz, Guy Shalev, Günter Klambauer, Sepp Hochreiter, and Grey Nearing. "Benchmarking a catchment-aware Long Short-Term Memory Network (LSTM) for large-scale hydrological modeling." arXiv preprint arXiv:1907.08456 (2019).</p><p>[2] Shalev, Guy, Ran El-Yaniv, Daniel Klotz, Frederik Kratzert, Asher Metzger, and Sella Nevo. "Accurate Hydrologic Modeling Using Less Information." arXiv preprint arXiv:1911.09427 (2019).</p>

scholarly journals Prediction of Head Movement in 360-Degree Videos Using Attention Model

Sensors ◽  
2021 ◽  
Vol 21 (11) ◽  
pp. 3678
Author(s):  
Dongwon Lee ◽  
Minji Choi ◽  
Joohyun Lee
Keyword(s):  
Machine Learning ◽  
Short Term Memory ◽  
Moving Average ◽  
The Other ◽  
Learning Models ◽  
Short Term ◽  
Term Memory ◽  
Attention Model ◽  
Long Short Term Memory ◽  
Machine Learning Models

In this paper, we propose a prediction algorithm, the combination of Long Short-Term Memory (LSTM) and attention model, based on machine learning models to predict the vision coordinates when watching 360-degree videos in a Virtual Reality (VR) or Augmented Reality (AR) system. Predicting the vision coordinates while video streaming is important when the network condition is degraded. However, the traditional prediction models such as Moving Average (MA) and Autoregression Moving Average (ARMA) are linear so they cannot consider the nonlinear relationship. Therefore, machine learning models based on deep learning are recently used for nonlinear predictions. We use the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) neural network methods, originated in Recurrent Neural Networks (RNN), and predict the head position in the 360-degree videos. Therefore, we adopt the attention model to LSTM to make more accurate results. We also compare the performance of the proposed model with the other machine learning models such as Multi-Layer Perceptron (MLP) and RNN using the root mean squared error (RMSE) of predicted and real coordinates. We demonstrate that our model can predict the vision coordinates more accurately than the other models in various videos.

Sign in / Sign up

Export Citation Format

Share Document