Towards global hybrid hydrological modeling by fusing deep learning and a conceptual model

2020 ◽  
Author(s):  
Basil Kraft ◽  
Martin Jung ◽  
Marco Körner ◽  
Markus Reichstein
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Conceptual Model ◽  
Hydrological Modeling ◽  
Snow Water Equivalent ◽  
Hydrological Cycle ◽  
Hybrid Approach ◽  
Internal Representation ◽  
Hybrid Modeling ◽  
Environmental Systems

<p>Deep (recurrent) neural networks have proven very useful to model multivariate sequential data streams of complex dynamic natural systems and have already been successfully applied to model hydrological processes. Compared to physically based models, however, the internal representation of a neural network is not directly interpretable and model predictions often lack physical consistency. Hybrid modeling is a promising approach that synergizes the advantage of process-based modeling (interpretability, theoretical foundations) and deep learning (data adaptivity, less prior knowledge required): By combining these two approaches, flexible and partially interpretable models can be created that have the potential to advance the understanding and predictability of environmental systems.</p><p>Here, we implement such a hybrid hydrological model on a global scale. The model consists of three main blocks: 1) A Long-Short-Term Memory (LSTM) model, which extracts temporal features from the meteorological forcing time-series. 2) A multi-branch neural network comprising of independent, fully connected layers, taking the LSTM state as input and yielding a set of latent, interpretable variables (e.g. soil moisture recharge). 3) A conceptual model block that implements hydrological balance equations, driven by the above interpretable variables. The model is trained simultaneously on global observation-based products of total water storage, snow water equivalent, evapotranspiration and runoff. To combine the different loss terms, we use self-paced task uncertainty weighing as done in state-of-the-art multi-task learning.</p><p>Preliminary results suggest that the hybrid modeling approach captures global patterns of the hydrological cycle’s variability that are consistent with observations and our process understanding. The approach opens doors to novel data-driven simulations, attribution and diagnostic assessments of water cycle variations globally. The presented approach is—to our knowledge—the first application of the hybrid approach to model environmental systems.</p>

scholarly journals HYBRID MODELING: FUSION OF A DEEP LEARNING APPROACH AND A PHYSICS-BASED MODEL FOR GLOBAL HYDROLOGICAL MODELING

2020 ◽  
Vol XLIII-B2-2020 ◽  
pp. 1537-1544
Author(s):  
B. Kraft ◽  
M. Jung ◽  
M. Körner ◽  
M. Reichstein
Keyword(s):  
Deep Learning ◽  
Hydrological Modeling ◽  
Snow Water Equivalent ◽  
Expert Knowledge ◽  
Water Cycle ◽  
Temporal Variations ◽  
Hybrid Modeling ◽  
Balance Model ◽  
Observation Data ◽  
New Paradigm

Abstract. Process-based models of complex environmental systems incorporate expert knowledge which is often incomplete and uncertain. With the growing amount of Earth observation data and advances in machine learning, a new paradigm is promising to synergize the advantages of deep learning in terms of data adaptiveness and performance for poorly understood processes with the advantages of process-based modeling in terms of interpretability and theoretical foundations: hybrid modeling. Here, we present such an end-to-end hybrid modeling approach that learns and predicts spatial-temporal variations of observed and unobserved (latent) hydrological variables globally. The model combines a dynamic neural network and a conceptual water balance model, constrained by the water cycle observational products of evapotranspiration, runoff, snow-water equivalent, and terrestrial water storage variations. We show that the model reproduces observed water cycle variations very well and that the emergent relations of runoff-generating processes are qualitatively consistent with our understanding. The presented model is – to our knowledge – the first of its kind and may contribute new insights about the dynamics of the global hydrological system.

scholarly journals A Statistics and Deep Learning Hybrid Method for Multivariate Time Series Forecasting and Mortality Modeling

Forecasting ◽  
2021 ◽  
Vol 4 (1) ◽  
pp. 1-25
Author(s):  
Thabang Mathonsi ◽  
Terence L. van Zyl
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Recurrent Neural Network ◽  
Short Term Memory ◽  
Hybrid Methods ◽  
Prediction Interval ◽  
Hybrid Approach ◽  
Computational Cost ◽  
Exponential Smoothing ◽  
Learning Methods

Hybrid methods have been shown to outperform pure statistical and pure deep learning methods at forecasting tasks and quantifying the associated uncertainty with those forecasts (prediction intervals). One example is Exponential Smoothing Recurrent Neural Network (ES-RNN), a hybrid between a statistical forecasting model and a recurrent neural network variant. ES-RNN achieves a 9.4% improvement in absolute error in the Makridakis-4 Forecasting Competition. This improvement and similar outperformance from other hybrid models have primarily been demonstrated only on univariate datasets. Difficulties with applying hybrid forecast methods to multivariate data include (i) the high computational cost involved in hyperparameter tuning for models that are not parsimonious, (ii) challenges associated with auto-correlation inherent in the data, as well as (iii) complex dependency (cross-correlation) between the covariates that may be hard to capture. This paper presents Multivariate Exponential Smoothing Long Short Term Memory (MES-LSTM), a generalized multivariate extension to ES-RNN, that overcomes these challenges. MES-LSTM utilizes a vectorized implementation. We test MES-LSTM on several aggregated coronavirus disease of 2019 (COVID-19) morbidity datasets and find our hybrid approach shows consistent, significant improvement over pure statistical and deep learning methods at forecast accuracy and prediction interval construction.

scholarly journals PREDICTING LAND PRICES AND MEASURING UNCERTAINTY BY COMBINING SUPERVISED AND UNSUPERVISED LEARNING

2021 ◽  
Vol 25 (2) ◽  
pp. 169-178
Author(s):  
Changro Lee
Keyword(s):  
Neural Network ◽  
Principal Component Analysis ◽  
Deep Learning ◽  
Unsupervised Learning ◽  
Supervised Learning ◽  
Hybrid Approach ◽  
Principal Component ◽  
Component Analysis ◽  
Land Prices ◽  
Property Valuation

Despite the popularity deep learning has been gaining, measuring the uncertainty within the result has not met expectations in many deep learning applications and this includes property valuation. In real-world tasks, however, rather than simply requiring predictions, assurance of the certainty of the predictions is also demanded. In this study, supervised learning is combined with unsupervised learning to bridge this gap. A method based on principal component analysis, a popular tool of unsupervised learning, was developed and used to represent the uncertainty in property valuation. Then, a neural network, a representative algorithm to implement supervised learning, was constructed, and trained to predict land prices. Finally, the uncertainty that was measured using principal component analysis was incorporated into the price predicted by the neural network. This hybrid approach is shown to be likely to improve the credibility of the valuation work. The findings of this study are expected to generate interest in the integration of the two learning approaches, thereby promoting the rapid adoption of deep learning tools in the property valuation industry.

Deep Learning for Three-Dimensional Volumetric Recovery of Cloud Fields

2021 ◽  
Author(s):  
Yael Sde-Chen ◽  
Yoav Y. Schechner ◽  
Vadim Holodovsky ◽  
Eshkol Eytan
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Large Scale ◽  
Optimization Technique ◽  
Hybrid Approach ◽  
Three Dimensional ◽  
Physical Nature ◽  
Key Factor ◽  
Volumetric Reconstruction ◽  
Real World Datasets

<p>Clouds are a key factor in Earth's energy budget and thus significantly affect climate and weather predictions. These effects are dominated by shallow warm clouds (shown by Sherwood et al., 2014, Zelinka et al., 2020) which tend to be small and heterogenous. Therefore, remote sensing of clouds and three-dimensional (3D) volumetric reconstruction of their internal properties are of significant importance.</p><p>Recovery of the volumetric information of the clouds relies on 3D radiative transfer, that models 3D multiple scattering. This model is complex and nonlinear. Thus, inverting the model poses a major challenge and typically requires using a simplification. A common relaxation assumes that clouds are horizontally uniform and infinitely broad, leading to one-dimensional modeling. However, generally this assumption is invalid since clouds are naturally highly heterogeneous. A novel alternative is to perform cloud retrieval by developing tools of 3D scattering tomography. Then, multiple satellite images of the clouds are acquired from different points of view. For example, simultaneous multi-view radiometric images of clouds are proposed by the CloudCT project, funded by the ERC. Unfortunately, 3D scattering tomography require high computational resources. This results, in practice, in slow run times and prevents large scale analysis. Moreover, existing scattering tomography is based on iterative optimization, which is sensitive to initialization.</p><p>In this work we introduce a deep neural network for 3D volumetric reconstruction of clouds. In recent years, supervised learning using deep neural networks has led to remarkable results in various fields ranging from computer vision to medical imaging. However, these deep learning techniques have not been extensively studied in the context of volumetric atmospheric science and specifically cloud research.</p><p>We present a convolutional neural network (CNN) whose architecture is inspired by the physical nature of clouds. Due to the lack of real-world datasets, we train the network in a supervised manner using a physics-based simulator that generates realistic volumetric cloud fields. In addition, we propose a hybrid approach, which combines the proposed neural network with an iterative physics-based optimization technique.</p><p>We demonstrate the recovery performance of our proposed method in cloud fields. In a single cloud-scale, our resulting quality is comparable to state-of-the-art methods, while run time improves by orders of magnitude. In contrast to existing physics-based methods, our network offers scalability, which enables the reconstruction of wider cloud fields. Finally, we show that the hybrid approach leads to improved retrieval in a fast process.</p>

scholarly journals Data Sampling Methods to Deal With the Big Data Multi-Class Imbalance Problem

2020 ◽  
Vol 10 (4) ◽  
pp. 1276 ◽  
Author(s):  
Eréndira Rendón ◽  
Roberto Alejo ◽  
Carlos Castorena ◽  
Frank J. Isidro-Ortega ◽  
Everardo E. Granda-Gutiérrez
Keyword(s):  
Neural Network ◽  
Big Data ◽  
Deep Learning ◽  
Sampling Methods ◽  
Hybrid Approach ◽  
Class Imbalance ◽  
Data Sampling ◽  
Class Imbalance Problem ◽  
Imbalance Problem ◽  
Output Noise

The class imbalance problem has been a hot topic in the machine learning community in recent years. Nowadays, in the time of big data and deep learning, this problem remains in force. Much work has been performed to deal to the class imbalance problem, the random sampling methods (over and under sampling) being the most widely employed approaches. Moreover, sophisticated sampling methods have been developed, including the Synthetic Minority Over-sampling Technique (SMOTE), and also they have been combined with cleaning techniques such as Editing Nearest Neighbor or Tomek’s Links (SMOTE+ENN and SMOTE+TL, respectively). In the big data context, it is noticeable that the class imbalance problem has been addressed by adaptation of traditional techniques, relatively ignoring intelligent approaches. Thus, the capabilities and possibilities of heuristic sampling methods on deep learning neural networks in big data domain are analyzed in this work, and the cleaning strategies are particularly analyzed. This study is developed on big data, multi-class imbalanced datasets obtained from hyper-spectral remote sensing images. The effectiveness of a hybrid approach on these datasets is analyzed, in which the dataset is cleaned by SMOTE followed by the training of an Artificial Neural Network (ANN) with those data, while the neural network output noise is processed with ENN to eliminate output noise; after that, the ANN is trained again with the resultant dataset. Obtained results suggest that best classification outcome is achieved when the cleaning strategies are applied on an ANN output instead of input feature space only. Consequently, the need to consider the classifier’s nature when the classical class imbalance approaches are adapted in deep learning and big data scenarios is clear.

scholarly journals Towards hybrid modeling of the global hydrological cycle

2021 ◽  
Author(s):  
Basil Kraft ◽  
Martin Jung ◽  
Marco Körner ◽  
Sujan Koirala ◽  
Markus Reichstein
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Soil Moisture ◽  
Data Streams ◽  
Hydrological Modeling ◽  
Water Cycle ◽  
Hybrid Modeling ◽  
Hydrological Models ◽  
Global Water Cycle ◽  
Global Water

Abstract. Progress in machine learning in conjunction with the increasing availability of relevant Earth observation data streams may help to overcome uncertainties of global hydrological models due to the complexity of the processes, diversity, and heterogeneity of the land surface and subsurface, as well as scale-dependency of processes and parameters. In this study, we exemplify a hybrid approach to global hydrological modeling that exploits the data-adaptiveness of machine learning for representing uncertain processes within a model structure based on physical principles like mass conservation. Our H2M model simulates the dynamics of snow, soil moisture, and groundwater pools globally at 1º spatial resolution and daily time step where simulated water fluxes depend on an embedded recurrent neural network. We trained the model simultaneously against observational products of terrestrial water storage variations (TWS), runoff, evapotranspiration, and snow water equivalent with a multi-task learning approach. We find that H2M is capable of reproducing the key patterns of global water cycle components with model performances being at least on par with four state-of-the-art global hydrological models. The neural network learned hydrological responses of evapotranspiration and runoff generation to antecedent soil moisture state that are qualitatively consistent with our understanding and theory. Simulated contributions of groundwater, soil moisture, and snowpack variability to TWS variations are plausible and within the large range of traditional GHMs. H2M indicates a somewhat stronger role of soil moisture for TWS variations in transitional and tropical regions compared to GHMs. Overall, we present a proof of concept for global hybrid hydrological modeling in providing a new, complementary, and data-driven perspective on global water cycle variations. With further increasing Earth observations, hybrid modeling has a large potential to advance our capability to monitor and understand the Earth system by facilitating a data-adaptive yet physically consistent, joint interpretation of heterogeneous data streams.

scholarly journals Implementation of Convolutional Neural Network to Realize a Real Time Emotion based Music Player

2019 ◽  
Vol 8 (2S11) ◽  
pp. 164-171
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Real Time ◽  
Data Science ◽  
Internal Representation ◽  
Learning Technology ◽  
Complex Data ◽  
Scale Invariant ◽  
Two Dimensional Image ◽  
Music Player

The ability to represent the world as a nested hierarchy of concepts, by defining each concept in relation to abstract representations has promoted deep learning to be widely used as a processing model for solving data science tasks. The era of digitalization has allowed the deep learning technology to flourish and machines with the ability to analyse huge amount of complex data would now be able to give progressively exact outcomes due to its supremacy in terms of accuracy when trained with massive amount of data. Convolutional Neural Networks(CNN), being a deep neural network with their ability to develop an internal representation of a two-dimensional image, allows the model to learn position and scale invariant structures in the data, which is important when working with images. For realizing emotion aware applications, the system must be highly accurate and in real time. In this paper, we provide the design and implementation details of a real time emotion based music player using CNN with the aim to reduce human effort and invoke the feasibility of Human Computer interaction(HCI).

scholarly journals Hybrid Modeling, HMM/NN Architectures, and Protein Applications

1996 ◽  
Vol 8 (7) ◽  
pp. 1541-1565 ◽  
Author(s):  
Pierre Baldi ◽  
Yves Chauvin
Keyword(s):  
Neural Network ◽  
Dynamic Programming ◽  
Hidden Markov Models ◽  
Markov Models ◽  
Hidden Markov ◽  
Hybrid Approach ◽  
Multiple Alignment ◽  
Hybrid Modeling ◽  
Complex Data ◽  
Model Structure

We describe a hybrid modeling approach where the parameters of a model are calculated and modulated by another model, typically a neural network (NN), to avoid both overfitting and underfitting. We develop the approach for the case of Hidden Markov Models (HMMs), by deriving a class of hybrid HMM/NN architectures. These architectures can be trained with unified algorithms that blend HMM dynamic programming with NN backpropagation. In the case of complex data, mixtures of HMMs or modulated HMMs must be used. NNs can then be applied both to the parameters of each single HMM, and to the switching or modulation of the models, as a function of input or context. Hybrid HMM/NN architectures provide a flexible NN parameterization for the control of model structure and complexity. At the same time, they can capture distributions that, in practice, are inaccessible to single HMMs. The HMM/NN hybrid approach is tested, in its simplest form, by constructing a model of the immunoglobulin protein family. A hybrid model is trained, and a multiple alignment derived, with less than a fourth of the number of parameters used with previous single HMMs.

scholarly journals FORECASTING AIRCRAFT MILES FLOWN TIME SERIES USING A DEEP LEARNING-BASED HYBRID APPROACH

Aviation ◽  
2018 ◽  
Vol 22 (1) ◽  
pp. 6-12 ◽  
Author(s):  
Victor SINEGLAZOV ◽  
Olena CHUMACHENKO ◽  
Vladyslav GORBATIUK
Keyword(s):  
Neural Network ◽  
Time Series ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Hybrid Approach ◽  
Practical Application ◽  
Backpropagation Algorithm ◽  
Anfis Model ◽  
Wide Range ◽  
Domestic Airlines

Neural network-based methods such as deep neural networks show great efficiency for a wide range of applications. In this paper, a deep learning-based hybrid approach to forecast the yearly revenue passenger kilometers time series of Australia’s major domestic airlines is proposed. The essence of the approach is to use a resilient error backpropagation algorithm with dropout for “tuning” the polynomial neural network, obtained as a result of a multi-layered GMDH algorithm. The article compares the performance of the suggested algorithm on the time series with other popular forecasting methods: deep belief network, multi-layered GMDH algorithm, Box-Jenkins method and the ANFIS model. The minimum reached MAE of the proposed algorithm was approximately 25% lower than the minimum MAE of the next best method – GMDH, thus indicating that the practical application of the algorithm can give good results compared with other well-known methods.

Sign in / Sign up

Export Citation Format

Share Document