scholarly journals Inverse Molecule Design with Invertible Neural Networks as Generative Models

2021 ◽  
Vol 14 (07) ◽  
pp. 305-315
Author(s):  
Wei Hu
Keyword(s):  
Neural Networks ◽  
Generative Models ◽  
Molecule Design

scholarly journals Off-the-shelf deep learning is not enough, and requires parsimony, Bayesianity, and causality

2021 ◽  
Vol 7 (1) ◽  
Author(s):  
Rama K. Vasudevan ◽  
Maxim Ziatdinov ◽  
Lukas Vlcek ◽  
Sergei V. Kalinin
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Deep Learning ◽  
Bayesian Methods ◽  
Deep Neural Networks ◽  
Applied Research ◽  
Modern Science ◽  
Generative Models ◽  
Knowledge Development ◽  
Physical Constraints

AbstractDeep neural networks (‘deep learning’) have emerged as a technology of choice to tackle problems in speech recognition, computer vision, finance, etc. However, adoption of deep learning in physical domains brings substantial challenges stemming from the correlative nature of deep learning methods compared to the causal, hypothesis driven nature of modern science. We argue that the broad adoption of Bayesian methods incorporating prior knowledge, development of solutions with incorporated physical constraints and parsimonious structural descriptors and generative models, and ultimately adoption of causal models, offers a path forward for fundamental and applied research.

scholarly journals Uncertainty-Aware Deep Classifiers Using Generative Models

2020 ◽  
Vol 34 (04) ◽  
pp. 5620-5627 ◽  
Author(s):  
Murat Sensoy ◽  
Lance Kaplan ◽  
Federico Cerutti ◽  
Maryam Saleki
Keyword(s):  
Neural Networks ◽  
Epistemic Uncertainty ◽  
Feature Space ◽  
Generative Models ◽  
Detection Methods ◽  
Generative Adversarial Networks ◽  
Data Sets ◽  
Bayesian Approaches ◽  
Data Set ◽  
Auxiliary Data

Deep neural networks are often ignorant about what they do not know and overconfident when they make uninformed predictions. Some recent approaches quantify classification uncertainty directly by training the model to output high uncertainty for the data samples close to class boundaries or from the outside of the training distribution. These approaches use an auxiliary data set during training to represent out-of-distribution samples. However, selection or creation of such an auxiliary data set is non-trivial, especially for high dimensional data such as images. In this work we develop a novel neural network model that is able to express both aleatoric and epistemic uncertainty to distinguish decision boundary and out-of-distribution regions of the feature space. To this end, variational autoencoders and generative adversarial networks are incorporated to automatically generate out-of-distribution exemplars for training. Through extensive analysis, we demonstrate that the proposed approach provides better estimates of uncertainty for in- and out-of-distribution samples, and adversarial examples on well-known data sets against state-of-the-art approaches including recent Bayesian approaches for neural networks and anomaly detection methods.

scholarly journals Biologically Plausible Sequence Learning with Spiking Neural Networks

2020 ◽  
Vol 34 (02) ◽  
pp. 1316-1323
Author(s):  
Zuozhu Liu ◽  
Thiparat Chotibut ◽  
Christopher Hillar ◽  
Shaowei Lin
Keyword(s):  
Neural Networks ◽  
Sequence Learning ◽  
Nervous Activity ◽  
Learning Rule ◽  
Generative Models ◽  
Spike Timing ◽  
Hopfield Network ◽  
Spiking Neural Networks ◽  
Theoretical Ground ◽  
Time Model

Motivated by the celebrated discrete-time model of nervous activity outlined by McCulloch and Pitts in 1943, we propose a novel continuous-time model, the McCulloch-Pitts network (MPN), for sequence learning in spiking neural networks. Our model has a local learning rule, such that the synaptic weight updates depend only on the information directly accessible by the synapse. By exploiting asymmetry in the connections between binary neurons, we show that MPN can be trained to robustly memorize multiple spatiotemporal patterns of binary vectors, generalizing the ability of the symmetric Hopfield network to memorize static spatial patterns. In addition, we demonstrate that the model can efficiently learn sequences of binary pictures as well as generative models for experimental neural spike-train data. Our learning rule is consistent with spike-timing-dependent plasticity (STDP), thus providing a theoretical ground for the systematic design of biologically inspired networks with large and robust long-range sequence storage capacity.

Discriminative and generative models for anatomical shape analysis on point clouds with deep neural networks

2021 ◽  
Vol 67 ◽  
pp. 101852
Author(s):  
Benjamín Gutiérrez-Becker ◽  
Ignacio Sarasua ◽  
Christian Wachinger
Keyword(s):  
Neural Networks ◽  
Shape Analysis ◽  
Deep Neural Networks ◽  
Point Clouds ◽  
Generative Models

scholarly journals Back-propagation learning in deep Spike-By-Spike networks

2019 ◽  
Author(s):  
David Rotermund ◽  
Klaus R. Pawelzik
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Action Potentials ◽  
Back Propagation ◽  
Learning Rule ◽  
Building Blocks ◽  
Classification Performance ◽  
Generative Models ◽  
Learning Method ◽  
Discrete Action

ABSTRACTNeural networks are important building blocks in technical applications. These artificial neural networks (ANNs) rely on noiseless continuous signals in stark contrast to the discrete action potentials stochastically exchanged among the neurons in real brains. A promising approach towards bridging this gap are the Spike-by-Spike (SbS) networks which represent a compromise between non-spiking and spiking versions of generative models that perform inference on their inputs. What is still missing are algorithms for finding weight sets that would optimize the output performances of deep SbS networks with many layers.Here, a learning rule for hierarchically organized SbS networks is derived. The properties of this approach are investigated and its functionality demonstrated by simulations. In particular, a Deep Convolutional SbS network for classifying handwritten digits (MNIST) is presented. When applied together with an optimizer this learning method achieves a classification performance of roughly 99.3% on the MNIST test data. Thereby it approaches the benchmark results of ANNs without extensive parameter optimization. We envision that with this learning rule SBS networks will provide a new basis for research in neuroscience and for technical applications, especially when they become implemented on specialized computational hardware.

scholarly journals Randomized SMILES Strings Improve the Quality of Molecular Generative Models

2019 ◽  
Author(s):  
Josep Arús-Pous ◽  
Simon Johansson ◽  
Oleksii Prykhodko ◽  
Esben Jannik Bjerrum ◽  
Christian Tyrchan ◽  
...  
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Chemical Space ◽  
Cell Types ◽  
Generative Models ◽  
The Other ◽  
Probability Models ◽  
String Representation ◽  
Almost All

Recurrent Neural Networks (RNNs) trained with a set of molecules represented as unique (canonical) SMILES strings, have shown the capacity to create large chemical spaces of valid and meaningful structures. Herein we perform an extensive benchmark on models trained with subsets of GDB-13 of different sizes (1 million , 10,000 and 1,000), with different SMILES variants (canonical, randomized and DeepSMILES), with two different recurrent cell types (LSTM and GRU) and with different hyperparameter combinations. To guide the benchmarks new metrics were developed that define the generated chemical space with respect to its uniformity, closedness and completeness. Results show that models that use LSTM cells trained with 1 million randomized SMILES, a non-unique molecular string representation, are able to generate larger chemical spaces than the other approaches and they represent more accurately the target chemical space. Specifically, a model was trained with randomized SMILES that was able to generate almost all molecules from GDB-13 with a quasi-uniform probability. Models trained with smaller samples show an even bigger improvement when trained with randomized SMILES models. Additionally, models were trained on molecules obtained from ChEMBL and illustrate again that training with randomized SMILES lead to models having a better representation of the drug-like chemical space. Namely, the model trained with randomized SMILES was able to generate at least double the amount of unique molecules with the same distribution of properties comparing to one trained with canonical SMILES.

scholarly journals Envision Foundational of Convolution Neural Network

2021 ◽  
Vol 10 (6) ◽  
pp. 54-60
Author(s):  
M Venkata Krishna Reddy* ◽  
Pradeep S.
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Computer Graphics ◽  
Convolutional Neural Networks ◽  
Generative Models ◽  
Activation Functions ◽  
Machine Learning Classification ◽  
Network Intrusion ◽  
Recent Advances

1. Bilal, A. Jourabloo, M. Ye, X. Liu, and L. Ren. Do Convolutional Neural Networks Learn Class Hierarchy? IEEE Transactions on Visualization and Computer Graphics, 24(1):152–162, Jan. 2018. 2. M. Carney, B. Webster, I. Alvarado, K. Phillips, N. Howell, J. Griffith, J. Jongejan, A. Pitaru, and A. Chen. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20. ACM, Honolulu, HI, USA, 2020. 3. A. Karpathy. CS231n Convolutional Neural Networks for Visual Recognition, 2016 4. M. Kahng, N. Thorat, D. H. Chau, F. B. Viegas, and M. Wattenberg. GANLab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation. IEEE Transactions on Visualization and Computer Graphics, 25(1):310–320, Jan. 2019. 5. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Deep Learning Workshop, 2015 6. M. Kahng, P. Y. Andrews, A. Kalro, and D. H. Chau. ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models. IEEE Transactions on Visualization and Computer Graphics, 24(1):88–97, Jan. 2018. 7. https://cs231n.github.io/convolutional-networks/ 8. https://www.analyticsvidhya.com/blog/2020/02/learn-imageclassification-cnn-convolutional-neural-networks-3-datasets/ 9. https://towardsdatascience.com/understanding-cnn-convolutionalneural- network-69fd626ee7d4 10. https://medium.com/@birdortyedi_23820/deep-learning-lab-episode-2- cifar- 10-631aea84f11e 11. J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, and T. Chen. Recent advances in convolutional neural networks. Pattern Recognition, 77:354–377, May 2018. 12. Hamid, Y., Shah, F.A. and Sugumaram, M. (2014), ―Wavelet neural network model for network intrusion detection system‖, International Journal of Information Technology, Vol. 11 No. 2, pp. 251-263 13. G Sreeram , S Pradeep, K SrinivasRao , B.Deevan Raju , Parveen Nikhat , ― Moving ridge neuronal espionage network simulation for reticulum invasion sensing‖. International Journal of Pervasive Computing and Communications.https://doi.org/10.1108/IJPCC-05- 2020-0036 14. E. Stevens, L. Antiga, and T. Viehmann. Deep Learning with PyTorch. O’Reilly Media, 2019. 15. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Deep Learning Workshop, 2015. 16. Aman Dureja, Payal Pahwa, ―Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks‖, Recent Advances in Computer Science , Vol 2, Issue 3, 2019 ,PP-156-161 17. https://missinglink.ai/guides/neural-network-concepts/7-types-neuralnetwork-activation-functions-right/

scholarly journals Brain network constraints and recurrent neural networks reproduce unique trajectories and state transitions seen over the span of minutes in resting-state fMRI

2020 ◽  
Vol 4 (2) ◽  
pp. 448-466
Author(s):  
Amrit Kashyap ◽  
Shella Keilholz
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Resting State ◽  
Large Scale ◽  
Brain Network ◽  
Generative Models ◽  
Network Activity ◽  
Fmri Data ◽  
State Transitions ◽  
Whole Brain

Large-scale patterns of spontaneous whole-brain activity seen in resting-state functional magnetic resonance imaging (rs-fMRI) are in part believed to arise from neural populations interacting through the structural network (Honey, Kötter, Breakspear, & Sporns, 2007 ). Generative models that simulate this network activity, called brain network models (BNM), are able to reproduce global averaged properties of empirical rs-fMRI activity such as functional connectivity (FC) but perform poorly in reproducing unique trajectories and state transitions that are observed over the span of minutes in whole-brain data (Cabral, Kringelbach, & Deco, 2017 ; Kashyap & Keilholz, 2019 ). The manuscript demonstrates that by using recurrent neural networks, it can fit the BNM in a novel way to the rs-fMRI data and predict large amounts of variance between subsequent measures of rs-fMRI data. Simulated data also contain unique repeating trajectories observed in rs-fMRI, called quasiperiodic patterns (QPP), that span 20 s and complex state transitions observed using k-means analysis on windowed FC matrices (Allen et al., 2012 ; Majeed et al., 2011 ). Our approach is able to estimate the manifold of rs-fMRI dynamics by training on generating subsequent time points, and it can simulate complex resting-state trajectories better than the traditional generative approaches.

Generative models for the transfer of knowledge in seismic interpretation with deep learning

2021 ◽  
Vol 40 (7) ◽  
pp. 534-542
Author(s):  
Ricard Durall ◽  
Valentin Tschannen ◽  
Norman Ettrich ◽  
Janis Keuper
Keyword(s):  
Neural Networks ◽  
Real Data ◽  
Generative Models ◽  
Training Data ◽  
Transfer Of Knowledge ◽  
Validation Data ◽  
Generalization Problem ◽  
Novel Method ◽  
Segmentation Task

Interpreting seismic data requires the characterization of a number of key elements such as the position of faults and main reflections, presence of structural bodies, and clustering of areas exhibiting a similar amplitude versus angle response. Manual interpretation of geophysical data is often a difficult and time-consuming task, complicated by lack of resolution and presence of noise. In recent years, approaches based on convolutional neural networks have shown remarkable results in automating certain interpretative tasks. However, these state-of-the-art systems usually need to be trained in a supervised manner, and they suffer from a generalization problem. Hence, it is highly challenging to train a model that can yield accurate results on new real data obtained with different acquisition, processing, and geology than the data used for training. In this work, we introduce a novel method that combines generative neural networks with a segmentation task in order to decrease the gap between annotated training data and uninterpreted target data. We validate our approach on two applications: the detection of diffraction events and the picking of faults. We show that when transitioning from synthetic training data to real validation data, our workflow yields superior results compared to its counterpart without the generative network.

scholarly journals Generative models of rich clubs in Hebbian neuronal networks and large-scale human brain networks

2014 ◽  
Vol 369 (1653) ◽  
pp. 20130531 ◽  
Author(s):  
Petra E. Vértes ◽  
Aaron Alexander-Bloch ◽  
Edward T. Bullmore
Keyword(s):  
Neural Networks ◽  
Human Brain ◽  
Large Scale ◽  
Hebbian Learning ◽  
Brain Connectivity ◽  
Brain Networks ◽  
Brain Network ◽  
Generative Models ◽  
Adaptive Behaviour ◽  
Topological Term

Rich clubs arise when nodes that are ‘rich’ in connections also form an elite, densely connected ‘club’. In brain networks, rich clubs incur high physical connection costs but also appear to be especially valuable to brain function. However, little is known about the selection pressures that drive their formation. Here, we take two complementary approaches to this question: firstly we show, using generative modelling, that the emergence of rich clubs in large-scale human brain networks can be driven by an economic trade-off between connection costs and a second, competing topological term. Secondly we show, using simulated neural networks, that Hebbian learning rules also drive the emergence of rich clubs at the microscopic level, and that the prominence of these features increases with learning time. These results suggest that Hebbian learning may provide a neuronal mechanism for the selection of complex features such as rich clubs. The neural networks that we investigate are explicitly Hebbian, and we argue that the topological term in our model of large-scale brain connectivity may represent an analogous connection rule. This putative link between learning and rich clubs is also consistent with predictions that integrative aspects of brain network organization are especially important for adaptive behaviour.

Sign in / Sign up

Export Citation Format

Share Document