scholarly journals From statistical inference to a differential learning rule for stochastic neural networks

2018 ◽  
Vol 8 (6) ◽  
pp. 20180033 ◽  
Author(s):  
Luca Saglietti ◽  
Federica Gerace ◽  
Alessandro Ingrosso ◽  
Carlo Baldassi ◽  
Riccardo Zecchina
Keyword(s):  
Neural Networks ◽  
Learning Rule ◽  
Generative Models ◽  
Stochastic Neural Networks ◽  
Differential Learning ◽  
Correlated Patterns ◽  
Classification Tasks ◽  
Hidden Neurons ◽  
External Stimuli ◽  
The Relationship

Stochastic neural networks are a prototypical computational device able to build a probabilistic representation of an ensemble of external stimuli. Building on the relationship between inference and learning, we derive a synaptic plasticity rule that relies only on delayed activity correlations, and that shows a number of remarkable features. Our delayed-correlations matching (DCM) rule satisfies some basic requirements for biological feasibility: finite and noisy afferent signals, Dale’s principle and asymmetry of synaptic connections, locality of the weight update computations. Nevertheless, the DCM rule is capable of storing a large, extensive number of patterns as attractors in a stochastic recurrent neural network, under general scenarios without requiring any modification: it can deal with correlated patterns, a broad range of architectures (with or without hidden neuronal states), one-shot learning with the palimpsest property, all the while avoiding the proliferation of spurious attractors. When hidden units are present, our learning rule can be employed to construct Boltzmann machine-like generative models, exploiting the addition of hidden neurons in feature extraction and classification tasks.

scholarly journals Biologically Plausible Sequence Learning with Spiking Neural Networks

2020 ◽  
Vol 34 (02) ◽  
pp. 1316-1323
Author(s):  
Zuozhu Liu ◽  
Thiparat Chotibut ◽  
Christopher Hillar ◽  
Shaowei Lin
Keyword(s):  
Neural Networks ◽  
Sequence Learning ◽  
Nervous Activity ◽  
Learning Rule ◽  
Generative Models ◽  
Spike Timing ◽  
Hopfield Network ◽  
Spiking Neural Networks ◽  
Theoretical Ground ◽  
Time Model

Motivated by the celebrated discrete-time model of nervous activity outlined by McCulloch and Pitts in 1943, we propose a novel continuous-time model, the McCulloch-Pitts network (MPN), for sequence learning in spiking neural networks. Our model has a local learning rule, such that the synaptic weight updates depend only on the information directly accessible by the synapse. By exploiting asymmetry in the connections between binary neurons, we show that MPN can be trained to robustly memorize multiple spatiotemporal patterns of binary vectors, generalizing the ability of the symmetric Hopfield network to memorize static spatial patterns. In addition, we demonstrate that the model can efficiently learn sequences of binary pictures as well as generative models for experimental neural spike-train data. Our learning rule is consistent with spike-timing-dependent plasticity (STDP), thus providing a theoretical ground for the systematic design of biologically inspired networks with large and robust long-range sequence storage capacity.

scholarly journals Back-propagation learning in deep Spike-By-Spike networks

2019 ◽  
Author(s):  
David Rotermund ◽  
Klaus R. Pawelzik
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Action Potentials ◽  
Back Propagation ◽  
Learning Rule ◽  
Building Blocks ◽  
Classification Performance ◽  
Generative Models ◽  
Learning Method ◽  
Discrete Action

ABSTRACTNeural networks are important building blocks in technical applications. These artificial neural networks (ANNs) rely on noiseless continuous signals in stark contrast to the discrete action potentials stochastically exchanged among the neurons in real brains. A promising approach towards bridging this gap are the Spike-by-Spike (SbS) networks which represent a compromise between non-spiking and spiking versions of generative models that perform inference on their inputs. What is still missing are algorithms for finding weight sets that would optimize the output performances of deep SbS networks with many layers.Here, a learning rule for hierarchically organized SbS networks is derived. The properties of this approach are investigated and its functionality demonstrated by simulations. In particular, a Deep Convolutional SbS network for classifying handwritten digits (MNIST) is presented. When applied together with an optimizer this learning method achieves a classification performance of roughly 99.3% on the MNIST test data. Thereby it approaches the benchmark results of ANNs without extensive parameter optimization. We envision that with this learning rule SBS networks will provide a new basis for research in neuroscience and for technical applications, especially when they become implemented on specialized computational hardware.

scholarly journals Building a Deep Learning Model to Generate Human Readable Text Using Recurrent Neural Networks and LSTM

2021 ◽  
Author(s):  
Anasse HANAFI ◽  
Mohammed BOUHORMA ◽  
Lotfi ELAACHAK
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Binary Classification ◽  
Generative Models ◽  
Large Field ◽  
Classification Problems ◽  
Classification Tasks ◽  
Multi Class Classification ◽  
Deep Learning Model ◽  
Readable Text

Abstract Machine learning (ML) is a large field of study that overlaps with and inherits ideas from many related fields such as artificial intelligence (AI). The main focus of the field is learning from previous experiences. Classification in ML is a supervised learning method, in which the computer program learns from the data given to it and make new classifications. There are many different types of classification tasks in ML and dedicated approaches to modeling that may be used for each. For example, classification predictive modeling involves assigning a class label to input samples, binary classification refers to predicting one of two classes and multi-class classification involves predicting one of more than two categories. Recurrent Neural Networks (RNNs) are very powerful sequence models for classification problems, however, in this paper, we will use RNNs as generative models, which means they can learn the sequences of a problem and then generate entirely a new sequence for the problem domain, with the hope to better control the output of the generated text, because it is not always possible to learn the exact distribution of the data either implicitly or explicitly.

scholarly journals Over-Parameterization and Generalization in Audio Classification

2021 ◽  
Author(s):  
Khaled Koutini ◽  
Hamid Eghbal-zadeh ◽  
Florian Henkel ◽  
Jan Schlüter ◽  
Gerhard Widmer
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Recording Device ◽  
Classification Models ◽  
Audio Classification ◽  
Scene Classification ◽  
Machine Listening ◽  
Substantial Problem ◽  
Classification Tasks ◽  
The Relationship

Convolutional Neural Networks (CNNs) have been dominating classification tasks in various domains, such as machine vision, machine listening, and natural language processing. In machine listening, while generally exhibiting very good generalization capabilities, CNNs are sensitive to the specific audio recording device used, which has been recognized as a substantial problem in the acoustic scene classification (DCASE) community. In this study, we investigate the relationship between over-parameterization of acoustic scene classification models, and their resulting generalization abilities. Our results indicate that increasing width improves generalization to unseen devices, even without an increase in the number of parameters.

Ozone Concentration Prediction using Artificial Neural Networks

2017 ◽  
Vol 68 (10) ◽  
pp. 2224-2227 ◽  
Author(s):  
Camelia Gavrila
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Ozone Concentration ◽  
Meteorological Data ◽  
Global Solar Radiation ◽  
Time Data ◽  
Marquardt Algorithm ◽  
Sensitivity Tests ◽  
Artificial Neural ◽  
The Relationship

The aim of this paper is to determine a mathematical model which establishes the relationship between ozone levels together with other meteorological data and air quality. The model is valid for any season and for any area and is based on real-time data measured in Bucharest and its surroundings. This study is based on research using artificial neural networks to model nonlinear relationships between the concentration of immission of ozone and the meteorological factors: relative humidity (RH), global solar radiation (SR), air temperature (TEMP). The ozone concentration depends on following primary pollutants: nitrogen oxides (NO, NO2), carbon monoxide (CO). To achieve this, the Levenberg-Marquardt algorithm was implemented in Scilab, a numerical computation software. Performed sensitivity tests proved the robustness of the model and its applicability in predicting the ozone on short-term.

The Aristocracy in England and Tuscany, 1000 - 1250

2019 ◽  
Author(s):  
Peter Coss
Keyword(s):  
Time Frame ◽  
Grand Narrative ◽  
English History ◽  
Internal Dynamics ◽  
Italian Experience ◽  
English Society ◽  
History Of ◽  
External Stimuli ◽  
The Relationship ◽  
Interpretative Framework

Part I of this book is an in-depth examination of the characteristics of the Tuscan aristocracy across the first two and a half centuries of the second millennium, as studied by Italian historians and others working within the Italian tradition: their origins, interests, strategies for survival and exercise of power; the structure and the several levels of aristocracy and how these interrelated; the internal dynamics and perceptions that governed aristocratic life; and the relationship to non-aristocratic sectors of society. It will look at how aristocratic society changed across this period and how far changes were internally generated as opposed to responses from external stimuli. The relationship between the aristocracy and public authority will also be examined. Part II of the book deals with England. The aim here is not a comparative study but to bring insights drawn from Tuscan history and Tuscan historiography into play in understanding the evolution of English society from around the year 1000 to around 1250. This part of the book draws on the breadth of English historiography but is also guided by the Italian experience. The book challenges the interpretative framework within which much English history of this period tends to be written—that is to say the grand narrative which revolves around Magna Carta and English exceptionalism—and seeks to avoid dangers of teleology, of idealism, and of essentialism. By offering a study of the aristocracy across a wide time-frame and with themes drawn from Italian historiography, I hope to obviate these tendencies and to appreciate the aristocracy firmly within its own contexts.

scholarly journals Scene Complexity: A New Perspective on Understanding the Scene Semantics of Remote Sensing and Designing Image-Adaptive Convolutional Neural Networks

2021 ◽  
Vol 13 (4) ◽  
pp. 742
Author(s):  
Jian Peng ◽  
Xiaoming Mei ◽  
Wenbo Li ◽  
Liang Hong ◽  
Bingyu Sun ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Neural Networks ◽  
Fundamental Problem ◽  
Semantic Representation ◽  
Feature Learning ◽  
Essential Elements ◽  
Complex Scene ◽  
Feature Representations ◽  
The Right ◽  
The Relationship

Scene understanding of remote sensing images is of great significance in various applications. Its fundamental problem is how to construct representative features. Various convolutional neural network architectures have been proposed for automatically learning features from images. However, is the current way of configuring the same architecture to learn all the data while ignoring the differences between images the right one? It seems to be contrary to our intuition: it is clear that some images are easier to recognize, and some are harder to recognize. This problem is the gap between the characteristics of the images and the learning features corresponding to specific network structures. Unfortunately, the literature so far lacks an analysis of the two. In this paper, we explore this problem from three aspects: we first build a visual-based evaluation pipeline of scene complexity to characterize the intrinsic differences between images; then, we analyze the relationship between semantic concepts and feature representations, i.e., the scalability and hierarchy of features which the essential elements in CNNs of different architectures, for remote sensing scenes of different complexity; thirdly, we introduce CAM, a visualization method that explains feature learning within neural networks, to analyze the relationship between scenes with different complexity and semantic feature representations. The experimental results show that a complex scene would need deeper and multi-scale features, whereas a simpler scene would need lower and single-scale features. Besides, the complex scene concept is more dependent on the joint semantic representation of multiple objects. Furthermore, we propose the framework of scene complexity prediction for an image and utilize it to design a depth and scale-adaptive model. It achieves higher performance but with fewer parameters than the original model, demonstrating the potential significance of scene complexity.

Sign in / Sign up

Export Citation Format

Share Document