scholarly journals Learning in Convolutional Neural Networks Accelerated by Transfer Entropy

Entropy ◽  
2021 ◽  
Vol 23 (9) ◽  
pp. 1218
Author(s):  
Adrian Moldovan ◽  
Angel Caţaron ◽  
Răzvan Andonie
Keyword(s):  
Information Transfer ◽  
Effective Connectivity ◽  
Transfer Entropy ◽  
Learning Mechanisms ◽  
Feedback Parameter ◽  
Smoothing Factor ◽  
Computational Overhead ◽  
Neural Information ◽  
Input Sample ◽  
Feedback Connections

Recently, there is a growing interest in applying Transfer Entropy (TE) in quantifying the effective connectivity between artificial neurons. In a feedforward network, the TE can be used to quantify the relationships between neuron output pairs located in different layers. Our focus is on how to include the TE in the learning mechanisms of a Convolutional Neural Network (CNN) architecture. We introduce a novel training mechanism for CNN architectures which integrates the TE feedback connections. Adding the TE feedback parameter accelerates the training process, as fewer epochs are needed. On the flip side, it adds computational overhead to each epoch. According to our experiments on CNN classifiers, to achieve a reasonable computational overhead–accuracy trade-off, it is efficient to consider only the inter-neural information transfer of the neuron pairs between the last two fully connected layers. The TE acts as a smoothing factor, generating stability and becoming active only periodically, not after processing each input sample. Therefore, we can consider the TE is in our model a slowly changing meta-parameter.

scholarly journals Learning in Feedforward Neural Networks Accelerated by Transfer Entropy

Entropy ◽  
2020 ◽  
Vol 22 (1) ◽  
pp. 102 ◽  
Author(s):  
Adrian Moldovan ◽  
Angel Caţaron ◽  
Răzvan Andonie
Keyword(s):  
Neural Networks ◽  
Time Series ◽  
Information Transfer ◽  
Feedforward Neural Networks ◽  
Theoretical Method ◽  
Transfer Entropy ◽  
Training Algorithms ◽  
Training Algorithm ◽  
Neural Connections ◽  
Feedback Connections

Current neural networks architectures are many times harder to train because of the increasing size and complexity of the used datasets. Our objective is to design more efficient training algorithms utilizing causal relationships inferred from neural networks. The transfer entropy (TE) was initially introduced as an information transfer measure used to quantify the statistical coherence between events (time series). Later, it was related to causality, even if they are not the same. There are only few papers reporting applications of causality or TE in neural networks. Our contribution is an information-theoretical method for analyzing information transfer between the nodes of feedforward neural networks. The information transfer is measured by the TE of feedback neural connections. Intuitively, TE measures the relevance of a connection in the network and the feedback amplifies this connection. We introduce a backpropagation type training algorithm that uses TE feedback connections to improve its performance.

Effective Connectivity Extracted from Resting-State fMRI Images Using Transfer Entropy

IRBM ◽  
2021 ◽  
Author(s):  
Z. Wu ◽  
X. Chen ◽  
M. Gao ◽  
M. Hong ◽  
Z. He ◽  
...  
Keyword(s):  
Resting State ◽  
Effective Connectivity ◽  
Resting State Fmri ◽  
Transfer Entropy

scholarly journals A Low-Order Model of Biological Neural Networks

2011 ◽  
Vol 23 (10) ◽  
pp. 2626-2682
Author(s):  
James Ting-Ho Lo
Keyword(s):  
Neural Networks ◽  
Logic Gate ◽  
Dendritic Tree ◽  
Spiking Neurons ◽  
Learning Mechanisms ◽  
Order Model ◽  
Biological Neural Networks ◽  
Feedback Connections ◽  
Component Models ◽  
Low Order

A biologically plausible low-order model (LOM) of biological neural networks is proposed. LOM is a recurrent hierarchical network of models of dendritic nodes and trees; spiking and nonspiking neurons; unsupervised, supervised covariance and accumulative learning mechanisms; feedback connections; and a scheme for maximal generalization. These component models are motivated and necessitated by making LOM learn and retrieve easily without differentiation, optimization, or iteration, and cluster, detect, and recognize multiple and hierarchical corrupted, distorted, and occluded temporal and spatial patterns. Four models of dendritic nodes are given that are all described as a hyperbolic polynomial that acts like an exclusive-OR logic gate when the model dendritic nodes input two binary digits. A model dendritic encoder that is a network of model dendritic nodes encodes its inputs such that the resultant codes have an orthogonality property. Such codes are stored in synapses by unsupervised covariance learning, supervised covariance learning, or unsupervised accumulative learning, depending on the type of postsynaptic neuron. A masking matrix for a dendritic tree, whose upper part comprises model dendritic encoders, enables maximal generalization on corrupted, distorted, and occluded data. It is a mathematical organization and idealization of dendritic trees with overlapped and nested input vectors. A model nonspiking neuron transmits inhibitory graded signals to modulate its neighboring model spiking neurons. Model spiking neurons evaluate the subjective probability distribution (SPD) of the labels of the inputs to model dendritic encoders and generate spike trains with such SPDs as firing rates. Feedback connections from the same or higher layers with different numbers of unit-delay devices reflect different signal traveling times, enabling LOM to fully utilize temporally and spatially associated information. Biological plausibility of the component models is discussed. Numerical examples are given to demonstrate how LOM operates in retrieving, generalizing, and unsupervised and supervised learning.

EEG TRANSFER ENTROPY TRACKS CHANGES IN INFORMATION TRANSFER ON THE ONSET OF VISION

2012 ◽  
Vol 17 ◽  
pp. 9-18 ◽  
Author(s):  
M. D. MADULARA ◽  
P. A. B. FRANCISCO ◽  
S. NAWANG ◽  
D. C. AROGANCIA ◽  
C. J. CELLUCCI ◽  
...  
Keyword(s):  
Mutual Information ◽  
Information Transfer ◽  
Occipital Lobe ◽  
Transfer Entropy ◽  
The Other ◽  
Information Measures ◽  
Eyes Closed ◽  
Eyes Open ◽  
Almost All ◽  
Nonlinear Correlations

We investigate the pairwise mutual information and transfer entropy of ten-channel, free-running electroencephalographs measured from thirteen subjects under two behavioral conditions: eyes open resting and eyes closed resting. Mutual information measures nonlinear correlations; transfer entropy determines the directionality of information transfer. For all channel pairs, mutual information is generally lower with eyes open compared to eyes closed indicating that EEG signals at different scalp sites become more dissimilar as the visual system is engaged. On the other hand, transfer entropy increases on average by almost two-fold when the eyes are opened. The largest one-way transfer entropies are to and from the Oz site consistent with the involvement of the occipital lobe in vision. The largest net transfer entropies are from F3 and F4 to almost all the other scalp sites.

scholarly journals The dynamics of information-driven coordination phenomena: A transfer entropy analysis

2016 ◽  
Vol 2 (4) ◽  
pp. e1501158 ◽  
Author(s):  
Javier Borge-Holthoefer ◽  
Nicola Perra ◽  
Bruno Gonçalves ◽  
Sandra González-Bailón ◽  
Alex Arenas ◽  
...  
Keyword(s):  
Information Transfer ◽  
Social Systems ◽  
Transfer Entropy ◽  
System Level ◽  
Directed Network ◽  
Characteristic Time Scale ◽  
Entropy Analysis ◽  
Collective Phenomena ◽  
Social Phenomena ◽  
Social Events

Data from social media provide unprecedented opportunities to investigate the processes that govern the dynamics of collective social phenomena. We consider an information theoretical approach to define and measure the temporal and structural signatures typical of collective social events as they arise and gain prominence. We use the symbolic transfer entropy analysis of microblogging time series to extract directed networks of influence among geolocalized subunits in social systems. This methodology captures the emergence of system-level dynamics close to the onset of socially relevant collective phenomena. The framework is validated against a detailed empirical analysis of five case studies. In particular, we identify a change in the characteristic time scale of the information transfer that flags the onset of information-driven collective phenomena. Furthermore, our approach identifies an order-disorder transition in the directed network of influence between social subunits. In the absence of clear exogenous driving, social collective phenomena can be represented as endogenously driven structural transitions of the information transfer network. This study provides results that can help define models and predictive algorithms for the analysis of societal events based on open source data.

scholarly journals A Feedback Model of Visual Attention

2004 ◽  
Vol 16 (2) ◽  
pp. 219-237 ◽  
Author(s):  
M. W. Spratling ◽  
M. H. Johnson
Keyword(s):  
Contextual Cueing ◽  
Common Mechanism ◽  
Biased Competition ◽  
Feedback Model ◽  
Cortical Feedback ◽  
Neural Information ◽  
Neurophysiological Data ◽  
Neural Information Processing ◽  
Ground Segmentation ◽  
Feedback Connections

Feedback connections are a prominent feature of cortical anatomy and are likely to have a significant functional role in neural information processing. We present a neural network model of cortical feedback that successfully simulates neurophysiological data associated with attention. In this domain, our model can be considered a more detailed, and biologically plausible, implementation of the biased competition model of attention. However, our model is more general as it can also explain a variety of other top-down processes in vision, such as figure/ground segmentation and contextual cueing. This model thus suggests that a common mechanism, involving cortical feedback pathways, is responsible for a range of phenomena and provides a unified account of currently disparate areas of research.

scholarly journals Structure-Function Relationships behind the Phenomenon of Cognitive Resilience in Neurology: Insights for Neuroscience and Medicine

2014 ◽  
Vol 2014 ◽  
pp. 1-28 ◽  
Author(s):  
David Rudrauf
Keyword(s):  
Structure Function ◽  
Information Transfer ◽  
Computational Models ◽  
Effective Connectivity ◽  
Treatment Strategies ◽  
Brain Structures ◽  
Future Research ◽  
Self Awareness ◽  
Normal Functions ◽  
And Control

The phenomenon of cognitive resilience, that is, the dynamical preservation of normal functions despite neurological disorders, demonstrates that cognition can be highly robust to devastating brain injury. Here, cognitive resilience is considered across a range of neurological conditions. Simple computational models of structure-function relationships are used to discuss hypotheses about the neural mechanisms of resilience. Resilience expresses functional redundancies in brain networks and suggests a process of dynamic rerouting of brain signals. This process is underlined by a global renormalization of effective connectivity, capable of restoring information transfer between spared brain structures via alternate pathways. Local mechanisms of synaptic plasticity mediate the renormalization at the lowest level of implementation, but it is also driven by top-down cognition, with a key role of self-awareness in fostering resilience. The presence of abstraction layers in brain computation and networking is hypothesized to account for the renormalization process. Future research directions and challenges are discussed regarding the understanding and control of resilience based on multimodal neuroimaging and computational neuroscience. The study of resilience will illuminate ways by which the brain can overcome adversity and help inform prevention and treatment strategies. It is relevant to combating the negative neuropsychological impact of aging and fostering cognitive enhancement.

Sign in / Sign up

Export Citation Format

Share Document