Early Prediction of DNN Activation Using Hierarchical Computations

Deep Neural Networks (DNNs) have set state-of-the-art performance numbers in diverse fields of electronics (computer vision, voice recognition), biology, bioinformatics, etc. However, the process of learning (training) from the data and application of the learnt information (inference) process requires huge computational resources. Approximate computing is a common method to reduce computation cost, but it introduces loss in task accuracy, which limits their application. Using an inherent property of Rectified Linear Unit (ReLU), a popular activation function, we propose a mathematical model to perform MAC operation using reduced precision for predicting negative values early. We also propose a method to perform hierarchical computation to achieve the same results as IEEE754 full precision compute. Applying this method on ResNet50 and VGG16 shows that up to 80% of ReLU zeros (which is 50% of all ReLU outputs) can be predicted and detected early by using just 3 out of 23 mantissa bits. This method is equally applicable to other floating-point representations.

Multiscale representations of community structures in attractor neural networks

Our cognition relies on the ability of the brain to segment hierarchically structured events on multiple scales. Recent evidence suggests that the brain performs this event segmentation based on the structure of state-transition graphs behind sequential experiences. However, the underlying circuit mechanisms are poorly understood. In this paper we propose an extended attractor network model for graph-based hierarchical computation which we call the Laplacian associative memory. This model generates multiscale representations for communities (clusters) of associative links between memory items, and the scale is regulated by the heterogenous modulation of inhibitory circuits. We analytically and numerically show that these representations correspond to graph Laplacian eigenvectors, a popular method for graph segmentation and dimensionality reduction. Finally, we demonstrate that our model exhibits chunked sequential activity patterns resembling hippocampal theta sequences. Our model connects graph theory and attractor dynamics to provide a biologically plausible mechanism for abstraction in the brain.

A direct interareal feedback-to-feedforward circuit in primate visual cortex

The mammalian sensory neocortex consists of hierarchically organized areas reciprocally connected via feedforward (FF) and feedback (FB) circuits. Several theories of hierarchical computation ascribe the bulk of the computational work of the cortex to looped FF-FB circuits between pairs of cortical areas. However, whether such corticocortical loops exist remains unclear. In higher mammals, individual FF-projection neurons send afferents almost exclusively to a single higher-level area. However, it is unclear whether FB-projection neurons show similar area-specificity, and whether they influence FF-projection neurons directly or indirectly. Using viral-mediated monosynaptic circuit tracing in macaque primary visual cortex (V1), we show that V1 neurons sending FF projections to area V2 receive monosynaptic FB inputs from V2, but not other V1-projecting areas. We also find monosynaptic FB-to-FB neuron contacts as a second motif of FB connectivity. Our results support the existence of FF-FB loops in primate cortex, and suggest that FB can rapidly and selectively influence the activity of incoming FF signals.

Learning from unexpected events in the neocortical microcircuit

Scientists have long conjectured that the neocortex learns the structure of the environment in a predictive, hierarchical manner. To do so, expected, predictable features are differentiated from unexpected ones by comparing bottom-up and top-down streams of data. It is theorized that the neocortex then changes the representation of incoming stimuli, guided by differences in the responses to expected and unexpected events. Such differences in cortical responses have been observed; however, it remains unknown whether these unexpected event signals govern subsequent changes in the brain’s stimulus representations, and, thus, govern learning. Here, we show that unexpected event signals predict subsequent changes in responses to expected and unexpected stimuli in individual neurons and distal apical dendrites that are tracked over a period of days. These findings were obtained by observing layer 2/3 and layer 5 pyramidal neurons in primary visual cortex of awake, behaving mice using two-photon calcium imaging. We found that many neurons in both layers 2/3 and 5 showed large differences between their responses to expected and unexpected events. These unexpected event signals also determined how the responses evolved over subsequent days, in a manner that was different between the somata and distal apical dendrites. This difference between the somata and distal apical dendrites may be important for hierarchical computation, given that these two compartments tend to receive bottom-up and top-down information, respectively. Together, our results provide novel evidence that the neocortex indeed instantiates a predictive hierarchical model in which unexpected events drive learning.

Associative memory networks for graph-based abstraction

Our cognition relies on the ability of the brain to segment hierarchically structured events on multiple scales. Recent evidence suggests that the brain performs this event segmentation based on the structure of state-transition graphs behind sequential experiences. However, the underlying circuit mechanisms are only poorly understood. In this paper, we propose an extended attractor network model for the graph-based hierarchical computation, called as Laplacian associative memory. This model generates multiscale representations for communities (clusters) of associative links between memory items, and the scale is regulated by heterogenous modulation of inhibitory circuits. We analytically and numerically show that these representations correspond to graph Laplacian eigenvectors, a popular method for graph segmentation and dimensionality reduction. Finally, we demonstrate that our model with asymmetricity exhibits chunking resembling to hippocampal theta sequences. Our model connects graph theory and the attractor dynamics to provide a biologically plausible mechanism for abstraction in the brain.

Multiscale integration organizes hierarchical computation in human auditory cortex

To derive meaning from sound, the brain must integrate information across tens (e.g. phonemes) to hundreds (e.g. words) of milliseconds, but the neural computations that enable multiscale integration remain unclear. Prior evidence suggests that human auditory cortex analyzes sound using both generic acoustic features (e.g. spectrotemporal modulation) and category-specific computations, but how these putatively distinct computations integrate temporal information is unknown. To answer this question, we developed a novel method to estimate neural integration periods and applied the method to intracranial recordings from human epilepsy patients. We show that integration periods increase three-fold as one ascends the auditory cortical hierarchy. Moreover, we find that electrodes with short integration periods (~50-150 ms) respond selectively to spectrotemporal modulations, while electrodes with long integration periods (~200-300 ms) show prominent selectivity for sound categories such as speech and music. These findings reveal how multiscale temporal analysis organizes hierarchical computation in human auditory cortex.

A direct interareal feedback-to-feedforward circuit in primate visual cortex

The mammalian sensory neocortex consists of hierarchically organized areas reciprocally connected via feedforward (FF) and feedback (FB) circuits. Several theories of hierarchical computation ascribe the bulk of the computational work of the cortex to looped FF-FB circuits between pairs of cortical areas. However, whether such corticocortical loops exist remains unclear. In higher mammals, FF projections send afferents almost exclusively to a single higher-level area. However, it is unclear whether FB projections show similar area-specificity, and whether they influence FF-projection neurons directly or indirectly. Using viral-mediated monosynaptic circuit tracing in macaque visual cortex, we find that neurons sending FF projections to a higher-level area receive monosynaptic FB inputs exclusively from that area. We also find monosynaptic FB-to-FB neuron contacts as a second motif of FB connectivity. Our results support the existence of FF-FB loops in primate cortex, and suggest that FB can rapidly and selectively influence the activity of incoming FF signals.

A direct interareal feedback-to-feedforward circuit in primate visual cortex

ABSTRACTThe mammalian sensory neocortex consists of hierarchically organized areas reciprocally connected via feedforward (FF) and feedback (FB) circuits. Several theories of hierarchical computation ascribe the bulk of the computational work of the cortex to looped FF-FB circuits between pairs of cortical areas. However, whether such corticocortical loops exist remains unclear. In higher mammals, FF projections send afferents almost exclusively to a single higher-level area. However, it is unclear whether FB projections show similar area-specificity, and whether they influence FF-projection neurons directly or indirectly. Using viral-mediated monosynaptic circuit tracing in macaque visual cortex, we find that neurons sending FF projections to a higher-level area receive monosynaptic FB inputs exclusively from that area. We also find monosynaptic FB-to-FB neuron contacts as a second motif of FB connectivity. Our results support the existence of FF-FB loops in primate cortex, and suggest that FB can rapidly and selectively influence the activity of incoming FF signals.

A large-scale, standardized physiological survey reveals higher order coding throughout the mouse visual cortex

SummaryTo understand how the brain processes sensory information to guide behavior, we must know how stimulus representations are transformed throughout the visual cortex. Here we report an open, large-scale physiological survey of neural activity in the awake mouse visual cortex: the Allen Brain Observatory Visual Coding dataset. This publicly available dataset includes cortical activity from nearly 60,000 neurons collected from 6 visual areas, 4 layers, and 12 transgenic mouse lines from 221 adult mice, in response to a systematic set of visual stimuli. Using this dataset, we reveal functional differences across these dimensions and show that visual cortical responses are sparse but correlated. Surprisingly, responses to different stimuli are largely independent, e.g. whether a neuron responds to natural scenes provides no information about whether it responds to natural movies or to gratings. We show that these phenomena cannot be explained by standard local filter-based models, but are consistent with multi-layer hierarchical computation, as found in deeper layers of standard convolutional neural networks.