Decoding of human identity by computer vision and neuronal vision

SummaryExtracting meaning from a dynamic and variable flow of incoming information is a major goal of both natural and artificial intelligence. Computer vision (CV) guided by deep learning (DL) has made significant strides in recognizing a specific identity despite highly variable attributes1,2. This is the same challenge faced by the nervous system and partially addressed by the concept cells—neurons exhibiting selective firing in response to specific persons/places, described in the human medial temporal lobe (MTL)3–6. Yet, access to neurons representing a particular concept is limited due to these neurons’ sparse coding. It is conceivable, however, that the information required for such decoding is present in relatively small neuronal populations. To evaluate how well neuronal populations encode identity information in natural settings, we recorded neuronal activity from multiple brain regions of nine neurosurgical epilepsy patients implanted with depth electrodes, while the subjects watched an episode of the TV series “24”. We implemented DL models that used the time-varying population neural data as inputs and decoded the visual presence of the main characters in each frame. Before training and testing the DL models, we devised a minimally supervised CV algorithm (with comparable performance against manually-labelled data7) to detect and label all the important characters in each frame. This methodology allowed us to compare “computer vision” with “neuronal vision”—footprints associated with each character present in the activity of a subset of neurons—and identify the brain regions that contributed to this decoding process. We then tested the DL models during a recognition memory task following movie viewing where subjects were asked to recognize clip segments from the presented episode. DL model activations were not only modulated by the presence of the corresponding characters but also by participants’ subjective memory of whether they had seen the clip segment, and by the associative strengths of the characters in the narrative plot. The described approach can offer novel ways to probe the representation of concepts in time-evolving dynamic behavioral tasks. Further, the results suggest that the information required to robustly decode concepts is present in the population activity of only tens of neurons even in brain regions beyond MTL.

Download Full-text

The locus of recognition memory signals in human cortex depends on the complexity of the memory representations

10.1101/2021.06.15.448519 ◽

2021 ◽

Author(s):

D. Merika W. Sanders ◽

Rosemary A. Cowell

Keyword(s):

Recognition Memory ◽

Medial Temporal Lobe ◽

Declarative Memory ◽

Memory Task ◽

Representational Content ◽

Brain Regions ◽

Complex Stimuli ◽

Memory Representations ◽

Memory Signals ◽

Feature Memory

Representational theories predict that brain regions contribute to cognition according to the information they represent (e.g., simple versus complex), contradicting the traditional notion that brain regions are specialized for cognitive functions (e.g., perception versus memory). In support of representational accounts, substantial evidence now attests that the Medial Temporal Lobe (MTL) is not specialized solely for long-term declarative memory, but underpins other functions including perception and future-imagining for complex stimuli and events. However, a complementary prediction has been less well explored, namely that the cortical locus of declarative memory may fall outside the MTL if the to-be-remembered content is sufficiently simple. Specifically, the locus should coincide with the optimal neural code for the representations being retrieved. To test this prediction, we manipulated the complexity of the to-be-remembered representations in a recognition memory task. First, participants in the scanner viewed novel 3D objects and scenes, and we used multivariate analyses to identify regions in the ventral visual-MTL pathway that preferentially coded for either simple features of the stimuli, or complex conjunctions of those features. Next, in a separate scan, we tested recognition memory for these stimuli and performed neuroimaging contrasts that revealed two memory signals ‒ feature memory and conjunction memory. Feature memory signals were found in visual cortex, while conjunction memory signals emerged in MTL. Further, the regions optimally representing features via preferential feature-coding coincided with those exhibiting feature memory signals. These findings suggest that representational content, rather than cognitive function, is the primary organizing principle in the ventral visual-MTL pathway.

Download Full-text

Medial temporal lobe high-frequency activity at feedback signals error and predicts subsequent memory during spatial learning

10.21203/rs.3.rs-194736/v1 ◽

2021 ◽

Author(s):

Rebecca Stevenson ◽

John Janecek ◽

Myra Larson ◽

Lilit Mnatsakanyan ◽

Sumeet Vadera ◽

...

Keyword(s):

Temporal Lobe ◽

High Frequency ◽

Medial Temporal Lobe ◽

Memory Performance ◽

Memory Task ◽

Object Location ◽

Opposite Pattern ◽

Feedback Processing ◽

Depth Electrodes ◽

High Frequency Activity

Abstract The ability to incorporate information about feedback is critical for associative learning. The medial temporal lobe (MTL) and prefrontal cortex (PFC) are thought to be involved in processing feedback as new associations are learned. However, the relative contributions of these regions to feedback processing and subsequent memory performance in humans are poorly understood. To address this question, we tested pre-surgical epilepsy patients with depth electrodes implanted in the MTL and PFC using a spatial memory task in which subjects learned object-location associations over time. We found increased high-frequency activity (HFA; 40-100 Hz), thought to reflect local excitatory activity, in the MTL and dorsolateral PFC (dlPFC) at feedback for high error trials. In the MTL, this HFA error signal predicted greater trial-by-trial decreases in error from one training block to the next indicating that these signals are involved in updating memory representations or modifying incorrect associations during learning. The opposite pattern of activity was observed during retrieval, with greater MTL and dlPFC HFA predicting lower error, replicating previous results from our group. Overall, these data suggest putative mechanisms for the learning of object-location associations.

Download Full-text

In vivo synaptic density relates to glucose metabolism at rest in healthy subjects, but is strongly modulated by regional differences

Journal of Cerebral Blood Flow & Metabolism ◽

10.1177/0271678x20981502 ◽

2021 ◽

pp. 0271678X2098150

Author(s):

June van Aalst ◽

Jenny Ceccarini ◽

Stefan Sunaert ◽

Patrick Dupont ◽

Michel Koole ◽

...

Keyword(s):

Glucose Metabolism ◽

Medial Temporal Lobe ◽

Regional Differences ◽

Specific Binding ◽

Aerobic Glycolysis ◽

Glucose Consumption ◽

Brain Regions ◽

Synaptic Density ◽

Energy Demands

Preclinical and postmortem studies have suggested that regional synaptic density and glucose consumption (CMRGlc) are strongly related. However, the relation between synaptic density and cerebral glucose metabolism in the human brain has not directly been assessed in vivo. Using [11C]UCB-J binding to synaptic vesicle glycoprotein 2 A (SV2A) as indicator for synaptic density and [18F]FDG for measuring cerebral glucose consumption, we studied twenty healthy female subjects (age 29.6 ± 9.9 yrs) who underwent a single-day dual-tracer protocol (GE Signa PET-MR). Global measures of absolute and relative CMRGlc and specific binding of [11C]UCB-J were indeed highly significantly correlated ( r > 0.47, p < 0.001). However, regional differences in relative [18F]FDG and [11C]UCB-J uptake were observed, with up to 19% higher [11C]UCB-J uptake in the medial temporal lobe (MTL) and up to 17% higher glucose metabolism in frontal and motor-related areas and thalamus. This pattern has a considerable overlap with the brain regions showing different levels of aerobic glycolysis. Regionally varying energy demands of inhibitory and excitatory synapses at rest may also contribute to this difference. Being unaffected by astroglial and/or microglial energy demands, changes in synaptic density in the MTL may therefore be more sensitive to early detection of pathological conditions compared to changes in glucose metabolism.

Download Full-text

Brain Activity-Based Metrics for Assessing Learning States in VR under Stress among Firefighters: An Explorative Machine Learning Approach in Neuroergonomics

Brain Sciences ◽

10.3390/brainsci11070885 ◽

2021 ◽

Vol 11 (7) ◽

pp. 885

Author(s):

Maher Abujelala ◽

Rohith Karthikeyan ◽

Oshin Tyagi ◽

Jing Du ◽

Ranjana K. Mehta

Keyword(s):

Machine Learning ◽

Environmental Conditions ◽

Brain Activity ◽

Memory Task ◽

Classification Problem ◽

Brain Regions ◽

Training Data ◽

Information Encoding ◽

Machine Learning Approach ◽

Encoding And Retrieval

The nature of firefighters` duties requires them to work for long periods under unfavorable conditions. To perform their jobs effectively, they are required to endure long hours of extensive, stressful training. Creating such training environments is very expensive and it is difficult to guarantee trainees’ safety. In this study, firefighters are trained in a virtual environment that includes virtual perturbations such as fires, alarms, and smoke. The objective of this paper is to use machine learning methods to discern encoding and retrieval states in firefighters during a visuospatial episodic memory task and explore which regions of the brain provide suitable signals to solve this classification problem. Our results show that the Random Forest algorithm could be used to distinguish between information encoding and retrieval using features extracted from fNIRS data. Our algorithm achieved an F-1 score of 0.844 and an accuracy of 79.10% if the training and testing data are obtained at similar environmental conditions. However, the algorithm’s performance dropped to an F-1 score of 0.723 and accuracy of 60.61% when evaluated on data collected under different environmental conditions than the training data. We also found that if the training and evaluation data were recorded under the same environmental conditions, the RPM, LDLPFC, RDLPFC were the most relevant brain regions under non-stressful, stressful, and a mix of stressful and non-stressful conditions, respectively.

Download Full-text

APOE ɛ4 Allele Moderates the Association Between Basal Forebrain Nuclei Volumes and Allocentric Navigation in Older Adults Without Dementia

Journal of Alzheimer s Disease ◽

10.3233/jad-215034 ◽

2022 ◽

pp. 1-17

Author(s):

Ondrej Lerch ◽

Martina Pařízková ◽

Martin Vyhnálek ◽

Zuzana Nedelská ◽

Jakub Hort ◽

...

Keyword(s):

Older Adults ◽

Basal Forebrain ◽

Medial Temporal Lobe ◽

Structural Equation ◽

Brain Aging ◽

Late Onset ◽

Brain Regions ◽

Real Space ◽

Complex Model ◽

E4 Allele

Background: Cholinergic deficit and medial temporal lobe (MTL) atrophy are hallmarks of Alzheimer’s disease (AD) leading to early allocentric spatial navigation (aSN) impairment. APOE ɛ4 allele (E4) is a major genetic risk factor for late-onset AD and contributes to cholinergic dysfunction. Basal forebrain (BF) nuclei, the major source of acetylcholine, project into multiple brain regions and, along with MTL and prefrontal cortex (PFC), are involved in aSN processing. Objective: We aimed to determine different contributions of individual BF nuclei atrophy to aSN in E4 positive and E4 negative older adults without dementia and assess whether they operate on aSN through MTL and PFC or independently from these structures. Methods: 120 participants (60 E4 positive, 60 E4 negative) from the Czech Brain Aging Study underwent structural MRI and aSN testing in real-space arena setting. Hippocampal and BF nuclei volumes and entorhinal cortex and PFC thickness were obtained. Associations between brain regions involved in aSN were assessed using MANOVA and complex model of mutual relationships was built using structural equation modelling (SEM). Results: Path analysis based on SEM modeling revealed that BF Ch1-2, Ch4p, and Ch4ai nuclei volumes were indirectly associated with aSN performance through MTL (pch1 - 2 = 0.039; pch4p = 0.042) and PFC (pch4ai = 0.044). In the E4 negative group, aSN was indirectly associated with Ch1-2 nuclei volumes (p = 0.015), while in the E4 positive group, there was indirect effect of Ch4p nucleus (p = 0.035). Conclusion: Our findings suggest that in older adults without dementia, BF nuclei affect aSN processing indirectly, through MTL and PFC, and that APOE E4 moderates these associations.

Download Full-text

Social Status and Arginine Vasotocin Neuronal Phenotypes in a Cichlid Fish

Brain Behavior and Evolution ◽

10.1159/000381251 ◽

2015 ◽

Vol 85 (3) ◽

pp. 203-213 ◽

Cited By ~ 15

Author(s):

Olinda Almeida ◽

Rui F. Oliveira

Keyword(s):

Social Status ◽

Social Behaviour ◽

Cichlid Fish ◽

Brain Regions ◽

Cell Number ◽

Arginine Vasotocin ◽

Cichlid Species ◽

Neuronal Populations ◽

Soma Size ◽

Species Specific

The nonapeptide arginine vasotocin (AVT) and its mammalian homologue arginine vasopressin play a key role in the regulation of social behaviour across vertebrates. In teleost fishes, three AVT neuronal populations have been described in the preoptic area (POA): the parvocellular (pPOA), the magnocellular (mPOA) and the gigantocellular (gPOA). Neurons from each of these areas project both to the pituitary and to other brain regions, where AVT is supposed to regulate neural circuits underlying social behaviour. However, in the fish species studied so far, there is considerable variation in which AVT neuronal populations are involved in behavioural modulation and in the direction of the effect. In this study, the association between AVT neuronal phenotypes and social status was investigated in the Mozambique tilapia (Oreochromis mossambicus). This species is an African female mouth-brooding cichlid fish in which males form breeding aggregations in which dominant males establish territories and subordinate males to act as floaters. With respect to sex differences in AVT neuronal phenotypes, females have a larger number of AVT neurons in the pPOA and mPOA. Within males, AVT appeared associated with social subordination, as indicated by the larger cell body areas of AVT neurons in mPOA and gPOA nuclei of non-territorial males. There were also positive correlations between submissive behaviour and the soma size of AVT cells in all three nuclei and AVT cell number in the mPOA. In summary, the results provide evidence for an involvement of AVT in the modulation of social behaviour in tilapia, but it was not possible to identify specific roles for specific AVT neuronal populations. The results presented here also contrast with those previously published for another cichlid species with a similar mating system, which highlights the species-specific nature of the pattern of association between AVT and social behaviour even within the same taxonomic family.

Download Full-text

Effects of amnesia on processing in the hippocampus and default mode network during a naturalistic memory task: a case study

10.31234/osf.io/e9n8c ◽

2018 ◽

Author(s):

Christiane Oedekoven ◽

James L. Keidel ◽

Stuart Anderson ◽

Angus Nisbet ◽

Chris Bird

Keyword(s):

Functional Connectivity ◽

Episodic Memory ◽

Memory Performance ◽

Memory Task ◽

Structural Mri ◽

Brain Regions ◽

Left Hippocampus ◽

Cortical Regions ◽

The Right ◽

Encoding And Retrieval

Despite their severely impaired episodic memory, individuals with amnesia are able to comprehend ongoing events. Online representations of a current event are thought to be supported by a network of regions centred on the posterior midline cortex (PMC). By contrast, episodic memory is widely believed to be supported by interactions between the hippocampus and these cortical regions. In this MRI study, we investigated the encoding and retrieval of lifelike events (video clips) in a patient with severe amnesia likely resulting from a stroke to the right thalamus, and a group of 20 age-matched controls. Structural MRI revealed grey matter reductions in left hippocampus and left thalamus in comparison to controls. We first characterised the regions activated in the controls while they watched and retrieved the videos. There were no differences in activation between the patient and controls in any of the regions. We then identified a widespread network of brain regions, including the hippocampus, that were functionally connected with the PMC in controls. However, in the patient there was a specific reduction in functional connectivity between the PMC and a region of left hippocampus when both watching and attempting to retrieve the videos. A follow up analysis revealed that in controls the functional connectivity between these regions when watching the videos was correlated with memory performance. Taken together, these findings support the view that the interactions between the PMC and the hippocampus enable the encoding and retrieval of multimodal representations of the contents of an event.

Download Full-text

Spatial inhibition of return is impaired in mild cognitive impairment and mild Alzheimer’s disease

10.1101/2020.05.11.089383 ◽

2020 ◽

Author(s):

Xiong Jiang ◽

James H. Howard ◽

G. Wiliam Rebeck ◽

R. Scott Turner

Keyword(s):

Alzheimer’S Disease ◽

Alzheimer's Disease ◽

Cognitive Impairment ◽

Mild Cognitive Impairment ◽

Medial Temporal Lobe ◽

Inhibition Of Return ◽

Memory Performance ◽

Brain Regions ◽

List Type ◽

Older Individuals

ABSTRACTSpatial inhibition of return (IOR) refers to the phenomenon by which individuals are slower to respond to stimuli appearing at a previously cued location compared to un-cued locations. Here we provide evidence supporting that spatial IOR is mildly impaired in individuals with mild cognitive impairment (MCI) or mild Alzheimer’s disease (AD), and the impairment is readily detectable using a novel double cue paradigm. Furthermore, reduced spatial IOR in high-risk healthy older individuals is associated with reduced memory and other neurocognitive task performance, suggesting that the novel double cue spatial IOR paradigm may be useful in detecting MCI and early AD.SIGNIFICANCE STATEMENTNovel double cue spatial inhibition of return (IOR) paradigm revealed a robust effect IOR deficits in individuals with mild cognitive impairment (MCI) or mild Alzheimer’s disease (AD)Spatial IOR effect correlates with memory performance in healthy older adults at a elevated risk of Alzheimer’s disease (with a family history or APOE e4 allele)The data suggests that double cue spatial IOR may be sensitive to detect early AD pathological changes, which may be linked to disease progress at the posterior brain regions (rather than the medial temporal lobe)

Download Full-text

Characterization of the hemodynamic response of the hippocampal and parahippocampal regions using FMRI

10.32920/ryerson.14644995.v1 ◽

2021 ◽

Author(s):

Priyanka Mehta

Keyword(s):

Spatial Memory ◽

Medial Temporal Lobe ◽

Dominant Role ◽

Memory Task ◽

Hemodynamic Response ◽

Hemodynamic Responses ◽

Memory Processing ◽

Time To Peak ◽

Hemodynamic Characteristics ◽

The Right

Previous neuroimaging studies have suggested a dominant role of the right medial temporal lobe (MTL) structures- the hippocampal and parahippocampal regions in spatial memory processing. However, the underlying physiological hemodynamic response functions (HRF) of the MTL substructures remain undefined. Given the neuroanatomical differences between these substructures, it is posited that their hemodynamic characteristics are distinct. In this study, the hemodynamic responses of the MTL substructures are investigated using an optimization algorithm that penalizes the curvature (i.e. second derivative) of HRF. The time-to-peak characteristic of the hemodynamic responses revealed that the right CA3 and DG subfields of the hippocampus are significantly more active than the right CA1 subfield during a specific spatial memory task. Further, the hemodynamic responses of the entorhinal, perirhinal and parahippocampal cortices are presented. Together, these findings may help advance our understanding of neurodegenerative diseases like epilepsy and Alzheimer’s disease that are strongly associated to hippocampal dysfunction.

Download Full-text