Higher-Order Conditioning in the Spatial Domain

Spatial learning and memory, the processes through which a wide range of living organisms encode, compute, and retrieve information from their environment to perform goal-directed navigation, has been systematically investigated since the early twentieth century to unravel behavioral and neural mechanisms of learning and memory. Early theories about learning to navigate space considered that animals learn through trial and error and develop responses to stimuli that guide them to a goal place. According to a trial-and error learning view, organisms can learn a sequence of motor actions that lead to a goal place, a strategy referred to as response learning, which contrasts with place learning where animals learn locations with respect to an allocentric framework. Place learning has been proposed to produce a mental representation of the environment and the cartesian relations between stimuli within it—which Tolman coined the cognitive map. We propose to revisit some of the best empirical evidence of spatial inference in animals, and then discuss recent attempts to account for spatial inferences within an associative framework as opposed to the traditional cognitive map framework. We will first show how higher-order conditioning can successfully account for inferential goal-directed navigation in a variety of situations and then how vectors derived from path integration can be integrated via higher-order conditioning, resulting in the generation of higher-order vectors that explain novel route taking. Finally, implications to cognitive map theories will be discussed.

Dissecting the genetic basis of learning, memory, and thermal tolerance in a multi-parental population of fruit flies

Learning, memory, and thermal tolerance are complex traits that are fundamental survival skills in many species. For example, small poikilotherms " like Drosophila melanogaster, have evolved physiological mechanisms to learn from and adapt to rapidly warming temperatures because their body temperature reflects that of their surroundings. Along with physiological mechanisms, these animals have also adapted cognitive practices, such as learning and memory, which help them escape extreme temperatures. Although scientific evidence shows that learning, memory, and thermal tolerance have been subjected to natural selection as a means of adapting to warmer temperatures, studies examining both traits (i.e., learning, and thermal tolerance) and how they relate genetically have not been done. Investigating the natural variation of the genes that control these traits is essential if we want to understand better how these traits arise within species. With recent advances in genomic technology, it is now feasible apart - down to a single gene - regions of the genome that influence complex phenotypes. For my Ph.D., I took a quantitative genetics and genomics approach to dissect the genetic basis of learning, memory, and thermal tolerance in Drosophila melanogaster. I hypothesized that there will be a shared and independent genetic locus controlling learning and memory and that there will be a trade-off between learning and thermal tolerance. To genetically dissect learning and memory performance, I used the Drosophila Synthetic Population Resource (DSPR), a multiparent mapping resource, consisting of a large set of recombinant inbred lines (RILs), which allows for high-resolution genome-wide scans, and the identification of loci contributing to naturally occurring genetic variation. Using a highly sensitive apparatus known as the "heat box", I trained flies with temperatures (24 [degrees] C and 41 [degrees] C, which is aversive) to learn to remain on one side of a chamber (place learning) and to remember thiory). An indl that spent most of the time on the side of the chamber associated with 24 [degrees] C will be considered to have high learning and memory. Immediately after, I assayed the flies for thermal tolerance, which is measured as the time to incapacitation at a constant high 41 [degrees] C temperature for 9.5 mins. I identified 16 different loci across the genome that significantly affect place learning and or memory performance, with 5 of these loci affecting both traits and eight loci that influence thermal tolerance. To identify transcriptomic differences associated with learning, memory, and thermal tolerance, I performed RNA-Sequencing (RNA-seq) on pooled samples of seven high-performing and seven low-performing RILs for all three traits and identified hundreds of genes with differences in expression. Integrating our transcriptomic results with the mapping results allowed us to identify nine promising candidate genes that control learning and twenty-one genes that fall between the most significant QTL that is possibly influenci thermal tolerance. I then used a few different methods to identify and validate genetic variants from the list of candidate genes within the most significant QTL (Q6) that influences thermal tolerance. First, I performed a structural variance analysis and found a few structural variants (SV; eg: inserts, deletions, or P-elements) located within candidate genes. Genes that contain SVs are known to influence phenotypic variation, and thus identifying any SV within candidate genes will provide a greater understanding of the genetic basis of a specific complex trait. Second, I functionally validated the candidate genes to identify causal genes using the UAS-GAL4/RNAi system. In this classical genetic system, a specific gene is disrupted through mutagenesis, which allows for the investigation of that gene on a specific phenotype. I then phenotyped lines to determine the phenotypic effect of each gene on thermal tolerance. We screened 20 UAS-RNAi lines (~1900 individuals) that were crossed with both pan neuronal and ubiquitously expressed Gal4 promoters and found 3 lines that were significant. Collectively, these results suggest that learning, memory, and thermal tolerance are highly polygenic and that multiple genes with both large and small effects influence these traits. Future work will aim to validate the function of candidate genes that influence learning and memory and identify the molecular pathways that the causal genes that influence thermal tolerance are associated with.

Bikes and Belonging: A Photographic Exploraiton of the Bike Host Program in Toronto

This major research project uses photography to explore questions of mobility, place-learning and belonging with newcomers participating in the Bike Host program in Toronto. Created by CultureLink Settlement Services in 2011, Bike Host loans bicycles out to immigrants and refugees and matches them with a cycling mentor. Through small group rides and large events, the participants have the opportunity to explore Toronto, gain confidence riding, make social connections, practice English and engage in volunteerism. For this project, a dozen participants also took pictures of how they were using their bicycles and shared their photos in small group, semi-structured discussions, which were recorded and analyzed. Four themes emerged: freedom, comfort and knowledge, discovery and belonging. The photographers found that compared to walking, they could travel further more quickly and with less effort, which prompted them to make more trips within their communities. The photographers also appreciated that, unlike with transit, they could leave whenever they wanted and take whichever route they wanted. This new mobility led to discovery, in both guided group rides to iconic Toronto destinations and in neighbourhood meanderings, undertaken independently along local streets and trails. Through this process, they filled in the gaps in their local cognitive maps. Increased familiarity led to an increased sense of belonging, as places that were once unfamiliar began to feel more like home.

Bikes and Belonging: A Photographic Exploraiton of the Bike Host Program in Toronto

This major research project uses photography to explore questions of mobility, place-learning and belonging with newcomers participating in the Bike Host program in Toronto. Created by CultureLink Settlement Services in 2011, Bike Host loans bicycles out to immigrants and refugees and matches them with a cycling mentor. Through small group rides and large events, the participants have the opportunity to explore Toronto, gain confidence riding, make social connections, practice English and engage in volunteerism. For this project, a dozen participants also took pictures of how they were using their bicycles and shared their photos in small group, semi-structured discussions, which were recorded and analyzed. Four themes emerged: freedom, comfort and knowledge, discovery and belonging. The photographers found that compared to walking, they could travel further more quickly and with less effort, which prompted them to make more trips within their communities. The photographers also appreciated that, unlike with transit, they could leave whenever they wanted and take whichever route they wanted. This new mobility led to discovery, in both guided group rides to iconic Toronto destinations and in neighbourhood meanderings, undertaken independently along local streets and trails. Through this process, they filled in the gaps in their local cognitive maps. Increased familiarity led to an increased sense of belonging, as places that were once unfamiliar began to feel more like home.

Place vs. Response Learning: History, Controversy, and Neurobiology

The present article provides a historical review of the place and response learning plus-maze tasks with a focus on the behavioral and neurobiological findings. The article begins by reviewing the conflict between Edward C. Tolman’s cognitive view and Clark L. Hull’s stimulus-response (S-R) view of learning and how the place and response learning plus-maze tasks were designed to resolve this debate. Cognitive learning theorists predicted that place learning would be acquired faster than response learning, indicating the dominance of cognitive learning, whereas S-R learning theorists predicted that response learning would be acquired faster, indicating the dominance of S-R learning. Here, the evidence is reviewed demonstrating that either place or response learning may be dominant in a given learning situation and that the relative dominance of place and response learning depends on various parametric factors (i.e., amount of training, visual aspects of the learning environment, emotional arousal, et cetera). Next, the neurobiology underlying place and response learning is reviewed, providing strong evidence for the existence of multiple memory systems in the mammalian brain. Research has indicated that place learning is principally mediated by the hippocampus, whereas response learning is mediated by the dorsolateral striatum. Other brain regions implicated in place and response learning are also discussed in this section, including the dorsomedial striatum, amygdala, and medial prefrontal cortex. An exhaustive review of the neurotransmitter systems underlying place and response learning is subsequently provided, indicating important roles for glutamate, dopamine, acetylcholine, cannabinoids, and estrogen. Closing remarks are made emphasizing the historical importance of the place and response learning tasks in resolving problems in learning theory, as well as for examining the behavioral and neurobiological mechanisms of multiple memory systems. How the place and response learning tasks may be employed in the future for examining extinction, neural circuits of memory, and human psychopathology is also briefly considered.

Individual differences in theta-band oscillations in a spatial memory network revealed by electroencephalography predict rapid place learning

Spatial memory has been closely related to the medial temporal lobe and theta oscillations are thought to play a key role. However, it remains difficult to investigate medial temporal lobe activation related to spatial memory with non-invasive electrophysiological methods in humans. Here, we combined the virtual delayed-matching-to-place task, reverse-translated from the watermaze delayed-matching-to-place task in rats, with high-density electroencephalography recordings. Healthy young volunteers performed this computerised task in a virtual circular arena, which contained a hidden target whose location moved to a new place every four trials, allowing the assessment of rapid memory formation. Using behavioural measures as predictor variables for source reconstructed frequency-specific electroencephalography power, we found that inter-individual differences in ‘search preference’ during ‘probe trials’, a measure of one-trial place learning known from rodent studies to be particularly hippocampus-dependent, correlated predominantly with distinct theta-band oscillations (approximately 7 Hz), particularly in the right temporal lobe, the right striatum and inferior occipital cortex or cerebellum. This pattern was found during both encoding and retrieval/expression, but not in control analyses and could not be explained by motor confounds. Alpha-activity in sensorimotor and parietal cortex contralateral to the hand used for navigation also correlated (inversely) with search preference. This latter finding likely reflects movement-related factors associated with task performance, as well as a frequency difference in (ongoing) alpha-rhythm for high-performers versus low-performers that may contribute to these results indirectly. Relating inter-individual differences in ongoing brain activity to behaviour in a continuous rapid place-learning task that is suitable for a variety of populations, we could demonstrate that memory-related theta-band activity in temporal lobe can be measured with electroencephalography recordings. This approach holds great potential for further studies investigating the interactions within this network during encoding and retrieval, as well as neuromodulatory impacts and age-related changes.

Reinforcement learning approaches to hippocampus-dependent flexible spatial navigation

Humans and non-human animals show great flexibility in spatial navigation, including the ability to return to specific locations based on as few as one single experience. To study spatial navigation in the laboratory, watermaze tasks, in which rats have to find a hidden platform in a pool of cloudy water surrounded by spatial cues, have long been used. Analogous tasks have been developed for human participants using virtual environments. Spatial learning in the watermaze is facilitated by the hippocampus. In particular, rapid, one-trial, allocentric place learning, as measured in the delayed-matching-to-place variant of the watermaze task, which requires rodents to learn repeatedly new locations in a familiar environment, is hippocampal dependent. In this article, we review some computational principles, embedded within a reinforcement learning framework, that utilise hippocampal spatial representations for navigation in watermaze tasks. We consider which key elements underlie their efficacy, and discuss their limitations in accounting for hippocampus-dependent navigation, both in terms of behavioural performance (i.e. how well do they reproduce behavioural measures of rapid place learning) and neurobiological realism (i.e. how well do they map to neurobiological substrates involved in rapid place learning). We discuss how an actor–critic architecture, enabling simultaneous assessment of the value of the current location and of the optimal direction to follow, can reproduce one-trial place learning performance as shown on watermaze and virtual delayed-matching-to-place tasks by rats and humans, respectively, if complemented with map-like place representations. The contribution of actor–critic mechanisms to delayed-matching-to-place performance is consistent with neurobiological findings implicating the striatum and hippocampo-striatal interaction in delayed-matching-to-place performance, given that the striatum has been associated with actor–critic mechanisms. Moreover, we illustrate that hierarchical computations embedded within an actor–critic architecture may help to account for aspects of flexible spatial navigation. The hierarchical reinforcement learning approach separates trajectory control via a temporal-difference error from goal selection via a goal prediction error and may account for flexible, trial-specific, navigation to familiar goal locations, as required in some arm-maze place memory tasks, although it does not capture one-trial learning of new goal locations, as observed in open field, including watermaze and virtual, delayed-matching-to-place tasks. Future models of one-shot learning of new goal locations, as observed on delayed-matching-to-place tasks, should incorporate hippocampal plasticity mechanisms that integrate new goal information with allocentric place representation, as such mechanisms are supported by substantial empirical evidence.

A general model of hippocampal and dorsal striatal learning and decision making

Humans and other animals use multiple strategies for making decisions. Reinforcement-learning theory distinguishes between stimulus–response (model-free; MF) learning and deliberative (model-based; MB) planning. The spatial-navigation literature presents a parallel dichotomy between navigation strategies. In “response learning,” associated with the dorsolateral striatum (DLS), decisions are anchored to an egocentric reference frame. In “place learning,” associated with the hippocampus, decisions are anchored to an allocentric reference frame. Emerging evidence suggests that the contribution of hippocampus to place learning may also underlie its contribution to MB learning by representing relational structure in a cognitive map. Here, we introduce a computational model in which hippocampus subserves place and MB learning by learning a “successor representation” of relational structure between states; DLS implements model-free response learning by learning associations between actions and egocentric representations of landmarks; and action values from either system are weighted by the reliability of its predictions. We show that this model reproduces a range of seemingly disparate behavioral findings in spatial and nonspatial decision tasks and explains the effects of lesions to DLS and hippocampus on these tasks. Furthermore, modeling place cells as driven by boundaries explains the observation that, unlike navigation guided by landmarks, navigation guided by boundaries is robust to “blocking” by prior state–reward associations due to learned associations between place cells. Our model, originally shaped by detailed constraints in the spatial literature, successfully characterizes the hippocampal–striatal system as a general system for decision making via adaptive combination of stimulus–response learning and the use of a cognitive map.