Higher-Order Conditioning in the Spatial Domain

Spatial learning and memory, the processes through which a wide range of living organisms encode, compute, and retrieve information from their environment to perform goal-directed navigation, has been systematically investigated since the early twentieth century to unravel behavioral and neural mechanisms of learning and memory. Early theories about learning to navigate space considered that animals learn through trial and error and develop responses to stimuli that guide them to a goal place. According to a trial-and error learning view, organisms can learn a sequence of motor actions that lead to a goal place, a strategy referred to as response learning, which contrasts with place learning where animals learn locations with respect to an allocentric framework. Place learning has been proposed to produce a mental representation of the environment and the cartesian relations between stimuli within it—which Tolman coined the cognitive map. We propose to revisit some of the best empirical evidence of spatial inference in animals, and then discuss recent attempts to account for spatial inferences within an associative framework as opposed to the traditional cognitive map framework. We will first show how higher-order conditioning can successfully account for inferential goal-directed navigation in a variety of situations and then how vectors derived from path integration can be integrated via higher-order conditioning, resulting in the generation of higher-order vectors that explain novel route taking. Finally, implications to cognitive map theories will be discussed.

Higher-Order Conditioning With Simultaneous and Backward Conditioned Stimulus: Implications for Models of Pavlovian Conditioning

In a new environment, humans and animals can detect and learn that cues predict meaningful outcomes, and use this information to adapt their responses. This process is termed Pavlovian conditioning. Pavlovian conditioning is also observed for stimuli that predict outcome-associated cues; a second type of conditioning is termed higher-order Pavlovian conditioning. In this review, we will focus on higher-order conditioning studies with simultaneous and backward conditioned stimuli. We will examine how the results from these experiments pose a challenge to models of Pavlovian conditioning like the Temporal Difference (TD) models, in which learning is mainly driven by reward prediction errors. Contrasting with this view, the results suggest that humans and animals can form complex representations of the (temporal) structure of the task, and use this information to guide behavior, which seems consistent with model-based reinforcement learning. Future investigations involving these procedures could result in important new insights on the mechanisms that underlie Pavlovian conditioning.

Higher-Order Conditioning and Dopamine: Charting a Path Forward

Higher-order conditioning involves learning causal links between multiple events, which then allows one to make novel inferences. For example, observing a correlation between two events (e.g., a neighbor wearing a particular sports jersey), later helps one make new predictions based on this knowledge (e.g., the neighbor’s wife’s favorite sports team). This type of learning is important because it allows one to benefit maximally from previous experiences and perform adaptively in complex environments where many things are ambiguous or uncertain. Two procedures in the lab are often used to probe this kind of learning, second-order conditioning (SOC) and sensory preconditioning (SPC). In second-order conditioning (SOC), we first teach subjects that there is a relationship between a stimulus and an outcome (e.g., a tone that predicts food). Then, an additional stimulus is taught to precede the predictive stimulus (e.g., a light leads to the food-predictive tone). In sensory preconditioning (SPC), this order of training is reversed. Specifically, the two neutral stimuli (i.e., light and tone) are first paired together and then the tone is paired separately with food. Interestingly, in both SPC and SOC, humans, rodents, and even insects, and other invertebrates will later predict that both the light and tone are likely to lead to food, even though they only experienced the tone directly paired with food. While these processes are procedurally similar, a wealth of research suggests they are associatively and neurobiologically distinct. However, midbrain dopamine, a neurotransmitter long thought to facilitate basic Pavlovian conditioning in a relatively simplistic manner, appears critical for both SOC and SPC. These findings suggest dopamine may contribute to learning in ways that transcend differences in associative and neurological structure. We discuss how research demonstrating that dopamine is critical to both SOC and SPC places it at the center of more complex forms of cognition (e.g., spatial navigation and causal reasoning). Further, we suggest that these more sophisticated learning procedures, coupled with recent advances in recording and manipulating dopamine neurons, represent a new path forward in understanding dopamine’s contribution to learning and cognition.

Higher-Order Conditioning: What Is Learnt and How it Is Expressed

Pairing a neutral conditioned stimulus (CS) with a motivationally significant unconditioned stimulus (US) results in the CS coming to elicit conditioned responses (CRs). The widespread significance and translational value of Pavlovian conditioning are increased by the fact that pairing two neutral CSs (A and X) enables conditioning with X to affect behavior to A. There are two traditional informal accounts of such higher-order conditioning, which build on more formal associative analyses of Pavlovian conditioning. But, higher-order conditioning and Pavlovian conditioning have characteristics that are beyond these accounts: Notably, the two are influenced in different ways by the same experimental manipulations, and both generate conditioned responses that do not reflect the US per se. Here, we present a formal analysis that sought to address these characteristics.

Second-Order Conditioning in Humans

In contrast to the large body of work demonstrating second-order conditioning (SOC) in non-human animals, the evidence for SOC in humans is scant. In this review, I examine the existing literature and suggest theoretical and procedural explanations for why SOC has been so elusive in humans. In particular, I discuss potential interactions with conditioned inhibition, whether SOC is rational, and propose critical parameters needed to obtain the effect. I conclude that SOC is a real but difficult phenomenon to obtain in humans, and suggest directions for future research.

Cortical Contributions to Higher-Order Conditioning: A Review of Retrosplenial Cortex Function

In higher-order conditioning paradigms, such as sensory preconditioning or second-order conditioning, discrete (e.g., phasic) or contextual (e.g., static) stimuli can gain the ability to elicit learned responses despite never being directly paired with reinforcement. The purpose of this mini-review is to examine the neuroanatomical basis of high-order conditioning, by selectively reviewing research that has examined the role of the retrosplenial cortex (RSC) in sensory preconditioning and second-order conditioning. For both forms of higher-order conditioning, we first discuss the types of associations that may occur and then review findings from RSC lesion/inactivation experiments. These experiments demonstrate a role for the RSC in sensory preconditioning, suggesting that this cortical region might contribute to higher-order conditioning via the encoding of neutral stimulus-stimulus associations. In addition, we address knowledge gaps, avenues for future research, and consider the contribution of the RSC to higher-order conditioning in relation to related brain structures.

Second-Order Conditioning and Conditioned Inhibition in Different Moments of the Same Training: The Effect of A+ and AX− Trial Number

The feature negative discrimination (A+/AX−) can result in X gaining excitatory properties (second-order conditioning, SOC) or in X gaining inhibitory properties (conditioned inhibition, CI), a challenging finding for most current associative learning theories. Research on the variables that modulate which of these phenomena would occur is scarce but has clearly identified the trial number as an important variable. In the set of experiments presented here, the effect of trial number was assessed in a magazine training task with rats as a function of both the conditioning sessions and the number of A+ and AX− trials per session, holding constant the total number of trials per session. The results indicated that SOC is most likely to be found at the beginning of training when there are many A+ and few AX− trials, and CI (as assessed by a retardation test) is most likely to be found at the end of training when there are few A+ and many AX− trials. Both phenomena were also found at different moments of training when the number of A+ trials was equal to the number of AX− trials. These results cannot be predicted by acquisition-focused associative models but can be predicted by theories that distinguish between learning and performance.

Causal evidence supporting the proposal that dopamine transients function as a temporal difference prediction error

Reward-evoked dopamine is well-established as a prediction error. However the central tenet of temporal difference accounts – that similar transients evoked by reward-predictive cues also function as errors – remains untested. To address this, we used two phenomena, second-order conditioning and blocking, in order to examine the role of dopamine in prediction error versus reward prediction. We show that optogenetically-shunting dopamine activity at the start of a reward-predicting cue prevents second-order conditioning without affecting blocking. These results support temporal difference accounts by providing causal evidence that cue-evoked dopamine transients function as prediction errors.