Dendritic integration of sensory and reward information facilitates learning

Learning goal-directed behaviours requires integration of separate information streams representing context, relevant stimuli and reward. Dendrites of pyramidal neurons are suitable sites for such integration, but it remains elusive how their responses adapt when an animal learns a new task. Here, we identify two distinct classes of dendritic responses that represent either contextual/sensory information or reward information and that differ in their task- and learning-related dynamics. Using longitudinal calcium imaging of apical dendritic tufts of L5 pyramidal neurons in mouse barrel cortex, we tracked dendritic activity across learning and analyzed both local dendritic branch signals and global apical tuft activity. During texture discrimination learning, sensory representations (including contextual and touch information) strengthened and converged on the reward-predicting tactile stimulus when mice became experts. In contrast, reward-associated responses were particularly strong in the naive condition and became less pronounced upon learning. When we blocked the representation of unexpected reward in naive animals with optogenetic inhibition, animals failed to learn until we released the block and learning proceeded normally. Our results suggest that reward signals in dendrites are essential for adjusting neuronal integration of converging inputs to facilitate adaptive behaviour.

Effect of reward on electrophysiological signatures of grid cell population activity in human spatial navigation

The regular equilateral triangular periodic firing pattern of grid cells in the entorhinal cortex is considered a regular metric for the spatial world, and the grid-like representation correlates with hexadirectional modulation of theta (4–8 Hz) power in the entorhinal cortex relative to the moving direction. However, researchers have not clearly determined whether grid cells provide only simple spatial measures in human behavior-related navigation strategies or include other factors such as goal rewards to encode information in multiple patterns. By analysing the hexadirectional modulation of EEG signals in the theta band in the entorhinal cortex of patients with epilepsy performing spatial target navigation tasks, we found that this modulation presents a grid pattern that carries target-related reward information. This grid-like representation is influenced by explicit goals and is related to the local characteristics of the environment. This study provides evidence that human grid cell population activity is influenced by reward information at the level of neural oscillations.

Reward Rapidly Enhances Visual Perception

Rewards exert a deep influence on our cognition and behavior. Here, we used a paradigm in which reward information was provided at either encoding or retrieval of a brief, masked stimulus to show that reward can also rapidly modulate perceptual encoding of visual information. Experiment 1 ( n = 30 adults) showed that participants’ response accuracy was enhanced when a to-be-encoded grating signaled high reward relative to low reward, but only when the grating was presented very briefly and participants reported that they were not consciously aware of it. Experiment 2 ( n = 29 adults) showed that there was no difference in participants’ response accuracy when reward information was instead provided at the stage of retrieval, ruling out an explanation of the reward-modulation effect in terms of differences in motivated retrieval. Taken together, our findings provide behavioral evidence consistent with a rapid reward modulation of visual perception, which may not require consciousness.

Rostral Anterior Cingulate Activations Inversely Relate To Reward Payoff Maximation & Predict Depressed Mood

Choice selection strategies and decision making are typically investigated using multiple-choice gambling paradigms that require participants to maximize reward payoff. However, research shows that performance in such paradigms suffers from individual biases towards the frequency of gains to choose smaller local gains over larger longer term gain, also referred to as melioration. Here, we developed a simple two-choice reward task, implemented in 186 healthy human adult subjects across the adult lifespan to understand the behavioral, computational, and neural bases of payoff maximization versus melioration. The observed reward choice behavior on this task was best explained by a reinforcement learning model of differential future reward prediction. Simultaneously recorded and source-localized electroencephalography (EEG) showed that diminished theta-band activations in the right rostral anterior cingulate cortex (rACC) correspond to greater reward payoff maximization, specifically during the presentation of cumulative reward information at the end of each task trial. Notably, these activations (greater rACC theta) predicted self-reported depressed mood symptoms, thereby showcasing a reward processing marker of potential clinical utility.

Selective outcome reinstatement during evaluation drives heuristics in risky choice

A ubiquitous feature of human decision making under risk is that individuals differ from each other, as well as from normativity, in how they incorporate reward and probability information. One possible explanation for these deviations is a desire to reduce the number of potential outcomes considered during choice evaluation. Although multiple behavioral models can be invoked involving selective consideration of choice outcomes, whether differences in these tendencies underlie behavioral differences in sensitivity to reward and probability information is unknown. Here we consider neural evidence where we exploit magnetoencephalography (MEG) to decode the actual choice outcomes participants consider when they decide between a gamble and a safe outcome. We show that variability in tendencies of individual participants to reinstate neural outcome representations, based on either their probability or reward, explains variability in the extent to which their choices reflect consideration of probability and reward information. In keeping with this we also show that participants who are higher in behavioral impulsivity fail to preferentially reinstate outcomes with higher probability. Our results suggest that neural differences in the degree to which outcomes are considered shape risk taking strategy, both in decision making tasks, as well as in real life.

Exploring reward-related attention selectivity deficits in Parkinson’s disease

An important aspect of managing a limited cognitive resource like attention is to use the reward value of stimuli to prioritize the allocation of attention to higher-value over lower-value stimuli. Recent evidence suggests this depends on dopaminergic signaling of reward. In Parkinson’s disease, both reward sensitivity and attention are impaired, but whether these deficits are directly related to one another is unknown. We tested whether Parkinson’s patients use reward information when automatically allocating their attention and whether this is modulated by dopamine replacement. We compared patients, tested both ON and OFF dopamine replacement medication, to older controls using a standard attention capture task. First, participants learned the different reward values of stimuli. Then, these reward-associated stimuli were used as distractors in a visual search task. We found that patients were generally distracted by the presence of the distractors but that the degree of distraction caused by the high-value and low-value distractors was similar. Furthermore, we found no evidence to support the possibility that dopamine replacement modulates the effect of reward on automatic attention allocation. Our results suggest a possible inability in Parkinson’s patients to use the reward value of stimuli when automatically allocating their attention, and raise the possibility that reward-driven allocation of resources may affect the adaptive modulation of other cognitive processes.

Coherent theta activity in the medial and orbital frontal cortices encodes reward value

This study examined how the medial frontal (MFC) and orbital frontal (OFC) cortices process reward information. We simultaneously recorded local field potentials in the two areas as rats consumed liquid sucrose rewards. Both areas exhibited a 4–8 Hz ‘theta’ rhythm that was phase-locked to the lick cycle. The rhythm tracked shifts in sucrose concentrations and fluid volumes, demonstrating that it is sensitive to differences in reward magnitude. The coupling between the rhythm and licking was stronger in MFC than OFC and varied with response vigor and absolute reward value in the MFC. Spectral analysis revealed zero-lag coherence between the cortical areas, and found evidence for a directionality of the rhythm, with MFC leading OFC. Our findings suggest that consummatory behavior generates simultaneous theta range activity in the MFC and OFC that encodes the value of consumed fluids, with the MFC having a top-down role in the control of consumption.

Selective attention is insensitive to reward and to dopamine in Parkinson’s disease

Patients with Parkinson’s disease exhibit reduced reward sensitivity in addition to early cognitive deficits, among which attention impairments are common. Attention allocation is controlled at multiple levels and recent work has shown that reward, in addition to its role in the top-down goal-directed control of attention, also guides the automatic allocation of attention resources, a process thought to rely on striatal dopamine. Whether Parkinson’s patients, due to their striatal dopamine loss, suffer from an inability to use reward information to guide the allocation of their attention is unknown. To address this question, we tested Parkinson’s patients (n=43) ON and OFF their dopaminergic medication, and compared them to a group of older controls (n=31). We used a standard two-phase attention capture task in which subjects were first implicitly trained to make colour-reward associations. In the second phase, the previously reward-associated colours were used as distractors in a visual search task. We found that patients did not use reward information to modulate their attention; they were similarly distracted by the presence of low and high-reward distractors. However, contrary to our predictions, we did not find evidence that dopamine modulated this inability to use reward to guide attention allocation. Additionally, we found slightly increased overall distractibility in Parkinson’s patients compared to older controls, but interestingly, the degree of distractibility was not influenced by dopamine replacement. Our results suggest that loss of reward-guided attention allocation may contribute to early attention deficits and raise the possibility that this inability to prioritize cognitive resource allocation could contribute to executive deficits more broadly in Parkinson’s disease.

Integrating reward information for prospective behaviour

Decision-making is often studied in a static context, such as deciding which option to select from those currently available. However, in everyday life we often also need to decide when to select an option to maximise reward. One possibility is that people track the latent reward of an option, updating expected changes in its value over time, to achieve appropriate selection timing. Contrary to this hypothesis, our electroencephalographic pattern analyses revealed that option properties like starting value and growth rate were translated into an estimate of when an option would become most valuable, far in advance of selecting it. The option’s latent reward could not be decoded independently from neural activity. These results suggest that decisions to exploit individual options with lawful reward trajectories can be made by transforming reward information into an estimate of optimal timing, rather than actively monitoring an option’s changing reward prospect.

The dynamics of explore–exploit decisions reveal a signal-to-noise mechanism for random exploration

Growing evidence suggests that behavioral variability plays a critical role in how humans manage the tradeoff between exploration and exploitation. In these decisions a little variability can help us to overcome the desire to exploit known rewards by encouraging us to randomly explore something else. Here we investigate how such ‘random exploration’ could be controlled using a drift-diffusion model of the explore–exploit choice. In this model, variability is controlled by either the signal-to-noise ratio with which reward is encoded (the ‘drift rate’), or the amount of information required before a decision is made (the ‘threshold’). By fitting this model to behavior, we find that while, statistically, both drift and threshold change when people randomly explore, numerically, the change in drift rate has by far the largest effect. This suggests that random exploration is primarily driven by changes in the signal-to-noise ratio with which reward information is represented in the brain.