Reliable population utility signal from diverse economic value coding in orbitofrontal neurons

Economic value encapsulates the subjective combination of reward magnitude and probability. We investigated the mechanism for subjective value computation in single neurons using an economic axiomatic approach. We found that single neurons in the macaque orbitofrontal cortex, known to be sensitive to reward magnitude and probability, encode the economic value functions (utility and probability weighting) in a heterogeneous manner, such that the activity of individual neurons did not match the animal's choices. However, the utility and probability weighting code from a population of these varied neurons reliably matched the animals' choices and risk attitudes. Thus, the neuronal population code for economic value amounted to a distributional representation of the formal economic functions. With a diverse single-unit economic value code converging into a reliable population-level utility code, this scheme suggests a brain mechanism for the flexible accommodation of multiple choice patterns and risk attitudes.

Influence of Rule and Reward-based Strategies on Inferences of Serial Order by Monkeys

Knowledge of transitive relationships between items can contribute to learning the order of a set of stimuli from pairwise comparisons. However, cognitive mechanisms of transitive inferences based on rank order remain unclear, as are contributions of reward magnitude and rule-based inference. To explore these issues, we created a conflict between rule- and reward-based learning during a serial ordering task. Rhesus macaques learned two lists, each containing five stimuli, that were trained exclusively with adjacent pairs. Selection of the higher-ranked item resulted in rewards. "Small reward" lists yielded 2 drops of fluid reward, while "large reward" lists yielded 5 drops. Following training of adjacent pairs, monkeys were tested on novels pairs. One item was selected from each list, such that a ranking rule could conflict with preferences for large rewards. Differences in associated reward magnitude had a strong influence on accuracy, but we also observed a symbolic distance effect. That provided evidence of a rule-based influence on decisions. Reaction time comparisons suggested a conflict between rule and reward-based processes. We conclude that performance reflects the contributions of two strategies, and that a model-based strategy is employed in the face of a strong countervailing reward incentive.

The role of ventromedial prefrontal cortex in reward valuation and future thinking during intertemporal choice

Intertemporal choices require trade-offs between short-term and long-term outcomes. Ventromedial prefrontal cortex (vmPFC) damage causes steep discounting of future rewards (delay discounting [DD]) and impoverished episodic future thinking (EFT). The role of vmPFC in reward valuation, EFT, and their interaction during intertemporal choice is still unclear. Here, 12 patients with lesions to vmPFC and 41 healthy controls chose between smaller-immediate and larger-delayed hypothetical monetary rewards while we manipulated reward magnitude and the availability of EFT cues. In the EFT condition, participants imagined personal events to occur at the delays associated with the larger-delayed rewards. We found that DD was steeper in vmPFC patients compared to controls, and not modulated by reward magnitude. However, EFT cues downregulated DD in vmPFC patients as well as controls. These findings indicate that vmPFC integrity is critical for the valuation of (future) rewards, but not to instill EFT in intertemporal choice.

Cue-motivated reward seeking is negatively regulated by expected reward magnitude in Pavlovian-instrumental transfer

The Pavlovian-instrumental transfer (PIT) paradigm is widely used to assay the motivational influence of reward-paired cues, which is reflected by their ability to stimulate instrumental reward-seeking behavior. Leading models of incentive learning assume that motivational value is assigned to cues based on the total amount of reward they signal (i.e., their state value). Based on recent findings, we lay out the alternative hypothesis that cue-elicited reward predictions may actually suppress the motivation to seek out new rewards through instrumental behavior in order to facilitate efficient retrieval of a reward that is already expected, before it is lost or stolen. According to this view, cue-motivated reward seeking should be inversely related to the magnitude of an expected reward, since there is more to lose by failing to secure a large reward than a small reward. We investigated the influence of expected reward magnitude on PIT expression. Hungry rats were initially trained to lever press for food pellets before undergoing Pavlovian conditioning, in which two distinct auditory cues signaled food pellet delivery at cue offset. Reward magnitude was varied across cues and groups. While all groups had at least one cue that signaled three food pellets, the alternate cue signaled either one (Group 1/3), three (Group 3/3), or nine food pellets (Group 3/9). PIT testing revealed that the motivational influence of reward-predictive cues on lever pressing varied inversely with expected reward magnitude, with the 1-pellet cue augmenting performance and the 3- and 9-pellet cues suppressing performance, particularly near the expected time of reward delivery. This pattern was mirrored by opposing changes in the food-port entry behavior, which varied positively with expected reward magnitude. We discuss how these findings may relate to cognitive control over cue-motivated behavior.

The role of ventromedial prefrontal cortex in reward valuation and future thinking during intertemporal choice

Intertemporal choices require trade-offs between short-term and long-term outcomes. Ventromedial prefrontal cortex (vmPFC) damage causes steep discounting of future rewards (delay discounting; DD) and impoverished episodic future thinking (EFT). The role of vmPFC in reward valuation, EFT, and their interaction during intertemporal choice is still unclear. Here, twelve patients with lesions to vmPFC and forty-one healthy controls chose between smallerimmediate and larger-delayed rewards while we manipulated reward magnitude and the availability of EFT cues. In the EFT condition, participants imagined personal events to occur at the delays associated with the larger-delayed rewards. We found that DD was steeper in vmPFC patients compared to controls, and not modulated by reward magnitude. However, EFT cues downregulated DD in vmPFC patients as well as controls. These findings indicate that vmPFC integrity is critical for the valuation of (future) rewards, but not to instill EFT in intertemporal choice.

When is it Worth Working for Water? A Utility Maximization Theory

In nature, amount of work an animal must do to obtain a resource like water depends on conditions in the environment. Conditions change, so it would behoove animals to allocate effort flexibly such that they work enough, but not more than necessary. To study this, we maintained rats in an environment where all water was earned in a task. We varied the reward magnitude and measured voluntary effort and water consumption. The rats did more trials per day when the reward per trial was smaller, yet worked for more water per day when rewards were larger. We propose an analytic model based on utility maximization which can account for these behavioral observations. The model is fit with per-day total work and consumption, but provides insight into the timing of trials and implicates lamina terminalis as a candidate neural substrate.

Lapses in perceptual decisions reflect exploration

Perceptual decision-makers often display a constant rate of errors independent of evidence strength. These 'lapses' are treated as a nuisance arising from noise tangential to the decision, e.g. inattention or motor errors. Here, we use a multisensory decision task in rats to demonstrate that these explanations cannot account for lapses' stimulus dependence. We propose a novel explanation: lapses reflect a strategic trade-off between exploiting known rewarding actions and exploring uncertain ones. We tested this model's predictions by selectively manipulating one action's reward magnitude or probability. As uniquely predicted by this model, changes were restricted to lapses associated with that action. Finally, we show that lapses are a powerful tool for assigning decision-related computations to neural structures based on disruption experiments (here, posterior striatum and secondary motor cortex). These results suggest that lapses reflect an integral component of decision-making and are informative about action values in normal and disrupted brain states.