Coherent theta activity in the medial and orbital frontal cortices encodes reward value

This study examined how the medial frontal (MFC) and orbital frontal (OFC) cortices process reward information. We simultaneously recorded local field potentials in the two areas as rats consumed liquid sucrose rewards. Both areas exhibited a 4–8 Hz ‘theta’ rhythm that was phase-locked to the lick cycle. The rhythm tracked shifts in sucrose concentrations and fluid volumes, demonstrating that it is sensitive to differences in reward magnitude. The coupling between the rhythm and licking was stronger in MFC than OFC and varied with response vigor and absolute reward value in the MFC. Spectral analysis revealed zero-lag coherence between the cortical areas, and found evidence for a directionality of the rhythm, with MFC leading OFC. Our findings suggest that consummatory behavior generates simultaneous theta range activity in the MFC and OFC that encodes the value of consumed fluids, with the MFC having a top-down role in the control of consumption.

Deliberative Decision-Making in Macaques Removes Reward-Driven Response Vigor

Most of our daily decisions are governed by one of two systems: an impulsive system driving instantaneous decisions and a deliberative system driving thoughtful ones. The impulsive system reacts to immediately available concrete rewards. In contrast, the deliberative system reacts to more delayed rewards and/or punishments, which imposes consideration of longer-term choice consequences. Contingency management for addiction treatment is hypothesized to engage deliberative processes. Ultimately, in both decision-making situations, an action is needed to enact the decision. Whether those actions differ in implementation is an open question whose answer could inform as to whether distinct neural systems are engaged. To explore whether there is evidence of separate mechanisms between deliberated and immediate choices, we trained monkeys to perform a decision-making task where they made a choice on a touch screen between two visual cues predicting different amounts of reward. In immediate choice (IC) trials, the cues appeared at the final response locations where subjects could immediately touch the chosen cue. In deliberated choice (DC) trials, compound cues appeared orthogonally to the response locations. After a delay, allowing for decision formation, an identifying cue component was displaced to the randomly assigned response locations, permitting subjects to reach for the chosen cue. Both trial types showed an effect of cue value on cue selection time. However, only IC trials showed an effect of the competing cue on response vigor (measured by movement duration) and a reach trajectory that deviated in the direction of the competing cue, suggesting a decision reexamination process. Reward modulation of response vigor implicates dopaminergic mechanisms. In DC trials, reach trajectories revealed a commitment to the chosen choice target, and reach vigor was not modulated by the value of the competing cue. Our results suggest that choice–action dynamics are shaped by competing offers only during instantaneous, impulsive choice. After a deliberated decision, choice–action dynamics are unaffected by the alternative offer cue, demonstrating a commitment to the choice. The potential relevance to contingency management is discussed.

Rhythmic activity in the medial and orbital frontal cortices tracks reward value and the vigor of consummatory behavior

ABSTRACTThis study examined how the medial frontal (MFC) and orbital frontal (OFC) cortices process reward information to guide behavior. We simultaneously recorded local field potentials in the two areas as rats consumed liquid sucrose rewards and examined how the areas collectively process reward information. Both areas exhibited a 4-8 Hz “theta” rhythm that was phase locked to the lick cycle. The rhythm similarly tracked shifts in sucrose concentrations and fluid volumes, suggesting that it is sensitive to general differences in reward magnitude. Differences between the MFC and OFC were noted, specifically that the rhythm varied with response vigor and absolute reward value in the MFC, but not the OFC. Our findings suggest that the MFC and OFC concurrently process reward information but have distinct roles in the control of consummatory behavior.

Proximity and Expectancy Modulate Response Vigor After Reward Omission

Both humans and some non-human animals tend to respond more vigorously after failing to obtain rewards. Such response invigoration becomes more pronounced when individuals have increased expectations of obtaining rewards during reward pursuit (expectancy), and when they perceive the eventual loss to be proximal to reward receipt (proximity). However, it was unclear whether proximity and expectancy may have distinct influences on response vigor. To investigate this question, we developed a computerized ’scratch card’ task, in which participants turned three cards one by one and won points when all three cards matched (AAA). After each game, they pressed keys to confirm the outcome and start a new game. We included three types of losses: AAB, where participants had increased expectancy of winning as the game evolved, and the final outcome was proximal to winning; ABB and ABA, with reduced expectancy, but high proximity to winning; and ABC, with reduced expectancy and low proximity to winning. In three online studies, we consistently observed that participants confirmed losses more quickly than wins. Importantly, detailed analyses of the different types of losses revealed that proximity reduced vigor, whereas expectancy increased it. Together, these findings are in line with general appraisal theories: the adjustments of response vigor may be triggered by the appraised discrepancy between the current state and a reference state (e.g., attaining one’s goal), and serve to close the gap and facilitate goal pursuit. These findings may also have implications for the effect of ‘near miss’ on gambling persistence. Further exploring how reward omission impacts response vigor may help us better understand the goal pursuit process, and how it becomes maladaptive under certain circumstances.

Proximity and Expectancy Modulate Response Vigor After Reward Omission

Both humans and some non-human animals tend to respond more vigorously after failing to obtain rewards. Such response invigoration becomes more pronounced when individuals have increased expectations of obtaining rewards during reward pursuit (expectancy), and when they perceive the eventual loss to be proximal to reward receipt (proximity). However, it was unclear whether proximity and expectancy may have distinct influences on response vigor. To investigate this question, we developed a computerized ’scratch card’ task, in which participants turned three cards one by one and won points when all three cards matched (AAA). After each game, they pressed keys to confirm the outcome and start a new game. We included three types of losses: AAB, where participants had increased expectancy of winning as the game evolved, and the final outcome was proximal to winning; ABB and ABA, with reduced expectancy, but high proximity to winning; and ABC, with reduced expectancy and low proximity to winning. In three online studies, we consistently observed that participants confirmed losses more quickly than wins. Importantly, detailed analyses of the different types of losses revealed that proximity reduced vigor, whereas expectancy increased it. Together, these findings are in line with general appraisal theories: the adjustments of response vigor may be triggered by the appraised discrepancy between the current state and a reference state (e.g., attaining one’s goal), and serve to close the gap and facilitate goal pursuit. These findings may also have implications for the effect of ‘near miss’ on gambling persistence. Further exploring how reward omission impacts response vigor may help us better understand the goal pursuit process, and how it becomes maladaptive under certain circumstances.

SUSTAINED DOPAMINERGIC PLATEAUS AND NORADRENERGIC DEPRESSIONS MEDIATE DISSOCIABLE ASPECTS OF EXPLOITATIVE STATES

ABSTRACTWe are constantly faced with the trade-off between exploiting actions with known outcomes and exploring alternative actions whose outcomes may be better. This balance has been hypothesized to rely on dopaminergic neurons of the substantia nigra pars compacta (SNc)1 and noradrenergic neurons of the locus coeruleus (LC)2–3. We developed a behavioral paradigm to capture exploitative and exploratory states, and imaged calcium dynamics in genetically-identified dopaminergic SNc neurons and noradrenergic LC neurons during state transitions. During exploitative states, characterized by motivated repetition of the same action choice, we found dichotomous changes in baseline activity in SNc and LC, with SNc showing higher and LC showing lower sustained activity. These sustained neural states emerged from the accumulation of lengthened positive responses and hysteretic dynamics in SNc networks, and lengthened negative responses in LC. Sustained activity could not be explained by classical reinforcement learning parameters, and in SNc but not LC, emerged in subpopulations coding for response vigor. Manipulating the sustained activity of SNc and LC revealed that dopaminergic activity primarily mediates engagement and motivation, whereas noradrenergic activity modulates action selection. These data uncover the emergence of sustained neural states in dopaminergic and noradrenergic networks that mediate dissociable aspects of exploitative bouts.

Ventral pallidal encoding of reward-seeking behavior depends on the underlying associative structure

Despite its being historically conceptualized as a motor expression site, emerging evidence suggests the ventral pallidum (VP) plays a more active role in integrating information to generate motivation. Here, we investigated whether rat VP cue responses would encode and contribute similarly to the vigor of reward-seeking behaviors trained under Pavlovian versus instrumental contingencies, when these behavioral responses consist of superficially similar locomotor response patterns but may reflect distinct underlying decision-making processes. We find that cue-elicited activity in many VP neurons predicts the latency of instrumental reward seeking, but not of Pavlovian response latency. Further, disruption of VP signaling increases the latency of instrumental but not Pavlovian reward seeking. This suggests that VP encoding of and contributions to response vigor are specific to the ability of incentive cues to invigorate reward-seeking behaviors upon which reward delivery is contingent.

Optimal Response Vigor and Choice Under Non-stationary Outcome Values

Within a rational framework a decision-maker selects actions based on the reward-maximisation principle which stipulates they acquire outcomes with the highest values at the lowest cost. Action selection can be divided into two dimensions: selecting an action from several alternatives, and choosing its vigor, i.e., how fast the selected action should be executed. Both of these dimensions depend on the values of the outcomes, and these values are often affected as more outcomes are consumed, and so are the actions. Despite this, previous works have addressed the computational substrates of optimal actions only in the specific condition that the values of outcomes are constant, and it is still unknown what the optimal actions are when the values of outcomes are non-stationary. Here, based on an optimal control framework, we derive a computational model for optimal actions under non-stationary outcome values. The results imply that even when the values of outcomes are changing, the optimal response rate is constant rather than decreasing. This finding shows that, in contrast to previous theories, the commonly observed changes in the actions cannot be purely attributed to the changes in the outcome values. We then prove that this observation can be explained based on the uncertainty about temporal horizons; e.g., in the case of experimental protocols, the session duration. We further show that when multiple outcomes are available, the model explains probability matching as well as maximisation choice strategies. The model provides, therefore, a quantitative analysis of optimal actions and explicit predictions for future testing.