scholarly journals Effort reinforces learning

2021 ◽  
Author(s):  
Huw Jarvis ◽  
Isabelle Stevenson ◽  
Amy Q Huynh ◽  
Emily Babbage ◽  
James P. Coxon ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Learning Task ◽  
Physical Force ◽  
Computational Framework ◽  
Subjective Value ◽  
Effort Discounting ◽  
Reinforced Learning

Humans routinely learn the value of actions by assessing their outcomes. Actions also require effort, and increasing evidence suggests that effort and learning share common neurophysiological substrates. Here, we asked whether effort could modulate teaching signals in a reinforcement learning task. Individuals (N=140) registered their choices by exerting predefined levels of physical force. Our key finding was that effort increased the subjective value of an outcome, regardless of whether that outcome was positive or negative. Moreover, across participants, the extent to which effort reinforced learning correlated with effort discounting, suggesting that effort has a greater effect on learning in those who are more averse to investing it. By integrating models of reinforcement learning with neuroeconomic frameworks of value-based decision-making, we show that learning is shaped by both rewards and the effort required to obtain them, thus revealing how effort and learning operate within a common computational framework.

scholarly journals Corticoinsular circuits encode subjective value expectation and violation for effortful goal-directed behavior

2018 ◽  
Vol 115 (22) ◽  
pp. E5233-E5242 ◽  
Author(s):  
Amanda R. Arulpragasam ◽  
Jessica A. Cooper ◽  
Makiah R. Nuutinen ◽  
Michael T. Treadway
Keyword(s):  
Decision Making ◽  
Relevant Information ◽  
Neural Mechanisms ◽  
Anterior Cingulate ◽  
Error Signal ◽  
Value Prediction ◽  
Subjective Value ◽  
Effort Discounting ◽  
Trial History ◽  
Goal Directed Behavior

We are presented with choices each day about how to invest our effort to achieve our goals. Critically, these decisions must frequently be made under conditions of incomplete information, where either the effort required or possible reward to be gained is uncertain. Such choices therefore require the development of potential value estimates to guide effortful goal-directed behavior. To date, however, the neural mechanisms for this expectation process are unknown. Here, we used computational fMRI during an effort-based decision-making task where trial-wise information about effort costs and reward magnitudes was presented separately over time, thereby allowing us to model distinct effort/reward computations as choice-relevant information unfolded. We found that ventromedial prefrontal cortex (vmPFC) encoded expected subjective value. Further, activity in dorsal anterior cingulate (dACC) and anterior insula (aI) reflected both effort discounting as well as a subjective value prediction error signal derived from trial history. While prior studies have identified these regions as being involved in effort-based decision making, these data demonstrate their specific role in the formation and maintenance of subjective value estimates as relevant information becomes available.

scholarly journals Effects of Depressive Symptoms, Feelings, and Interoception on Reward-Based Decision-Making: Investigation Using Reinforcement Learning Model

2020 ◽  
Vol 10 (8) ◽  
pp. 508
Author(s):  
Hiroyoshi Ogishima ◽  
Shunta Maeda ◽  
Yuki Tanaka ◽  
Hironori Shimada
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Depressive Symptoms ◽  
Emotional Experience ◽  
Learning Task ◽  
Experimental Manipulation ◽  
Detection Task ◽  
Control Group ◽  
Before And After ◽  
Experimental Group

Background: In this study, we examined the relationships between reward-based decision-making in terms of learning rate, memory rate, exploration rate, and depression-related subjective emotional experience, in terms of interoception and feelings, to understand how reward-based decision-making is impaired in depression. Methods: In all, 52 university students were randomly assigned to an experimental group and a control group. To manipulate interoception, the participants in the experimental group were instructed to tune their internal somatic sense to the skin-conductance-response waveform presented on a display. The participants in the control group were only instructed to stay relaxed. Before and after the manipulation, the participants completed a probabilistic reversal-learning task to assess reward-based decision-making using reinforcement learning modeling. Similarly, participants completed a probe-detection task, a heartbeat-detection task, and self-rated scales. Results: The experimental manipulation of interoception was not successful. In the baseline testing, reinforcement learning modeling indicated a marginally-significant correlation between the exploration rate and depressive symptoms. However, the exploration rate was significantly associated with lower interoceptive attention and higher depressive feeling. Conclusions: The findings suggest that situational characteristics may be closely involved in reward exploration and highlight the clinically-meaningful possibility that intervention for affective processes may impact reward-based decision-making in those with depression.

scholarly journals Trading off the cost of conflict against expected rewards

2018 ◽  
Author(s):  
Nura Sidarus ◽  
Stefano Palminteri ◽  
Valérian Chambon
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Cognitive Control ◽  
Reversal Learning ◽  
Computational Models ◽  
Learning Task ◽  
Learning Rates ◽  
Different Types ◽  
Irrelevant Distractors ◽  
The Cost

AbstractValue-based decision-making involves trading off the cost associated with an action against its expected reward. Research has shown that both physical and mental effort constitute such subjective costs, biasing choices away from effortful actions, and discounting the value of obtained rewards. Facing conflicts between competing action alternatives is considered aversive, as recruiting cognitive control to overcome conflict is effortful. Yet, it remains unclear whether conflict is also perceived as a cost in value-based decisions. The present study investigated this question by embedding irrelevant distractors (flanker arrows) within a reversal-learning task, with intermixed free and instructed trials. Results showed that participants learned to adapt their choices to maximize rewards, but were nevertheless biased to follow the suggestions of irrelevant distractors. Thus, the perceived cost of being in conflict with an external suggestion could sometimes trump internal value representations. By adapting computational models of reinforcement learning, we assessed the influence of conflict at both the decision and learning stages. Modelling the decision showed that conflict was avoided when evidence for either action alternative was weak, demonstrating that the cost of conflict was traded off against expected rewards. During the learning phase, we found that learning rates were reduced in instructed, relative to free, choices. Learning rates were further reduced by conflict between an instruction and subjective action values, whereas learning was not robustly influenced by conflict between one’s actions and external distractors. Our results show that the subjective cost of conflict factors into value-based decision-making, and highlights that different types of conflict may have different effects on learning about action outcomes.

Planning Complexity Registers as a Cost in Metacontrol

2018 ◽  
Vol 30 (10) ◽  
pp. 1391-1404 ◽  
Author(s):  
Wouter Kool ◽  
Samuel J. Gershman ◽  
Fiery A. Cushman
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Formal Analysis ◽  
Learning Task ◽  
Costs And Benefits ◽  
Model Free ◽  
Versus Model ◽  
Specific Manner ◽  
Decision Making Processes ◽  
Planning Approaches

Decision-making algorithms face a basic tradeoff between accuracy and effort (i.e., computational demands). It is widely agreed that humans can choose between multiple decision-making processes that embody different solutions to this tradeoff: Some are computationally cheap but inaccurate, whereas others are computationally expensive but accurate. Recent progress in understanding this tradeoff has been catalyzed by formalizing it in terms of model-free (i.e., habitual) versus model-based (i.e., planning) approaches to reinforcement learning. Intuitively, if two tasks offer the same rewards for accuracy but one of them is much more demanding, we might expect people to rely on habit more in the difficult task: Devoting significant computation to achieve slight marginal accuracy gains would not be “worth it.” We test and verify this prediction in a sequential reinforcement learning task. Because our paradigm is amenable to formal analysis, it contributes to the development of a computational model of how people balance the costs and benefits of different decision-making processes in a task-specific manner; in other words, how we decide when hard thinking is worth it.

scholarly journals Influenca: a gamified assessment of value-based decision-making for longitudinal studies

2021 ◽  
Author(s):  
Monja P. Neuser ◽  
Franziska Kräutlein ◽  
Anne Kühnel ◽  
Vanessa Teckentrup ◽  
Jennifer Svaldi ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Learning Task ◽  
Reward Learning ◽  
Initial Validation ◽  
Cross Platform ◽  
Test Retest Reliability ◽  
Ecological Momentary ◽  
Momentary Assessment ◽  
Effective Medication

AbstractReinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. Here, we developed an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) for repeated assessment over weeks. In this task, players have to identify the most effective medication by selecting the best option after integrating offered points with changing probabilities (according to random Gaussian walks). Participants can complete up to 31 levels with 150 trials each. To encourage replay on their preferred device, in-game screens provide feedback on the progress. Using an initial validation sample of 127 players (2904 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show low to medium intra-class correlations (ICC: 0.22-0.52), indicating substantial within- and between-subject variance. Notably, state items showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.

scholarly journals Neural Signatures of Prediction Errors in a Decision-Making Task Are Modulated by Action Execution Failures

2018 ◽  
Author(s):  
Samuel D. McDougle ◽  
Peter A. Butcher ◽  
Darius Parvin ◽  
Fasial Mushtaq ◽  
Yael Niv ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Prediction Error ◽  
Learning Task ◽  
Prediction Errors ◽  
Action Execution ◽  
Model Driven ◽  
Assignment Of Credit ◽  
Level Information ◽  
High Level

AbstractDecisions must be implemented through actions, and actions are prone to error. As such, when an expected outcome is not obtained, an individual should not only be sensitive to whether the choice itself was suboptimal, but also whether the action required to indicate that choice was executed successfully. The intelligent assignment of credit to action execution versus action selection has clear ecological utility for the learner. To explore this scenario, we used a modified version of a classic reinforcement learning task in which feedback indicated if negative prediction errors were, or were not, associated with execution errors. Using fMRI, we asked if prediction error computations in the human striatum, a key substrate in reinforcement learning and decision making, are modulated when a failure in action execution results in the negative outcome. Participants were more tolerant of non-rewarded outcomes when these resulted from execution errors versus when execution was successful but the reward was withheld. Consistent with this behavior, a model-driven analysis of neural activity revealed an attenuation of the signal associated with negative reward prediction error in the striatum following execution failures. These results converge with other lines of evidence suggesting that prediction errors in the mesostriatal dopamine system integrate high-level information during the evaluation of instantaneous reward outcomes.

scholarly journals Pupil responses as indicators of value-based decision-making

2018 ◽  
Author(s):  
Joanne C. Van Slooten ◽  
Sara Jahfari ◽  
Tomas Knapen ◽  
Jan Theeuwes
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Computational Model ◽  
Cognitive Processes ◽  
Learning Task ◽  
Brain Regions ◽  
Computational Approach ◽  
Prediction Errors ◽  
Reward Prediction ◽  
Exciting Possibility

AbstractPupil responses have been used to track cognitive processes during decision-making. Studies have shown that in these cases the pupil reflects the joint activation of many cortical and subcortical brain regions, also those traditionally implicated in value-based learning. However, how the pupil tracks value-based decisions and reinforcement learning is unknown. We combined a reinforcement learning task with a computational model to study pupil responses during value-based decisions, and decision evaluations. We found that the pupil closely tracks reinforcement learning both across trials and participants. Prior to choice, the pupil dilated as a function of trial-by-trial fluctuations in value beliefs. After feedback, early dilation scaled with value uncertainty, whereas later constriction scaled with reward prediction errors. Our computational approach systematically implicates the pupil in value-based decisions, and the subsequent processing of violated value beliefs, ttese dissociable influences provide an exciting possibility to non-invasively study ongoing reinforcement learning in the pupil.

scholarly journals How the level of reward awareness changes the computational and electrophysiological signatures of reinforcement learning

2018 ◽  
Author(s):  
C.M.C. Correa ◽  
S. Noorman ◽  
J. Jiang ◽  
S. Palminteri ◽  
M.X Cohen ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Present Report ◽  
Subjective Evaluation ◽  
Computational Modelling ◽  
Learning Task ◽  
Reward Processing ◽  
Neural Activities ◽  
Electrophysiological Recordings ◽  
Neural Computations

AbstractThe extent to which subjective awareness influences reward processing, and thereby affects future decisions is currently largely unknown. In the present report, we investigated this question in a reinforcement-learning framework, combining perceptual masking, computational modeling and electroencephalographic recordings (human male and female participants). Our results indicate that degrading the visibility of the reward decreased -without completely obliterating- the ability of participants to learn from outcomes, but concurrently increased their tendency to repeat previous choices. We dissociated electrophysiological signatures evoked by the reward-based learning processes from those elicited by the reward-independent repetition of previous choices and showed that these neural activities were significantly modulated by reward visibility. Overall, this report sheds new light on the neural computations underlying reward-based learning and decision-making and highlights that awareness is beneficial for the trial-by-trial adjustment of decision-making strategies.Significance statementThe notion of reward is strongly associated with subjective evaluation, related to conscious processes such as “pleasure”, “liking” and “wanting”. Here we show that degrading reward visibility in a reinforcement learning task decreases -without completely obliterating- the ability of participants to learn from outcomes, but concurrently increases subjects tendency to repeat previous choices. Electrophysiological recordings, in combination with computational modelling, show that neural activities were significantly modulated by reward visibility. Overall, we dissociate different neural computations underlying reward-based learning and decision-making, which highlights a beneficial role of reward awareness in adjusting decision-making strategies.

scholarly journals Sources of confidence in value-based choice

2021 ◽  
Author(s):  
Rafael Polania
Keyword(s):  
Decision Making ◽  
Cognitive Effort ◽  
Metacognitive Awareness ◽  
Decision Quality ◽  
Comparison Process ◽  
Immediate Feedback ◽  
Computational Framework ◽  
Decision Behavior ◽  
Economic Choice ◽  
Subjective Value

"Which meal would you like, chicken or pasta? Chicken please. ...hmmm not sure. No sorry, I prefer pasta". Confidence, the subjective estimate of decision quality, is an essential component of decision making. It is necessary for learning from mistakes in the absence of immediate feedback and guiding future actions. Despite its importance, it remains unclear where confidence judgments originate from, especially for decisions that rely on individual subjective values and preferences. Here, we devised a behavioural paradigm and a computational framework that allowed us to formally tease apart the sources of confidence in value-based decisions. In line with canonical decision theories, we found that trial-to-trial fluctuations in the precision of value encoding impact economic choice consistency. Surprisingly, however, and contrary to canonical theories of confidence, this uncertainty has no influence on confidence reports. Instead, we find that confidence reflects the degree of balance and cognitive effort with which the choice alternatives have been compared. Specifically, we show that confidence emerges from endogenous attentional effort towards choice alternatives and down-stream noise in the comparison process. These findings caution a direct translation of canonical frameworks of confidence based on perceptual decision behavior into the value-based choice domain. In addition our computational framework provides an explanation for confidence miss-attributions in economic behaviour and reveals the mechanistic interplay of endogenous attentional states and subjective value for guiding decisions and metacognitive awareness.

scholarly journals Model-based hierarchical reinforcement learning and human action control

2014 ◽  
Vol 369 (1655) ◽  
pp. 20130480 ◽  
Author(s):  
Matthew Botvinick ◽  
Ari Weinstein
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Hierarchical Model ◽  
Human Action ◽  
Action Control ◽  
Computational Framework ◽  
Hierarchical Reinforcement Learning ◽  
Model Based ◽  
Human Decision ◽  
Areas Of Interest

Recent work has reawakened interest in goal-directed or ‘model-based’ choice, where decisions are based on prospective evaluation of potential action outcomes. Concurrently, there has been growing attention to the role of hierarchy in decision-making and action control. We focus here on the intersection between these two areas of interest, considering the topic of hierarchical model-based control. To characterize this form of action control, we draw on the computational framework of hierarchical reinforcement learning, using this to interpret recent empirical findings. The resulting picture reveals how hierarchical model-based mechanisms might play a special and pivotal role in human decision-making, dramatically extending the scope and complexity of human behaviour.

Sign in / Sign up

Export Citation Format

Share Document