scholarly journals Hunger improves reinforcement-driven but not planned action

2021 ◽  
Author(s):  
Maaike M.H. van Swieten ◽  
Rafal Bogacz ◽  
Sanjay G. Manohar
Keyword(s):  
Decision Making ◽  
Learning Systems ◽  
Sequential Decision Making ◽  
Sequential Decision ◽  
Context Sensitive ◽  
Decision Systems ◽  
Model Based ◽  
Model Free ◽  
Model Free Control ◽  
Overall Performance

AbstractHuman decisions can be reflexive or planned, being governed respectively by model-free and model-based learning systems. These two systems might differ in their responsiveness to our needs. Hunger drives us to specifically seek food rewards, but here we ask whether it might have more general effects on these two decision systems. On one hand, the model-based system is often considered flexible and context-sensitive, and might therefore be modulated by metabolic needs. On the other hand, the model-free system’s primitive reinforcement mechanisms may have closer ties to biological drives. Here, we tested participants on a well-established two-stage sequential decision-making task that dissociates the contribution of model-based and model-free control. Hunger enhanced overall performance by increasing model-free control, without affecting model-based control. These results demonstrate a generalized effect of hunger on decision-making that enhances reliance on primitive reinforcement learning, which in some situations translates into adaptive benefits.

scholarly journals Hunger improves reinforcement-driven but not planned action

2021 ◽  
Author(s):  
Maaike M.H. van Swieten ◽  
Rafal Bogacz ◽  
Sanjay G. Manohar
Keyword(s):  
Decision Making ◽  
Sequential Decision ◽  
Context Sensitive ◽  
Decision Systems ◽  
Model Based ◽  
Model Free ◽  
Prevalence Of Obesity ◽  
Model Free Control ◽  
Decision Mode

AbstractHuman decisions can be reflexive or planned, being governed respectively by model-free and model-based learning systems. These two systems might differ in their responsiveness to our needs. Hunger drives us to specifically seek food rewards, but here we ask whether it might have more general effects on these two decision systems. On one hand, the model-based system is often considered flexible and context-sensitive, and might therefore be modulated by metabolic needs. On the other hand, the model-free system’s primitive reinforcement mechanisms may have closer ties to biological drives. Here, we tested participants on a well-established two-stage sequential decision-making task that dissociates the contribution of model-based and model-free control. Hunger enhanced overall performance by increasing model-free control, without affecting model-based control. These results demonstrate a generalised effect of hunger on decision-making that enhances reliance on primitive reinforcement learning, which in some situations translates into adaptive benefits.Significance statementThe prevalence of obesity and eating disorder is steadily increasing. To counteract problems related to eating, people need to make rational decisions. However, appetite may switch us to a different decision mode, making it harder to achieve long-term goals. Here we show that planned and reinforcement-driven actions are differentially sensitive to hunger. Hunger specifically affected reinforcement-driven actions, and did not affect the planning of actions. Our data shows that people behave differently when they are hungry. We also provide a computational model of how the behavioural changes might arise.

scholarly journals Subjective optimality in finite sequential decision-making

2021 ◽  
Vol 17 (12) ◽  
pp. e1009633
Author(s):  
Yeonju Sin ◽  
HeeYoung Seon ◽  
Yun Kyoung Shin ◽  
Oh-Sang Kwon ◽  
Dongil Chung
Keyword(s):  
Decision Making ◽  
Time Window ◽  
Programming Model ◽  
Secretary Problem ◽  
Optimal Decision ◽  
Sequential Decision Making ◽  
Sequential Decision ◽  
Model Based ◽  
Subjective Valuation ◽  
Optimal Decision Making

Many decisions in life are sequential and constrained by a time window. Although mathematically derived optimal solutions exist, it has been reported that humans often deviate from making optimal choices. Here, we used a secretary problem, a classic example of finite sequential decision-making, and investigated the mechanisms underlying individuals’ suboptimal choices. Across three independent experiments, we found that a dynamic programming model comprising subjective value function explains individuals’ deviations from optimality and predicts the choice behaviors under fewer and more opportunities. We further identified that pupil dilation reflected the levels of decision difficulty and subsequent choices to accept or reject the stimulus at each opportunity. The value sensitivity, a model-based estimate that characterizes each individual’s subjective valuation, correlated with the extent to which individuals’ physiological responses tracked stimuli information. Our results provide model-based and physiological evidence for subjective valuation in finite sequential decision-making, rediscovering human suboptimality in subjectively optimal decision-making processes.

scholarly journals Higher Meta-cognitive Ability Predicts Less Reliance on Over Confident Habitual Learning System

2019 ◽  
Author(s):  
Sara Ershadmanesh ◽  
Mostafa Miandari ◽  
Abdol-hossein Vahabie ◽  
Majid Nili Ahmadabadi
Keyword(s):  
Decision Making ◽  
Cognitive Ability ◽  
Learning Strategies ◽  
Learning Style ◽  
Learning Systems ◽  
Learning System ◽  
Variable Environment ◽  
Free System ◽  
Model Based ◽  
Model Free

AbstractMany studies on human and animals have provided evidence for the contribution of goal-directed and habitual valuation systems in learning and decision-making. These two systems can be modeled using model-based (MB) and model-free (MF) algorithms in Reinforcement Learning (RL) framework. Here, we study the link between the contribution of these two learning systems to behavior and meta-cognitive capabilities. Using computational modeling we showed that in a highly variable environment, where both learning strategies have chance level performances, model-free learning predicts higher confidence in decisions compared to model-based strategy. Our experimental results showed that the subjects’ meta-cognitive ability is negatively correlated with the contribution of model-free system to their behavior while having no correlation with the contribution of model-based system. Over-confidence of the model-free system justifies this counter-intuitive result. This is a new explanation for individual difference in learning style.

scholarly journals Novelty is not Surprise: Exploration and learning in human sequential decision-making

2020 ◽  
Author(s):  
He A. Xu ◽  
Alireza Modirshanechi ◽  
Marco P. Lehmann ◽  
Wulfram Gerstner ◽  
Michael H. Herzog
Keyword(s):  
Decision Making ◽  
Prediction Error ◽  
Event Related Potential ◽  
Sequential Decision Making ◽  
Sequential Decision ◽  
Model Free ◽  
Reward Prediction ◽  
Rate Of Learning ◽  
Human Participants ◽  
Exploration Exploitation

AbstractDrivers of reinforcement learning (RL), beyond reward, are controversially debated. Novelty and surprise are often used equivocally in this debate. Here, using a deep sequential decision-making paradigm, we show that reward, novelty, and surprise play different roles in human RL. Surprise controls the rate of learning, whereas novelty and the novelty prediction error (NPE) drive exploration. Exploitation is dominated by model-free (habitual) action choices. A theory that takes these separate effects into account predicts on average 73 percent of the action choices of human participants after the first encounter of a reward and allows us to dissociate surprise and novelty in the EEG signal. While the event-related potential (ERP) at around 300ms is positively correlated with surprise, novelty, NPE, reward, and the reward prediction error, the ERP response to novelty and NPE starts earlier than that to surprise.

scholarly journals Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

2021 ◽  
Vol 17 (6) ◽  
pp. e1009070
Author(s):  
He A. Xu ◽  
Alireza Modirshanechi ◽  
Marco P. Lehmann ◽  
Wulfram Gerstner ◽  
Michael H. Herzog
Keyword(s):  
Decision Making ◽  
Environmental Changes ◽  
Free Action ◽  
Human Action ◽  
Sequential Decision Making ◽  
Sequential Decision ◽  
World Model ◽  
Model Free ◽  
The World ◽  
Human Participants

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.

scholarly journals Metacontrol of decision-making strategies in human aging

2019 ◽  
Author(s):  
Florian Bolenz ◽  
Wouter Kool ◽  
Andrea M.F. Reiter ◽  
Ben Eppinger
Keyword(s):  
Older Adults ◽  
Decision Making ◽  
Cognitive Abilities ◽  
Reduced Model ◽  
Task Structure ◽  
Costs And Benefits ◽  
Sequential Decision ◽  
Model Based Control ◽  
Model Based ◽  
Model Free

When making decisions, humans employ different strategies which are commonly formalized as model-free and model-based reinforcement learning. While previous research has reported reduced model-based control with aging, it remains unclear whether this is due to limited cognitive capacities or a reduced willingness to engage in an effortful strategy. Moreover, it is not clear how aging affects the metacontrol of decision making, i.e. the dynamic adaptation of decision-making strategies to varying situational demands. To this end, we tested younger and older adults in a sequential decision-making task that dissociates model-free and model-based control. In contrast to previous research, in this study we applied a task in which model-based control led to higher payoffs in terms of monetary reward. Moreover, we manipulated the costs and benefits associated with model-based control by varying reward magnitude as well as the stability of the task structure. Compared to younger adults, older adults showed reduced reliance on model-based decision making and less adaptation of decision-making strategies to varying costs and benefits of model-based control. Our findings suggest that aging affects the dynamic metacontrol of decision-making strategies and that reduced model-based control in older adults is due to limited cognitive abilities to represent the structure of the task.

scholarly journals Metacontrol of decision-making strategies in human aging

eLife ◽  
2019 ◽  
Vol 8 ◽  
Author(s):  
Florian Bolenz ◽  
Wouter Kool ◽  
Andrea MF Reiter ◽  
Ben Eppinger
Keyword(s):  
Older Adults ◽  
Decision Making ◽  
Cognitive Abilities ◽  
Cross Sectional Study ◽  
Reduced Model ◽  
Task Structure ◽  
Sequential Decision ◽  
Cross Sectional ◽  
Model Based ◽  
Model Free

Humans employ different strategies when making decisions. Previous research has reported reduced reliance on model-based strategies with aging, but it remains unclear whether this is due to cognitive or motivational factors. Moreover, it is not clear how aging affects the metacontrol of decision making, that is the dynamic adaptation of decision-making strategies to varying situational demands. In this cross-sectional study, we tested younger and older adults in a sequential decision-making task that dissociates model-free and model-based strategies. In contrast to previous research, model-based strategies led to higher payoffs. Moreover, we manipulated the costs and benefits of model-based strategies by varying reward magnitude and the stability of the task structure. Compared to younger adults, older adults showed reduced model-based decision making and less adaptation of decision-making strategies. Our findings suggest that aging affects the metacontrol of decision-making strategies and that reduced model-based strategies in older adults are due to limited cognitive abilities.

scholarly journals Individual differences in dopamine function underlying the balance between model-based and model-free control

2019 ◽  
Author(s):  
Ying Lee ◽  
Lorenz Deserno ◽  
Nils B. Kroemer ◽  
Shakoor Pooseh ◽  
Liane Oehme ◽  
...  
Keyword(s):  
Individual Differences ◽  
Control Individual ◽  
General Notion ◽  
Sequential Decision ◽  
Model Based ◽  
Model Free ◽  
Positron Emission ◽  
Model Free Control ◽  
Induced Changes ◽  
The Relationship

AbstractReinforcement learning involves a balance between model-free (MF) and model-based (MB) systems. Recent studies suggest that individuals with either pharmacologically enhanced levels of dopamine (DA) or higher baseline levels of DA exhibit more MB control. However, it remains unknown whether such pharmacological effects depend on baseline DA.Here, we investigated whether effects of L-DOPA on the balance of MB/MF control depend on ventral striatal baseline DA. Sixty participants had two functional magnetic resonance imaging (fMRI) scans while performing a two-stage sequential decision-making task under 150 mg L-DOPA or placebo (counterbalanced), followed by a 4-hour 18F-DOPA positron emission tomography (PET) scan (on a separate occasion).We found an interaction between baseline DA levels and L-DOPA induced changes in MB control. Individuals with higher baseline DA levels showed a greater L-DOPA induced enhancement in MB control. Surprisingly, we found a corresponding drug-by-baseline DA interaction on MF, but not MB learning signals in the ventromedial prefrontal cortex. We did not find a significant interaction between baseline DA levels and L-DOPA effects on MF control or MB/MF balance.In sum, our findings point to a baseline dependency of L-DOPA effects on differential aspects of MB and MF control. Individual differences in DA washout may be an important moderator of L-DOPA effects. Overall, our findings complement the general notion where higher DA levels is related to a greater reliance on MB control. Although the relationship between phasic DA firing and MF learning is conventionally assumed in the animal literature, the relationship between DA and MF control is not as straightforward and requires further clarification.

Sign in / Sign up

Export Citation Format

Share Document