scholarly journals Dopamine Enhances Model-Free Credit Assignment Through Boosting of Retrospective Model-Based Inference

2021 ◽  
Vol 89 (9) ◽  
pp. S94
Author(s):  
Lorenz Deserno ◽  
Rani Moran ◽  
Ying Lee ◽  
Jochen Michely ◽  
Peter Dayan ◽  
...  
Keyword(s):  
Credit Assignment ◽  
Model Based ◽  
Model Free

scholarly journals Credit assignment to state-independent task representations and its relationship with model-based decision making

2019 ◽  
Vol 116 (32) ◽  
pp. 15871-15876 ◽  
Author(s):  
Nitzan Shahar ◽  
Rani Moran ◽  
Tobias U. Hauser ◽  
Rogier A. Kievit ◽  
Daniel McNamee ◽  
...  
Keyword(s):  
Decision Task ◽  
Credit Assignment ◽  
Free System ◽  
Assigned Value ◽  
Model Based ◽  
Independent Task ◽  
Model Free ◽  
Motor Representations ◽  
The Impact ◽  
Task Representations

Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment’s structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.

scholarly journals Retrospective model-based inference guides model-free credit assignment

2019 ◽  
Vol 10 (1) ◽  
Author(s):  
Rani Moran ◽  
Mehdi Keramati ◽  
Peter Dayan ◽  
Raymond J. Dolan
Keyword(s):  
Credit Assignment ◽  
Model Based ◽  
Model Free

scholarly journals Dopamine enhances model-free credit assignment through boosting of retrospective model-based inference

eLife ◽  
2021 ◽  
Vol 10 ◽  
Author(s):  
Lorenz Deserno ◽  
Rani Moran ◽  
Jochen Michely ◽  
Ying Lee ◽  
Peter Dayan ◽  
...  
Keyword(s):  
Drug Effect ◽  
Prediction Errors ◽  
Credit Assignment ◽  
Double Blind ◽  
Cooperative Interactions ◽  
Model Based ◽  
Model Free ◽  
Reward Prediction ◽  
Dependent Change ◽  
Within Subjects

Dopamine is implicated in representing model-free (MF) reward prediction errors a as well as influencing model-based (MB) credit assignment and choice. Putative cooperative interactions between MB and MF systems include a guidance of MF credit assignment by MB inference. Here, we used a double-blind, placebo-controlled, within-subjects design to test an hypothesis that enhancing dopamine levels boosts the guidance of MF credit assignment by MB inference. In line with this, we found that levodopa enhanced guidance of MF credit assignment by MB inference, without impacting MF and MB influences directly. This drug effect correlated negatively with a dopamine-dependent change in purely MB credit assignment, possibly reflecting a trade-off between these two MB components of behavioural control. Our findings of a dopamine boost in MB inference guidance of MF learning highlights a novel DA influence on MB-MF cooperative interactions.

scholarly journals Dopamine enhances model-free credit assignment through boosting of retrospective model-based inference

2021 ◽  
Author(s):  
Lorenz Deserno ◽  
Rani Moran ◽  
Jochen Michely ◽  
Ying Lee ◽  
Peter Dayan ◽  
...  
Keyword(s):  
Working Memory ◽  
Drug Effect ◽  
Prediction Errors ◽  
Credit Assignment ◽  
Double Blind ◽  
Cooperative Interactions ◽  
Model Based ◽  
Model Free ◽  
Dependent Change ◽  
Within Subjects

AbstractDopamine is implicated in signalling model-free (MF) reward prediction errors and various aspects of model-based (MB) credit assignment and choice. Recently, we showed that cooperative interactions between MB and MF systems include guidance of MF credit assignment by MB inference. Here, we used a double-blind, placebo-controlled, within-subjects design to test the hypothesis that enhancing dopamine levels, using levodopa, boosts the guidance of MF credit assignment by MB inference. We found that levodopa enhanced retrospective guidance of MF credit assignment by MB inference, without impacting on MF and MB influences per se. This drug effect positively correlated with working memory, but only in a context where reward needed to be recalled for MF credit assignment. The dopaminergic enhancement in MB-MF interactions correlated negatively with a dopamine-dependent change in MB credit assignment, possibly reflecting a potential trade-off between these two components of behavioural control. Thus, our findings demonstrate that dopamine boosts MB inference during guidance of MF learning, supported in part by working memory, but trading-off with a dopaminergic enhancement of MB credit assignment. The findings highlight a novel role for a DA influence on MB-MF interactions.

Representation, abstraction, and simple-minded sophisticates

2020 ◽  
Vol 43 ◽  
Author(s):  
Peter Dayan
Keyword(s):  
Decision Theory ◽  
Predictive Coding ◽  
Bayesian Decision Theory ◽  
Bayesian Decision ◽  
Model Based ◽  
Model Free

Abstract Bayesian decision theory provides a simple formal elucidation of some of the ways that representation and representational abstraction are involved with, and exploit, both prediction and its rather distant cousin, predictive coding. Both model-free and model-based methods are involved.

Model-Based and Model-Free Social Cognition

2019 ◽  
Author(s):  
Leor M Hackel ◽  
Jeffrey Jordan Berg ◽  
Björn Lindström ◽  
David Amodio
Keyword(s):  
Reinforcement Learning ◽  
Social Cognition ◽  
Learning Strategies ◽  
Memory Systems ◽  
Learning Task ◽  
Financial Advisors ◽  
Model Based ◽  
Model Free ◽  
Systems Model ◽  
Task Assessment

Do habits play a role in our social impressions? To investigate the contribution of habits to the formation of social attitudes, we examined the roles of model-free and model-based reinforcement learning in social interactions—computations linked in past work to habit and planning, respectively. Participants in this study learned about novel individuals in a sequential reinforcement learning paradigm, choosing financial advisors who led them to high- or low-paying stocks. Results indicated that participants relied on both model-based and model-free learning, such that each independently predicted choice during the learning task and self-reported liking in a post-task assessment. Specifically, participants liked advisors who could provide large future rewards as well as advisors who had provided them with large rewards in the past. Moreover, participants varied in their use of model-based and model-free learning strategies, and this individual difference influenced the way in which learning related to self-reported attitudes: among participants who relied more on model-free learning, model-free social learning related more to post-task attitudes. We discuss implications for attitudes, trait impressions, and social behavior, as well as the role of habits in a memory systems model of social cognition.

scholarly journals Author Correction: Reliance on model-based and model-free control in obesity

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Lieneke K. Janssen ◽  
Florian P. Mahner ◽  
Florian Schlagenhauf ◽  
Lorenz Deserno ◽  
Annette Horstmann
Keyword(s):  
Model Based ◽  
Model Free ◽  
Model Free Control

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Model-Based and Model-Free Control of DC-DC Converters with High-Order Dynamics and Limited Measurements

2020 ◽  
pp. 1-1
Author(s):  
Javier Loranca ◽  
Jonathan Carlos Mayo Maldonado ◽  
Gerardo Escobar ◽  
Carlos Villarreal-Hernandez ◽  
Thabiso Maupong ◽  
...  
Keyword(s):  
High Order ◽  
Model Based ◽  
Model Free ◽  
Model Free Control
Sign in / Sign up

Export Citation Format

Share Document