scholarly journals Model-based foraging using latent-cause inference

2021 ◽  
Author(s):  
Nora Harhen ◽  
Catherine A. Hartley ◽  
Aaron Bornstein
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Structure Learning ◽  
Learning Ability ◽  
Decision Strategy ◽  
Model Based ◽  
Statistical Regularities ◽  
Spatio Temporal ◽  
Human Foraging ◽  
Foraging Task

Foraging has been suggested to provide a more ecologicallyvalidcontext for studying decision-making. However, the environmentsused in human foraging tasks fail to capture thestructure of real world environments which contain multiplelevels of spatio-temporal regularities. We ask if foragers detectthese statistical regularities and use them to construct amodel of the environment that guides their patch-leaving decisions.We propose a model of how foragers might accomplishthis, and test its predictions in a foraging task with a structuredenvironment that includes patches of varying quality andpredictable transitions. Here, we show that human foragingdecisions reflect ongoing, statistically-optimal structure learning.Participants modulated decisions based on the current andpotential future context. From model fits to behavior, we canidentify an individual’s structure learning ability and relate itto decision strategy. These findings demonstrate the utility ofleveraging model-based reinforcement learning to understandforaging behavior.

scholarly journals Revealing Neurocomputational Mechanisms of Reinforcement Learning and Decision-Making With the hBayesDM Package

2017 ◽  
Vol 1 ◽  
pp. 24-57 ◽  
Author(s):  
Woo-Young Ahn ◽  
Nathaniel Haines ◽  
Lei Zhang
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Computational Modeling ◽  
Bayesian Modeling ◽  
Hierarchical Bayesian ◽  
Hierarchical Bayesian Modeling ◽  
Model Based ◽  
Modeling Approaches ◽  
Wide Range ◽  
User Friendly

Reinforcement learning and decision-making (RLDM) provide a quantitative framework and computational theories with which we can disentangle psychiatric conditions into the basic dimensions of neurocognitive functioning. RLDM offer a novel approach to assessing and potentially diagnosing psychiatric patients, and there is growing enthusiasm for both RLDM and computational psychiatry among clinical researchers. Such a framework can also provide insights into the brain substrates of particular RLDM processes, as exemplified by model-based analysis of data from functional magnetic resonance imaging (fMRI) or electroencephalography (EEG). However, researchers often find the approach too technical and have difficulty adopting it for their research. Thus, a critical need remains to develop a user-friendly tool for the wide dissemination of computational psychiatric methods. We introduce an R package called hBayesDM (hierarchical Bayesian modeling of Decision-Making tasks), which offers computational modeling of an array of RLDM tasks and social exchange games. The hBayesDM package offers state-of-the-art hierarchical Bayesian modeling, in which both individual and group parameters (i.e., posterior distributions) are estimated simultaneously in a mutually constraining fashion. At the same time, the package is extremely user-friendly: users can perform computational modeling, output visualization, and Bayesian model comparisons, each with a single line of coding. Users can also extract the trial-by-trial latent variables (e.g., prediction errors) required for model-based fMRI/EEG. With the hBayesDM package, we anticipate that anyone with minimal knowledge of programming can take advantage of cutting-edge computational-modeling approaches to investigate the underlying processes of and interactions between multiple decision-making (e.g., goal-directed, habitual, and Pavlovian) systems. In this way, we expect that the hBayesDM package will contribute to the dissemination of advanced modeling approaches and enable a wide range of researchers to easily perform computational psychiatric research within different populations.

Parallel Computing of Spatio-Temporal Model Based on Deep Reinforcement Learning

2021 ◽  
pp. 391-403
Author(s):  
Zhiqiang Lv ◽  
Jianbo Li ◽  
Zhihao Xu ◽  
Yue Wang ◽  
Haoran Li
Keyword(s):  
Parallel Computing ◽  
Reinforcement Learning ◽  
Model Based ◽  
Temporal Model ◽  
Spatio Temporal

scholarly journals Compositional generalization in multi-armed bandits

2021 ◽  
Author(s):  
Tankred Saanum ◽  
Eric Schulz ◽  
Maarten Speekenbrink
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Knowledge Transfer ◽  
Computational Model ◽  
Transfer Learning ◽  
Structure Learning ◽  
Reward Learning ◽  
Substantial Evidence ◽  
Better Than

To what extent do human reward learning and decision-making rely on the ability to represent and generate richly structured relationships between options? We provide evidence that structure learning and the principle of compositionality play crucial roles in human reinforcement learning. In a new multi-armed bandit paradigm, we found evidence that participants are able to learn representations of different reward structures and combine them to make correct generalizations about options in novel contexts. Moreover, we found substantial evidence that participants transferred knowledge of simpler reward structures to make compositional generalizations about rewards in complex contexts. This allowed participants to accumulate more rewards earlier, and to explore less whenever such knowledge transfer was possible. We also provide a computational model which is able to generalize and compose knowledge for complex reward structures. This model describes participant behaviour in the compositional generalization task better than various other models of decision-making and transfer learning.

scholarly journals Model based planners reflect on their model-free propensities

2021 ◽  
Vol 17 (1) ◽  
pp. e1008552
Author(s):  
Rani Moran ◽  
Mehdi Keramati ◽  
Raymond J. Dolan
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Drug Abuse ◽  
Learning Theory ◽  
Planning Model ◽  
Present Evidence ◽  
Model Based ◽  
Model Free ◽  
Short And Long Term

Dual-reinforcement learning theory proposes behaviour is under the tutelage of a retrospective, value-caching, model-free (MF) system and a prospective-planning, model-based (MB), system. This architecture raises a question as to the degree to which, when devising a plan, a MB controller takes account of influences from its MF counterpart. We present evidence that such a sophisticated self-reflective MB planner incorporates an anticipation of the influences its own MF-proclivities exerts on the execution of its planned future actions. Using a novel bandit task, wherein subjects were periodically allowed to design their environment, we show that reward-assignments were constructed in a manner consistent with a MB system taking account of its MF propensities. Thus, in the task participants assigned higher rewards to bandits that were momentarily associated with stronger MF tendencies. Our findings have implications for a range of decision making domains that includes drug abuse, pre-commitment, and the tension between short and long-term decision horizons in economics.

scholarly journals Model-based hierarchical reinforcement learning and human action control

2014 ◽  
Vol 369 (1655) ◽  
pp. 20130480 ◽  
Author(s):  
Matthew Botvinick ◽  
Ari Weinstein
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Hierarchical Model ◽  
Human Action ◽  
Action Control ◽  
Computational Framework ◽  
Hierarchical Reinforcement Learning ◽  
Model Based ◽  
Human Decision ◽  
Areas Of Interest

Recent work has reawakened interest in goal-directed or ‘model-based’ choice, where decisions are based on prospective evaluation of potential action outcomes. Concurrently, there has been growing attention to the role of hierarchy in decision-making and action control. We focus here on the intersection between these two areas of interest, considering the topic of hierarchical model-based control. To characterize this form of action control, we draw on the computational framework of hierarchical reinforcement learning, using this to interpret recent empirical findings. The resulting picture reveals how hierarchical model-based mechanisms might play a special and pivotal role in human decision-making, dramatically extending the scope and complexity of human behaviour.

scholarly journals Reduced model-based decision-making in gambling disorder

2019 ◽  
Vol 9 (1) ◽  
Author(s):  
Florent Wyckmans ◽  
A. Ross Otto ◽  
Miriam Sebold ◽  
Nathaniel Daw ◽  
Antoine Bechara ◽  
...  
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Gambling Disorder ◽  
Reaction Times ◽  
Learning System ◽  
Behavioral Learning ◽  
Model Based ◽  
Model Free ◽  
Effective Interventions ◽  
Compulsive Behaviors

AbstractCompulsive behaviors (e.g., addiction) can be viewed as an aberrant decision process where inflexible reactions automatically evoked by stimuli (habit) take control over decision making to the detriment of a more flexible (goal-oriented) behavioral learning system. These behaviors are thought to arise from learning algorithms known as “model-based” and “model-free” reinforcement learning. Gambling disorder, a form of addiction without the confound of neurotoxic effects of drugs, showed impaired goal-directed control but the way in which problem gamblers (PG) orchestrate model-based and model-free strategies has not been evaluated. Forty-nine PG and 33 healthy participants (CP) completed a two-step sequential choice task for which model-based and model-free learning have distinct and identifiable trial-by-trial learning signatures. The influence of common psychopathological comorbidities on those two forms of learning were investigated. PG showed impaired model-based learning, particularly after unrewarded outcomes. In addition, PG exhibited faster reaction times than CP following unrewarded decisions. Troubled mood, higher impulsivity (i.e., positive and negative urgency) and current and chronic stress reported via questionnaires did not account for those results. These findings demonstrate specific reinforcement learning and decision-making deficits in behavioral addiction that advances our understanding and may be important dimensions for designing effective interventions.

scholarly journals Revealing neuro-computational mechanisms of reinforcement learning and decision-making with the hBayesDM package

2016 ◽  
Author(s):  
Woo-Young Ahn ◽  
Nathaniel Haines ◽  
Lei Zhang
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Computational Modeling ◽  
Social Exchange ◽  
Latent Variables ◽  
Psychiatric Research ◽  
Model Based ◽  
Modeling Approaches ◽  
Wide Range ◽  
User Friendly

AbstractReinforcement learning and decision-making (RLDM) provide a quantitative framework and computational theories, with which we can disentangle psychiatric conditions into basic dimensions of neurocognitive functioning. RLDM offer a novel approach to assess and potentially diagnose psychiatric patients, and there is growing enthusiasm on RLDM and Computational Psychiatry among clinical researchers. Such a framework can also provide insights into the brain substrates of particular RLDM processes as exemplified by model-based functional magnetic resonance imaging (fMRI) or electroencephalogram (EEG). However, many researchers often find the approach too technical and have difficulty adopting it for their research. Thus, there remains a critical need to develop a user-friendly tool for the wide dissemination of computational psychiatric methods. We introduce an R package called hBayesDM (hierarchical Bayesian modeling of Decision-Making tasks), which offers computational modeling on an array of RLDM tasks and social exchange games. The hBayesDM package offers state-of-the-art hierarchical Bayesian modeling, where both individual and group parameters (i.e., posterior distributions) are estimated simultaneously in a mutually constraining fashion. At the same time, it is extremely user-friendly: users can perform computational modeling, output visualization, and Bayesian model comparisons–each with a single line of coding. Users can also extract trial-by-trial latent variables (e.g., prediction errors) required for model-based fMRI/EEG. With the hBayesDM package, we anticipate that anyone with minimal knowledge of programming can take advantage of cutting-edge computational modeling approaches and investigate the underlying processes of and interactions between multiple decision-making (e.g., goal-directed, habitual, and Pavlovian) systems. In this way, it is our expectation that the hBayesDM package will contribute to the dissemination of advanced modeling approaches and enable a wide range of researchers to easily perform computational psychiatric research within their populations.

Serotonergic Neurotransmission and Its Role in Negative Affect

2017 ◽  
Author(s):  
Andreas Heinz
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Mental Disorders ◽  
Negative Affect ◽  
Dopaminergic Neurotransmission ◽  
Model Based ◽  
Model Free ◽  
Versus Model ◽  
Serotonergic Neurotransmission

While dopaminergic neurotransmission has largely been implicated in reinforcement learning and model-based versus model-free decision making, serotonergic neurotransmission has been implicated in encoding aversive outcomes. Accordingly, serotonin dysfunction has been observed in disorders characterized by negative affect including depression, anxiety and addiction. Serotonin dysfunction in these mental disorders is described and its association with negative affect is discussed.

Sign in / Sign up

Export Citation Format

Share Document