Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices

Abstract The recent years have witnessed a dramatic increase in the use of reinforcement learning (RL) models in social, cognitive and affective neuroscience. This approach, in combination with neuroimaging techniques such as functional magnetic resonance imaging, enables quantitative investigations into latent mechanistic processes. However, increased use of relatively complex computational approaches has led to potential misconceptions and imprecise interpretations. Here, we present a comprehensive framework for the examination of (social) decision-making with the simple Rescorla–Wagner RL model. We discuss common pitfalls in its application and provide practical suggestions. First, with simulation, we unpack the functional role of the learning rate and pinpoint what could easily go wrong when interpreting differences in the learning rate. Then, we discuss the inevitable collinearity between outcome and prediction error in RL models and provide suggestions of how to justify whether the observed neural activation is related to the prediction error rather than outcome valence. Finally, we suggest posterior predictive check is a crucial step after model comparison, and we articulate employing hierarchical modeling for parameter estimation. We aim to provide simple and scalable explanations and practical guidelines for employing RL models to assist both beginners and advanced users in better implementing and interpreting their model-based analyses.

Download Full-text

Using reinforcement learning models in social neuroscience: frameworks, pitfalls, and suggestions of best practices

10.31234/osf.io/uthw2 ◽

2019 ◽

Author(s):

Lei Zhang ◽

Lukas Lengersdorff ◽

Nace Mikus ◽

Jan Gläscher ◽

Claus Lamm

Keyword(s):

Reinforcement Learning ◽

Prediction Error ◽

Model Comparison ◽

Hierarchical Modeling ◽

Learning Rate ◽

Neural Activation ◽

Affective Neuroscience ◽

Outcome Valence ◽

Practical Guidelines ◽

Reinforcement Learning Models

Recent years have witnessed a dramatic increase in the use of reinforcement learning (RL) models in social, cognitive and affective neuroscience. This approach, in combination with neuroimaging techniques such as functional magnetic resonance imaging, enables quantitative investigations into latent mechanistic processes. However, increased use of relatively complex computational approaches has led to potential misconceptions and imprecise interpretations. Here, we present a comprehensive framework for the examination of (social) decision-making with the simple Rescorla-Wagner RL model. We discuss common pitfalls in its application and provide practical suggestions. First, with simulation, we unpack the functional role of the learning rate and pinpoint what could easily go wrong when interpreting differences in the learning rate. Then, we discuss the inevitable collinearity between outcome and prediction error in RL models and provide suggestions of how to justify whether the observed neural activation is related to the prediction error rather than outcome valence. Finally, we suggest posterior predictive check is a crucial step after model comparison, and we articulate employing hierarchical modeling for parameter estimation. We aim to provide simple and scalable explanations and practical guidelines for employing RL models to assist both beginners and advanced users in better implementing and interpreting their model-based analyses.

Download Full-text

Modeling changes in probabilistic reinforcement learning during adolescence

PLoS Computational Biology ◽

10.1371/journal.pcbi.1008524 ◽

2021 ◽

Vol 17 (7) ◽

pp. e1008524

Author(s):

Liyu Xia ◽

Sarah L. Master ◽

Maria K. Eckstein ◽

Beth Baribault ◽

Ronald E. Dahl ◽

...

Keyword(s):

Reinforcement Learning ◽

Learning Task ◽

Learning Rate ◽

Integration Time ◽

Daily Experience ◽

Salivary Testosterone ◽

Probabilistic Uncertainty ◽

Probabilistic Reinforcement ◽

Hierarchical Bayesian Methods ◽

Reinforcement Learning Models

In the real world, many relationships between events are uncertain and probabilistic. Uncertainty is also likely to be a more common feature of daily experience for youth because they have less experience to draw from than adults. Some studies suggest probabilistic learning may be inefficient in youths compared to adults, while others suggest it may be more efficient in youths in mid adolescence. Here we used a probabilistic reinforcement learning task to test how youth age 8-17 (N = 187) and adults age 18-30 (N = 110) learn about stable probabilistic contingencies. Performance increased with age through early-twenties, then stabilized. Using hierarchical Bayesian methods to fit computational reinforcement learning models, we show that all participants’ performance was better explained by models in which negative outcomes had minimal to no impact on learning. The performance increase over age was driven by 1) an increase in learning rate (i.e. decrease in integration time scale); 2) a decrease in noisy/exploratory choices. In mid-adolescence age 13-15, salivary testosterone and learning rate were positively related. We discuss our findings in the context of other studies and hypotheses about adolescent brain development.

Download Full-text

Scaling prediction errors to reward variability benefits error-driven learning in humans

Journal of Neurophysiology ◽

10.1152/jn.00483.2015 ◽

2015 ◽

Vol 114 (3) ◽

pp. 1628-1640 ◽

Cited By ~ 27

Author(s):

Kelly M. J. Diederen ◽

Wolfram Schultz

Keyword(s):

Standard Deviation ◽

Prediction Error ◽

Probability Distributions ◽

Learning Rate ◽

Learning Performance ◽

Prediction Errors ◽

Impaired Performance ◽

Specific Standard ◽

Standard Deviations ◽

Reinforcement Learning Models

Effective error-driven learning requires individuals to adapt learning to environmental reward variability. The adaptive mechanism may involve decays in learning rate across subsequent trials, as shown previously, and rescaling of reward prediction errors. The present study investigated the influence of prediction error scaling and, in particular, the consequences for learning performance. Participants explicitly predicted reward magnitudes that were drawn from different probability distributions with specific standard deviations. By fitting the data with reinforcement learning models, we found scaling of prediction errors, in addition to the learning rate decay shown previously. Importantly, the prediction error scaling was closely related to learning performance, defined as accuracy in predicting the mean of reward distributions, across individual participants. In addition, participants who scaled prediction errors relative to standard deviation also presented with more similar performance for different standard deviations, indicating that increases in standard deviation did not substantially decrease “adapters'” accuracy in predicting the means of reward distributions. However, exaggerated scaling beyond the standard deviation resulted in impaired performance. Thus efficient adaptation makes learning more robust to changing variability.

Download Full-text

L-DOPA Reduces Model-Free Control of Behavior by Attenuating the Transfer of Value to Action

10.1101/086116 ◽

2016 ◽

Cited By ~ 2

Author(s):

Nils B. Kroemer ◽

Ying Lee ◽

Shakoor Pooseh ◽

Ben Eppinger ◽

Thomas Goschke ◽

...

Keyword(s):

Reinforcement Learning ◽

Prediction Error ◽

Behavioral Control ◽

Decision Task ◽

Learning Models ◽

Model Free ◽

Markov Decision ◽

Model Free Control ◽

Reinforcement Learning Models ◽

The Brain

AbstractDopamine is a key neurotransmitter in reinforcement learning and action control. Recent findings suggest that these components are inherently entangled. Here, we tested if increases in dopamine tone by administration of L-DOPA upregulate deliberative “model-based” control of behavior or reflexive “model-free” control as predicted by dual-control reinforcement-learning models. Alternatively, L-DOPA may impair learning as suggested by “value” or “thrift” theories of dopamine. To this end, we employed a two-stage Markov decision-task to investigate the effect of L-DOPA (randomized cross-over) on behavioral control while brain activation was measured using fMRI. L-DOPA led to attenuated model-free control of behavior as indicated by the reduced impact of reward on choice and increased stochasticity of model-free choices. Correspondingly, in the brain, L-DOPA decreased the effect of reward while prediction-error signals were unaffected. Taken together, our results suggest that L-DOPA reduces model-free control of behavior by attenuating the transfer of value to action.

Download Full-text

Signed and unsigned reward prediction errors dynamically enhance learning and memory

eLife ◽

10.7554/elife.61077 ◽

2021 ◽

Vol 10 ◽

Author(s):

Nina Rouhani ◽

Yael Niv

Keyword(s):

Reinforcement Learning ◽

Locus Coeruleus ◽

Learning And Memory ◽

Learning Rate ◽

Prediction Errors ◽

Learning Models ◽

The Past ◽

Reward Prediction ◽

Midbrain Dopamine ◽

Reinforcement Learning Models

Memory helps guide behavior, but which experiences from the past are prioritized? Classic models of learning posit that events associated with unpredictable outcomes as well as, paradoxically, predictable outcomes, deploy more attention and learning for those events. Here, we test reinforcement learning and subsequent memory for those events, and treat signed and unsigned reward prediction errors (RPEs), experienced at the reward-predictive cue or reward outcome, as drivers of these two seemingly contradictory signals. By fitting reinforcement learning models to behavior, we find that both RPEs contribute to learning by modulating a dynamically changing learning rate. We further characterize the effects of these RPE signals on memory, and show that both signed and unsigned RPEs enhance memory, in line with midbrain dopamine and locus-coeruleus modulation of hippocampal plasticity, thereby reconciling separate findings in the literature.

Download Full-text

Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences

10.1101/295022 ◽

2018 ◽

Author(s):

Sophie Bavard ◽

Maël Lebreton ◽

Mehdi Khamassi ◽

Giorgio Coricelli ◽

Stefano Palminteri

Keyword(s):

Reinforcement Learning ◽

Reference Point ◽

Model Comparison ◽

Contextual Information ◽

Task Structure ◽

Perceptual Decision Making ◽

Small Magnitude ◽

State Dependent ◽

Outcome Valence ◽

The Cost

AbstractIn economics and in perceptual decision-making contextual effects are well documented, where decision weights are adjusted as a function of the distribution of stimuli. Yet, in reinforcement learning literature whether and how contextual information pertaining to decision states is integrated in learning algorithms has received comparably little attention. Here, in an attempt to fill this gap, we investigated reinforcement learning behavior and its computational substrates in a task where we orthogonally manipulated both outcome valence and magnitude, resulting in systematic variations in state-values. Over two experiments, model comparison indicated that subjects’ behavior is best accounted for by an algorithm which includes both reference point-dependence and range-adaptation – two crucial features of state-dependent valuation. In addition, we found state-dependent outcome valuation to progressively emerge over time, to be favored by increasing outcome information and to be correlated with explicit understanding of the task structure. Finally, our data clearly show that, while being locally adaptive (for instance in negative valence and small magnitude contexts), state-dependent valuation comes at the cost of seemingly irrational choices, when options are extrapolated out from their original contexts.

Download Full-text

Modeling the Hemodynamic Response Function for Prediction Errors in the Ventral Striatum

10.1101/800136 ◽

2019 ◽

Author(s):

Gecia Bravo-Hermsdorff ◽

Yael Niv

Keyword(s):

Response Function ◽

Prediction Error ◽

Statistical Power ◽

Computational Models ◽

Model Comparison ◽

Ventral Striatum ◽

Hemodynamic Response ◽

Neural Activation ◽

Prediction Errors ◽

Hemodynamic Response Function

AbstractTo make sensible inferences about neural activation from fMRI data it is important to accurately model the hemodynamic response function (HRF), i.e., the hemodynamic response evoked by a punctuate neural event. HRF models have been derived for sensory areas, where it is relatively clear what events cause a neural impulse response. However, this is obviously harder to do for higher order cortices such as prefrontal areas. Therefore, one HRF model is commonly used for analyzing activity throughout the brain, despite the fact that hemodynamics are known to vary across regions. For instance, several fMRI studies use a canonical HRF to analyze ventral striatum (VS) activity where converging evidence indicates that reward prediction error signals drive neural activity. However, the VS is a target of prominent dopaminergic projections, known to modulate vasculature and affect BOLD activity, suggesting that the HRF in the VS may be especially different from those in other brain areas. To address this, we use data from an experiment focused on learning from prediction-error signals to derive a VS-specific HRF model (VS-HRF). We show that this new VS-HRF increases statistical power in model comparison. Our result is of particular relevance to studies comparing computational models of learning and/or decision making in the VS, and for connectivity analyses, where the use of an (even slightly) inaccurate HRF model can lead to erroneous conclusions. More broadly, our study highlights the importance of the choice of HRF model in determining the significance of the results obtained in classical univariate fMRI analysis.

Download Full-text

Joint Modeling of Reaction Times and Choice Improves Parameter Identifiability in Reinforcement Learning Models

10.1101/306720 ◽

2018 ◽

Author(s):

Ian C. Ballard ◽

Samuel M. McClure

Keyword(s):

Reinforcement Learning ◽

Model Fitting ◽

Reaction Times ◽

Learning Rate ◽

List Type ◽

Learning Models ◽

Parameter Identifiability ◽

Bayesian Priors ◽

Reinforcement Learning Models ◽

Parameters Of Reinforcement

AbstractBackgroundReinforcement learning models provide excellent descriptions of learning in multiple species across a variety of tasks. Many researchers are interested in relating parameters of reinforcement learning models to neural measures, psychological variables or experimental manipulations. We demonstrate that parameter identification is difficult because a range of parameter values provide approximately equal quality fits to data. This identification problem has a large impact on power: we show that a researcher who wants to detect a medium sized correlation (r = .3) with 80% power between a variable and learning rate must collect 60% more subjects than specified by a typical power analysis in order to account for the noise introduced by model fitting.New MethodWe derive a Bayesian optimal model fitting technique that takes advantage of information contained in choices and reaction times to constrain parameter estimates.ResultsWe show using simulation and empirical data that this method substantially improves the ability to recover learning rates.Comparison with Existing MethodsWe compare this method against the use of Bayesian priors. We show in simulations that the combined use of Bayesian priors and reaction times confers the highest parameter identifiability. However, in real data where the priors may have been misspecified, the use of Bayesian priors interferes with the ability of reaction time data to improve parameter identifiability.ConclusionsWe present a simple technique that takes advantage of readily available data to substantially improve the quality of inferences that can be drawn from parameters of reinforcement learning models.Highlights–Parameters of reinforcement learning models are particularly difficult to estimate–Incorporating reaction times into model fitting improves parameter identifiability–Bayesian weighting of choice and reaction times improves the power of analyses assessing learning rate

Download Full-text

Surprise acts as a reducer of outcome value in human reinforcement learning

10.31234/osf.io/5ha3y ◽

2019 ◽

Author(s):

Motofumi Sumiya ◽

Kentaro Katahira

Keyword(s):

Decision Making ◽

Reinforcement Learning ◽

Prediction Error ◽

Simulation Analysis ◽

Learning Task ◽

Risk Averse ◽

Choice Preference ◽

Proposed Model ◽

Modulation Parameter ◽

Reinforcement Learning Models

Surprise occurs because of differences between a decision outcome and its predicted outcome (prediction error), regardless of whether the error is positive or negative. It has recently been postulated that surprise affects the reward value of the action outcome itself; studies have indicated that increasing surprise, as absolute value of prediction error, decreases the value of the outcome. However, how surprise affects the value of the outcome and subsequent decision making is unclear. We suggested that, on the assumption that surprise decreases the outcome value, agents will increase their risk averse choices when an outcome is often surprisal. Here, we propose the surprise-sensitive utility model, a reinforcement learning model that states that surprise decreases the outcome value, to explain how surprise affects subsequent decision-making. To investigate the assumption, we compared this model with previous reinforcement learning models on a risky probabilistic learning task with simulation analysis, and model selection with two experimental datasets with different tasks and population. We further simulated a simple decision-making task to investigate how parameters within the proposed model modulate the choice preference. As a result, we found the proposed model explains the risk averse choices in a manner similar to the previous models, and risk averse choices increased as the surprise-based modulation parameter of outcome value increased. The model fits these datasets better than the other models, with same free parameters, thus providing a more parsimonious and robust account for risk averse choices. These findings indicate that surprise acts as a reducer of outcome value and decreases the action value for risky choices in which prediction error often occurs.

Download Full-text

Stimulation of the vagus nerve reduces learning in a go/no-go reinforcement learning task

10.1101/535260 ◽

2019 ◽

Cited By ~ 2

Author(s):

Anne Kühnel ◽

Vanessa Teckentrup ◽

Monja P. Neuser ◽

Quentin J. M. Huys ◽

Caroline Burrasch ◽

...

Keyword(s):

Reinforcement Learning ◽

Vagus Nerve ◽

Learning Task ◽

Afferent Input ◽

Learning Rate ◽

Action Execution ◽

Transcutaneous Vagus Nerve Stimulation ◽

Metabolic States ◽

Reinforcement Learning Models ◽

The Impact

AbstractWhen facing decisions to approach rewards or to avoid punishments, we often figuratively go with our gut, and the impact of metabolic states such as hunger on motivation are well documented. However, whether and how vagal feedback signals from the gut influence instrumental actions is unknown. Here, we investigated the effect of non-invasive transcutaneous vagus nerve stimulation (tVNS) vs. sham (randomized cross-over design) on approach and avoidance behavior using an established go/no-go reinforcement learning paradigm (Guitart-Masip et al., 2012) in 39 healthy, participants after an overnight fast. First, mixed-effects logistic regression analysis of choice accuracy showed that tVNS acutely impaired decision-making, p = .045. Computational reinforcement learning models identified the cause of this as a reduction in the learning rate through tVNS (Δα = −0.092, pboot= .002), particularly after punishment (ΔαPun= −0.081, pboot= .012 vs. ΔαRew= −0.031, p = .22). However, tVNS had no effect on go biases, Pavlovian response biases or response time. Hence, tVNS appeared to influence learning rather than action execution. These results highlight a novel role of vagal afferent input in modulating reinforcement learning by tuning the learning rate according to homeostatic needs.

Download Full-text