scholarly journals Asymmetric and adaptive reward coding via normalized reinforcement learning

2021 ◽  
Author(s):  
Kenway Louie
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Computational Models ◽  
Nonlinear Function ◽  
Value Functions ◽  
Learning Rates ◽  
Error Coding ◽  
Asymmetric Learning ◽  
Reward Functions ◽  
The Brain

Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-guided reinforcement learning (RL) algorithms. While standard RL assumes linear reward functions, reward-related neural activity is a saturating, nonlinear function of reward; however, the computational and behavioral implications of nonlinear RL are unknown. Here, we show that nonlinear RL incorporating the canonical divisive normalization computation introduces an intrinsic and tunable asymmetry in prediction error coding. At the behavioral level, this asymmetry explains empirical variability in risk preferences typically attributed to asymmetric learning rates. At the neural level, diversity in asymmetries provides a computational mechanism for recently proposed theories of distributional RL, allowing the brain to learn the full probability distribution of future rewards. This behavioral and computational flexibility argues for an incorporation of biologically valid value functions in computational models of learning and decision-making.

scholarly journals Prefrontal solution to the bias-variance tradeoff during reinforcement learning

2020 ◽  
Author(s):  
Dongjae Kim ◽  
Jaeseung Jeong ◽  
Sang Wan Lee
Keyword(s):  
Adaptive Control ◽  
Reinforcement Learning ◽  
Prediction Error ◽  
Brain Regions ◽  
Decision Task ◽  
Prediction Errors ◽  
Model Based ◽  
Model Free ◽  
Bias Variance ◽  
The Brain

AbstractThe goal of learning is to maximize future rewards by minimizing prediction errors. Evidence have shown that the brain achieves this by combining model-based and model-free learning. However, the prediction error minimization is challenged by a bias-variance tradeoff, which imposes constraints on each strategy’s performance. We provide new theoretical insight into how this tradeoff can be resolved through the adaptive control of model-based and model-free learning. The theory predicts the baseline correction for prediction error reduces the lower bound of the bias–variance error by factoring out irreducible noise. Using a Markov decision task with context changes, we showed behavioral evidence of adaptive control. Model-based behavioral analyses show that the prediction error baseline signals context changes to improve adaptability. Critically, the neural results support this view, demonstrating multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free learning.One sentence summaryA theoretical, behavioral, computational, and neural account of how the brain resolves the bias-variance tradeoff during reinforcement learning is described.

scholarly journals Multi-Task Reinforcement Learning in Humans

2019 ◽  
Author(s):  
Momchil S. Tomov ◽  
Eric Schulz ◽  
Samuel J. Gershman
Keyword(s):  
Reinforcement Learning ◽  
State Of The Art ◽  
Task Demands ◽  
Human Intelligence ◽  
Value Functions ◽  
Complex Environments ◽  
Multiple Features ◽  
Model Free ◽  
Reward Functions ◽  
Study Participants

ABSTRACTThe ability to transfer knowledge across tasks and generalize to novel ones is an important hallmark of human intelligence. Yet not much is known about human multi-task reinforcement learning. We study participants’ behavior in a novel two-step decision making task with multiple features and changing reward functions. We compare their behavior to two state-of-the-art algorithms for multi-task reinforcement learning, one that maps previous policies and encountered features to new reward functions and one that approximates value functions across tasks, as well as to standard model-based and model-free algorithms. Across three exploratory experiments and a large preregistered experiment, our results provide strong evidence for a strategy that maps previously learned policies to novel scenarios. These results enrich our understanding of human reinforcement learning in complex environments with changing task demands.

scholarly journals Trading off the cost of conflict against expected rewards

2018 ◽  
Author(s):  
Nura Sidarus ◽  
Stefano Palminteri ◽  
Valérian Chambon
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Cognitive Control ◽  
Reversal Learning ◽  
Computational Models ◽  
Learning Task ◽  
Learning Rates ◽  
Different Types ◽  
Irrelevant Distractors ◽  
The Cost

AbstractValue-based decision-making involves trading off the cost associated with an action against its expected reward. Research has shown that both physical and mental effort constitute such subjective costs, biasing choices away from effortful actions, and discounting the value of obtained rewards. Facing conflicts between competing action alternatives is considered aversive, as recruiting cognitive control to overcome conflict is effortful. Yet, it remains unclear whether conflict is also perceived as a cost in value-based decisions. The present study investigated this question by embedding irrelevant distractors (flanker arrows) within a reversal-learning task, with intermixed free and instructed trials. Results showed that participants learned to adapt their choices to maximize rewards, but were nevertheless biased to follow the suggestions of irrelevant distractors. Thus, the perceived cost of being in conflict with an external suggestion could sometimes trump internal value representations. By adapting computational models of reinforcement learning, we assessed the influence of conflict at both the decision and learning stages. Modelling the decision showed that conflict was avoided when evidence for either action alternative was weak, demonstrating that the cost of conflict was traded off against expected rewards. During the learning phase, we found that learning rates were reduced in instructed, relative to free, choices. Learning rates were further reduced by conflict between an instruction and subjective action values, whereas learning was not robustly influenced by conflict between one’s actions and external distractors. Our results show that the subjective cost of conflict factors into value-based decision-making, and highlights that different types of conflict may have different effects on learning about action outcomes.

scholarly journals Computational modelling of social cognition and behaviour—a reinforcement learning primer

2020 ◽  
Author(s):  
Patricia L Lockwood ◽  
Miriam C Klein-Flügge
Keyword(s):  
Reinforcement Learning ◽  
Social Cognition ◽  
Observational Learning ◽  
Impression Formation ◽  
Computational Models ◽  
Computational Modelling ◽  
Neural Systems ◽  
Neural Basis ◽  
The Past ◽  
The Brain

Abstract Social neuroscience aims to describe the neural systems that underpin social cognition and behaviour. Over the past decade, researchers have begun to combine computational models with neuroimaging to link social computations to the brain. Inspired by approaches from reinforcement learning theory, which describes how decisions are driven by the unexpectedness of outcomes, accounts of the neural basis of prosocial learning, observational learning, mentalizing and impression formation have been developed. Here we provide an introduction for researchers who wish to use these models in their studies. We consider both theoretical and practical issues related to their implementation, with a focus on specific examples from the field.

scholarly journals Dissociation between asymmetric value updating and perseverance in human reinforcement learning

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Michiyo Sugawara ◽  
Kentaro Katahira
Keyword(s):  
Reinforcement Learning ◽  
Hybrid Model ◽  
Empirical Data ◽  
Choice Behavior ◽  
Learning Rate ◽  
Web Based ◽  
Learning Rates ◽  
Asymmetric Learning ◽  
Open Datasets ◽  
Key Parameter

AbstractThe learning rate is a key parameter in reinforcement learning that determines the extent to which novel information (outcome) is incorporated in guiding subsequent actions. Numerous studies have reported that the magnitude of the learning rate in human reinforcement learning is biased depending on the sign of the reward prediction error. However, this asymmetry can be observed as a statistical bias if the fitted model ignores the choice autocorrelation (perseverance), which is independent of the outcomes. Therefore, to investigate the genuine process underlying human choice behavior using empirical data, one should dissociate asymmetry in learning and perseverance from choice behavior. The present study addresses this issue by using a Hybrid model incorporating asymmetric learning rates and perseverance. First, by conducting simulations, we demonstrate that the Hybrid model can identify the true underlying process. Second, using the Hybrid model, we show that empirical data collected from a web-based experiment are governed by perseverance rather than asymmetric learning. Finally, we apply the Hybrid model to two open datasets in which asymmetric learning was reported. As a result, the asymmetric learning rate was validated in one dataset but not another.

scholarly journals L-DOPA Reduces Model-Free Control of Behavior by Attenuating the Transfer of Value to Action

2016 ◽  
Author(s):  
Nils B. Kroemer ◽  
Ying Lee ◽  
Shakoor Pooseh ◽  
Ben Eppinger ◽  
Thomas Goschke ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Behavioral Control ◽  
Decision Task ◽  
Learning Models ◽  
Model Free ◽  
Markov Decision ◽  
Model Free Control ◽  
Reinforcement Learning Models ◽  
The Brain

AbstractDopamine is a key neurotransmitter in reinforcement learning and action control. Recent findings suggest that these components are inherently entangled. Here, we tested if increases in dopamine tone by administration of L-DOPA upregulate deliberative “model-based” control of behavior or reflexive “model-free” control as predicted by dual-control reinforcement-learning models. Alternatively, L-DOPA may impair learning as suggested by “value” or “thrift” theories of dopamine. To this end, we employed a two-stage Markov decision-task to investigate the effect of L-DOPA (randomized cross-over) on behavioral control while brain activation was measured using fMRI. L-DOPA led to attenuated model-free control of behavior as indicated by the reduced impact of reward on choice and increased stochasticity of model-free choices. Correspondingly, in the brain, L-DOPA decreased the effect of reward while prediction-error signals were unaffected. Taken together, our results suggest that L-DOPA reduces model-free control of behavior by attenuating the transfer of value to action.

scholarly journals Reinforcement learning for robotic manipulation using simulated locomotion demonstrations

2021 ◽  
Author(s):  
Ozsel Kilinc ◽  
Giovanni Montana
Keyword(s):  
Reinforcement Learning ◽  
Space Exploration ◽  
Approaches To Learning ◽  
Robotic Manipulation ◽  
Rigid Object ◽  
State Action ◽  
Learning Rates ◽  
Recent Developments ◽  
Manipulation Task ◽  
Reward Functions

AbstractMastering robotic manipulation skills through reinforcement learning (RL) typically requires the design of shaped reward functions. Recent developments in this area have demonstrated that using sparse rewards, i.e. rewarding the agent only when the task has been successfully completed, can lead to better policies. However, state-action space exploration is more difficult in this case. Recent RL approaches to learning with sparse rewards have leveraged high-quality human demonstrations for the task, but these can be costly, time consuming or even impossible to obtain. In this paper, we propose a novel and effective approach that does not require human demonstrations. We observe that every robotic manipulation task could be seen as involving a locomotion task from the perspective of the object being manipulated, i.e. the object could learn how to reach a target state on its own. In order to exploit this idea, we introduce a framework whereby an object locomotion policy is initially obtained using a realistic physics simulator. This policy is then used to generate auxiliary rewards, called simulated locomotion demonstration rewards (SLDRs), which enable us to learn the robot manipulation policy. The proposed approach has been evaluated on 13 tasks of increasing complexity, and can achieve higher success rate and faster learning rates compared to alternative algorithms. SLDRs are especially beneficial for tasks like multi-object stacking and non-rigid object manipulation.

scholarly journals Individual differences in successful self-regulation of the dopaminergic midbrain

2019 ◽  
Author(s):  
Lydia Hellrung ◽  
Matthias Kirschner ◽  
James Sulzer ◽  
Ronald Sladky ◽  
Frank Scharnowski ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Individual Differences ◽  
Prediction Error ◽  
Self Regulation ◽  
Reward Sensitivity ◽  
Neurofeedback Training ◽  
Feedback Group ◽  
Brain Functions ◽  
Error Coding ◽  
Monetary Incentive Delay

AbstractThe dopaminergic midbrain is associated with brain functions, such as reinforcement learning, motivation and decision-making that are often disturbed in neuropsychiatric disease. Previous research has shown that activity in the dopaminergic midbrain can be endogenously modulated via neurofeedback, suggesting potential for non-pharmacological interventions. However, the robustness of endogenous modulation, a requirement for clinical translation, is unclear. Here, we examined how self-modulation capability relates to regulation transfer. Moreover, to elucidate potential mechanisms underlying successful self-regulation, we studied individual prediction error coding, and, during an independent monetary incentive delay (MID) task, individual reward sensitivity. Fifty-nine participants underwent neurofeedback training either in a veridical or inverted feedback group. Successful self-regulation was associated with post-training activity within the cognitive control network and accompanied by decreasing prefrontal prediction error signals and increased prefrontal reward sensitivity in the MID task. The correlative link of dopaminergic self-regulation with individual differences in prefrontal prediction error and reward sensitivity suggests that reinforcement learning contributes to successful self-regulation. Our findings therefore provide new insights in the control of dopaminergic midbrain activity and pave the way to improve neurofeedback training in neuropsychiatric patients.

scholarly journals Reward-Free Reinforcement Learning Algorithm Using Prediction Network

2020 ◽  
Author(s):  
Zhen Yu ◽  
Yimin Feng ◽  
Lijun Liu
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Value Functions ◽  
Learning Method ◽  
Reward Function ◽  
Network Training ◽  
Learning Tasks ◽  
Reward Value ◽  
Policy Gradient ◽  
Reward Functions

In general reinforcement learning tasks, the formulation of reward functions is a very important step in reinforcement learning. The reward function is not easy to formulate in a large number of systems. The network training effect is sensitive to the reward function, and different reward value functions will get different results. For a class of systems that meet specific conditions, the traditional reinforcement learning method is improved. A state quantity function is designed to replace the reward function, which is more efficient than the traditional reward function. At the same time, the predictive network link is designed so that the network can learn the value of the general state by using the special state. The overall structure of the network will be improved based on the Deep Deterministic Policy Gradient (DDPG) algorithm. Finally, the algorithm was successfully applied in the environment of FrozenLake, and achieved good performance. The experiment proves the effectiveness of the algorithm and realizes rewardless reinforcement learning in a class of systems.

scholarly journals Computational modelling of social cognition and behaviour – a reinforcement learning primer

2019 ◽  
Author(s):  
Patricia Lockwood ◽  
Miriam Klein-Flugge
Keyword(s):  
Reinforcement Learning ◽  
Social Cognition ◽  
Observational Learning ◽  
Impression Formation ◽  
Computational Models ◽  
Computational Modelling ◽  
Neural Systems ◽  
Neural Basis ◽  
The Past ◽  
The Brain

Social neuroscience aims to describe the neural systems that underpin social cognition and behaviour. Over the past decade, researchers have begun to combine computational models with neuroimaging to link social computations to the brain. Inspired by approaches from reinforcement learning theory, which describes how decisions are driven by the unexpectedness of outcomes, accounts of the neural basis of prosocial learning, observational learning, mentalising and impression formation have been developed. Here we provide an introduction for researchers who wish to use these models in their studies. We consider both theoretical and practical issues related to their implementation, with a focus on specific examples from the field.

Sign in / Sign up

Export Citation Format

Share Document