Deep Reinforcement Learning using Cyclical Learning Rates

AbstractWe are living in a dynamic world in which stochastic relationships between cues and outcome events create different sources of uncertainty1 (e.g. the fact that not all grey clouds bring rain). Living in an uncertain world continuously probes learning systems in the brain, guiding agents to make better decisions. This is a type of value-based decision-making which is very important for survival in the wild and long-term evolutionary fitness. Consequently, reinforcement learning (RL) models describing cognitive/computational processes underlying learning-based adaptations have been pivotal in behavioural2,3 and neural sciences4–6, as well as machine learning7,8. This paper demonstrates the suitability of novel update rules for RL, based on a nonlinear relationship between prediction errors (i.e. difference between the agent’s expectation and the actual outcome) and learning rates (i.e. a coefficient with which agents update their beliefs about the environment), that can account for learning-based adaptations in the face of environmental uncertainty. These models illustrate how learners can flexibly adapt to dynamically changing environments.

Download Full-text

EFFECTS OF COMMUNICATION ON GROUP LEARNING RATES IN A MULTI-AGENT ENVIRONMENT

Advances in Complex Systems ◽

10.1142/s0219525903000979 ◽

2003 ◽

Vol 06 (03) ◽

pp. 405-426 ◽

Cited By ~ 1

Author(s):

PAUL DARBYSHIRE

Keyword(s):

Reinforcement Learning ◽

Cognitive Abilities ◽

Complex Adaptive System ◽

Machine Learning Techniques ◽

Simulation Techniques ◽

Learning Rates ◽

Learning Techniques ◽

Complex Adaptive ◽

Rate Of Learning ◽

Multi Agent

Distillations utilize multi-agent based modeling and simulation techniques to study warfare as a complex adaptive system at the conceptual level. The focus is placed on the interactions between the agents to facilitate study of cause and effect between individual interactions and overall system behavior. Current distillations do not utilize machine-learning techniques to model the cognitive abilities of individual combatants but employ agent control paradigms to represent agents as highly instinctual entities. For a team of agents implementing a reinforcement-learning paradigm, the rate of learning is not sufficient for agents to adapt to this hostile environment. However, by allowing the agents to communicate their respective rewards for actions performed as the simulation progresses, the rate of learning can be increased sufficiently to significantly increase the teams chances of survival. This paper presents the results of trials to measure the success of a team-based approach to the reinforcement-learning problem in a distillation, using reward communication to increase learning rates.

Download Full-text

Velocity estimation in reinforcement learning

10.1101/432492 ◽

2018 ◽

Author(s):

Carlos Velazquez ◽

Manuel Villarreal ◽

Arturo Bouzas

Keyword(s):

Reinforcement Learning ◽

Hierarchical Structure ◽

Predictive Accuracy ◽

Information Criterion ◽

Individual Variability ◽

Velocity Estimation ◽

Hyperbolic Function ◽

Prediction Errors ◽

Signal To Noise ◽

Learning Rates

The current work aims to study how people make predictions, under a reinforcement learning framework, in an environment that fluctuates from trial to trial and is corrupted with Gaussian noise. A computer-based experiment was developed where subjects were required to predict the future location of a spaceship that orbited around planet Earth. Its position was sampled from a Gaussian distribution with the mean changing at a variable velocity and four different values of variance that defined our signal-to-noise conditions. Three error-driven algorithms using a Bayesian approach were proposed as candidates to describe our data. The first is the standard delta-rule. The second and third models are delta rules incorporating a velocity component which is updated using prediction errors. The third model additionally assumes a hierarchical structure where individual learning rates for velocity and decision noise come from Gaussian distributions with means following a hyperbolic function. We used leave-one-out cross-validation and the Widely Applicable Information Criterion to compare the predictive accuracy of these models. In general, our results provided evidence in favor of the hierarchical model and highlight two main conclusions. First, when facing an environment that fluctuates from trial to trial, people can learn to estimate its velocity to make predictions. Second, learning rates for velocity and decision noise are influenced by uncertainty constraints represented by the signal-to-noise ratio. This higher order control was modeled using a hierarchical structure, which qualitatively accounts for individual variability and is able to generalize and make predictions about new subjects on each experimental condition.

Download Full-text

Trading off the cost of conflict against expected rewards

10.1101/412809 ◽

2018 ◽

Author(s):

Nura Sidarus ◽

Stefano Palminteri ◽

Valérian Chambon

Keyword(s):

Decision Making ◽

Reinforcement Learning ◽

Cognitive Control ◽

Reversal Learning ◽

Computational Models ◽

Learning Task ◽

Learning Rates ◽

Different Types ◽

Irrelevant Distractors ◽

The Cost

AbstractValue-based decision-making involves trading off the cost associated with an action against its expected reward. Research has shown that both physical and mental effort constitute such subjective costs, biasing choices away from effortful actions, and discounting the value of obtained rewards. Facing conflicts between competing action alternatives is considered aversive, as recruiting cognitive control to overcome conflict is effortful. Yet, it remains unclear whether conflict is also perceived as a cost in value-based decisions. The present study investigated this question by embedding irrelevant distractors (flanker arrows) within a reversal-learning task, with intermixed free and instructed trials. Results showed that participants learned to adapt their choices to maximize rewards, but were nevertheless biased to follow the suggestions of irrelevant distractors. Thus, the perceived cost of being in conflict with an external suggestion could sometimes trump internal value representations. By adapting computational models of reinforcement learning, we assessed the influence of conflict at both the decision and learning stages. Modelling the decision showed that conflict was avoided when evidence for either action alternative was weak, demonstrating that the cost of conflict was traded off against expected rewards. During the learning phase, we found that learning rates were reduced in instructed, relative to free, choices. Learning rates were further reduced by conflict between an instruction and subjective action values, whereas learning was not robustly influenced by conflict between one’s actions and external distractors. Our results show that the subjective cost of conflict factors into value-based decision-making, and highlights that different types of conflict may have different effects on learning about action outcomes.

Download Full-text

Effect of lysergic acid diethylamide (LSD) on reinforcement learning in humans

10.1101/2020.12.04.412189 ◽

2020 ◽

Author(s):

Jonathan W. Kanen ◽

Qiang Luo ◽

Mojtaba R. Kandroodi ◽

Rudolf N. Cardinal ◽

Trevor W. Robbins ◽

...

Keyword(s):

Reinforcement Learning ◽

Computational Modelling ◽

Lysergic Acid ◽

Learning Rate ◽

Lysergic Acid Diethylamide ◽

Immediate Feedback ◽

Healthy Humans ◽

Learning Rates ◽

Psychedelic Drugs ◽

The Impact

AbstractThe non-selective serotonin 2A (5-HT2A) receptor agonist lysergic acid diethylamide (LSD) holds promise as a treatment for some psychiatric disorders. Psychedelic drugs such as LSD have been suggested to have therapeutic actions through their effects on learning. The behavioural effects of LSD in humans, however, remain largely unexplored. Here we examined how LSD affects probabilistic reversal learning in healthy humans. Conventional measures assessing sensitivity to immediate feedback (“win-stay” and “lose-shift” probabilities) were unaffected, whereas LSD increased the impact of the strength of initial learning on perseveration. Computational modelling revealed that the most pronounced effect of LSD was enhancement of the reward learning rate. The punishment learning rate was also elevated. Increased reinforcement learning rates suggest LSD induced a state of heightened plasticity. These results indicate a potential mechanism through which revision of maladaptive associations could occur.

Download Full-text

Explaining Valence Asymmetries in Value Learning: A Reinforcement Learning Account

10.31234/osf.io/23kuf ◽

2022 ◽

Author(s):

Chenxu Hao ◽

Lilian E. Cabrera-Haro ◽

Ziyong Lin ◽

Patricia Reuter-Lorenz ◽

Richard L. Lewis

Keyword(s):

Reinforcement Learning ◽

Downstream Processing ◽

Learning Task ◽

Model Parameters ◽

Special Role ◽

Learning Rates ◽

The Asymmetry ◽

Value Learning ◽

Choice Policy ◽

Better Than

To understand how acquired value impacts how we perceive and process stimuli, psychologists have developed the Value Learning Task (VLT; e.g., Raymond & O’Brien, 2009). The task consists of a series of trials in which participants attempt to maximize accumulated winnings as they make choices from a pair of presented images associated with probabilistic win, loss, or no-change outcomes. Despite the task having a symmetric outcome structure for win and loss pairs, people learn win associations better than loss associations (Lin, Cabrera-Haro, & Reuter-Lorenz, 2020). This asymmetry could lead to differences when the stimuli are probed in subsequent tasks, compromising inferences about how acquired value affects downstream processing. We investigate the nature of the asymmetry using a standard error-driven reinforcement learning model with a softmax choice rule. Despite having no special role for valence, the model yields the asymmetry observed in human behavior, whether the model parameters are set to maximize empirical fit, or task payoff. The asymmetry arises from an interaction between a neutral initial value estimate and a choice policy that exploits while exploring, leading to more poorly discriminated value estimates for loss stimuli. We also show how differences in estimated individual learning rates help to explain individual differences in the observed win-loss asymmetries, and how the final value estimates produced by the model provide a simple account of a post-learning explicit value categorization task.

Download Full-text

Diffusion modeling reveals reinforcement learning impairments in gambling disorder that are linked to attenuated ventromedial prefrontal cortex value representations

10.1101/2020.06.03.131359 ◽

2020 ◽

Cited By ~ 2

Author(s):

Antonius Wiehler ◽

Jan Peters

Keyword(s):

Prefrontal Cortex ◽

Reinforcement Learning ◽

Gambling Disorder ◽

Ventromedial Prefrontal Cortex ◽

Response Threshold ◽

Boundary Separation ◽

Learning Impairments ◽

Model Based ◽

Learning Rates ◽

Over Time

AbstractGambling disorder is associated with deficits in classical feedback-based learning tasks, but the computational mechanisms underlying such learning impairments are still poorly understood. Here, we examined this question using a combination of computational modeling and functional resonance imaging (fMRI) in gambling disorder participants (n=23) and matched controls (n=19). Participants performed a simple reinforcement learning task with two pairs of stimuli (80% vs. 20% reinforcement rates per pair). As predicted, gamblers made significantly fewer selections of the optimal stimulus, while overall response times (RTs) were not significantly different between groups. We then used comprehensive modeling using reinforcement learning drift diffusion models (RLDDMs) in combination with hierarchical Bayesian parameter estimation to shed light on the computational underpinnings of this performance impairment. In both groups, an RLDDM in which both non-decision time and response threshold (boundary separation) changed over the course of the experiment accounted for the data best. The model showed good parameter recovery, and posterior predictive checks revealed that in both groups, the model reproduced the evolution of both accuracy and RTs over time. Examination of the group-wise posterior distributions revealed that the learning impairment in gamblers was attributable to both reduced learning rates and a more rapid reduction in boundary separation over time, compared to controls. Furthermore, gamblers also showed substantially shorter non-decision times. Model-based imaging analyses then revealed that value representations in gamblers in the ventromedial prefrontal cortex were attenuated compared to controls, and these effects were partly associated with model-based learning rates. Exploratory analyses revealed that a more anterior ventromedial prefrontal cortex cluster showed attenuations in value representations in proportion to gambling disorder severity in gamblers. Taken together, our findings reveal computational mechanisms underlying reinforcement learning impairments in gambling disorder, and confirm the ventromedial prefrontal cortex and as a critical neural hub in this disorder.

Download Full-text

Feasibility Analysis and Application of Reinforcement Learning Algorithm Based on Dynamic Parameter Adjustment

Algorithms ◽

10.3390/a13090239 ◽

2020 ◽

Vol 13 (9) ◽

pp. 239 ◽

Cited By ~ 1

Author(s):

Menglin Li ◽

Xueqiang Gu ◽

Chengyi Zeng ◽

Yuan Feng

Keyword(s):

Machine Learning ◽

Reinforcement Learning ◽

Learning Algorithm ◽

Dynamic Parameter ◽

Learning Rate ◽

New Method ◽

Practical Application ◽

Learning Rates ◽

Dynamic Parameter Adjustment ◽

Reinforcement Learning Algorithm

Reinforcement learning, as a branch of machine learning, has been gradually applied in the control field. However, in the practical application of the algorithm, the hyperparametric approach to network settings for deep reinforcement learning still follows the empirical attempts of traditional machine learning (supervised learning and unsupervised learning). This method ignores part of the information generated by agents exploring the environment contained in the updating of the reinforcement learning value function, which will affect the performance of the convergence and cumulative return of reinforcement learning. The reinforcement learning algorithm based on dynamic parameter adjustment is a new method for setting learning rate parameters of deep reinforcement learning. Based on the traditional method of setting parameters for reinforcement learning, this method analyzes the advantages of different learning rates at different stages of reinforcement learning and dynamically adjusts the learning rates in combination with the temporal-difference (TD) error values to achieve the advantages of different learning rates in different stages to improve the rationality of the algorithm in practical application. At the same time, by combining the Robbins–Monro approximation algorithm and deep reinforcement learning algorithm, it is proved that the algorithm of dynamic regulation learning rate can theoretically meet the convergence requirements of the intelligent control algorithm. In the experiment, the effect of this method is analyzed through the continuous control scenario in the standard experimental environment of ”Car-on-The-Hill” of reinforcement learning, and it is verified that the new method can achieve better results than the traditional reinforcement learning in practical application. According to the model characteristics of the deep reinforcement learning, a more suitable setting method for the learning rate of the deep reinforcement learning network proposed. At the same time, the feasibility of the method has been proved both in theory and in the application. Therefore, the method of setting the learning rate parameter is worthy of further development and research.

Download Full-text

Social Anxiety and Dynamic Social Reinforcement Learning in a Volatile Environment

Clinical Psychological Science ◽

10.1177/2167702619858425 ◽

2019 ◽

Vol 7 (6) ◽

pp. 1372-1388

Author(s):

Miranda L. Beltzer ◽

Stephen Adams ◽

Peter A. Beling ◽

Bethany A. Teachman

Keyword(s):

Reinforcement Learning ◽

Social Anxiety ◽

Social Reinforcement ◽

Dynamic Learning ◽

Socially Anxious ◽

Role Changes ◽

Learning Rates ◽

New Information ◽

Dynamic Updating ◽

Reward And Punishment

Adaptive social behavior requires learning probabilities of social reward and punishment and updating these probabilities when they change. Given prior research on aberrant reinforcement learning in affective disorders, this study examines how social anxiety affects probabilistic social reinforcement learning and dynamic updating of learned probabilities in a volatile environment. Two hundred and twenty-two online participants completed questionnaires and a computerized ball-catching game with changing probabilities of reward and punishment. Dynamic learning rates were estimated to assess the relative importance ascribed to new information in response to volatility. Mixed-effects regression was used to analyze throw patterns as a function of social anxiety symptoms. Higher social anxiety predicted fewer throws to the previously punishing avatar and different learning rates after certain role changes, suggesting that social anxiety may be characterized by difficulty updating learned social probabilities. Socially anxious individuals may miss the chance to learn that a once-punishing situation no longer poses a threat.

Download Full-text

Dissociation between asymmetric value updating and perseverance in human reinforcement learning

Scientific Reports ◽

10.1038/s41598-020-80593-7 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Michiyo Sugawara ◽

Kentaro Katahira

Keyword(s):

Reinforcement Learning ◽

Hybrid Model ◽

Empirical Data ◽

Choice Behavior ◽

Learning Rate ◽

Web Based ◽

Learning Rates ◽

Asymmetric Learning ◽

Open Datasets ◽

Key Parameter

AbstractThe learning rate is a key parameter in reinforcement learning that determines the extent to which novel information (outcome) is incorporated in guiding subsequent actions. Numerous studies have reported that the magnitude of the learning rate in human reinforcement learning is biased depending on the sign of the reward prediction error. However, this asymmetry can be observed as a statistical bias if the fitted model ignores the choice autocorrelation (perseverance), which is independent of the outcomes. Therefore, to investigate the genuine process underlying human choice behavior using empirical data, one should dissociate asymmetry in learning and perseverance from choice behavior. The present study addresses this issue by using a Hybrid model incorporating asymmetric learning rates and perseverance. First, by conducting simulations, we demonstrate that the Hybrid model can identify the true underlying process. Second, using the Hybrid model, we show that empirical data collected from a web-based experiment are governed by perseverance rather than asymmetric learning. Finally, we apply the Hybrid model to two open datasets in which asymmetric learning was reported. As a result, the asymmetric learning rate was validated in one dataset but not another.

Download Full-text