scholarly journals Evolutionary Algorithms for Reinforcement Learning

1999 ◽  
Vol 11 ◽  
pp. 241-276 ◽  
Author(s):  
D. E. Moriarty ◽  
A. C. Schultz ◽  
J. J. Grefenstette
Keyword(s):  
Reinforcement Learning ◽  
Evolutionary Algorithms ◽  
Value Function ◽  
Learning Problems ◽  
Temporal Difference ◽  
Genetic Operators ◽  
Credit Assignment ◽  
Learning Problem ◽  
Difference Methods ◽  
Temporal Difference Methods

There are two distinct approaches to solving reinforcement learning problems, namely, searching in value function space and searching in policy space. Temporal difference methods and evolutionary algorithms are well-known examples of these approaches. Kaelbling, Littman and Moore recently provided an informative survey of temporal difference methods. This article focuses on the application of evolutionary algorithms to the reinforcement learning problem, emphasizing alternative policy representations, credit assignment methods, and problem-specific genetic operators. Strengths and weaknesses of the evolutionary approach to reinforcement learning are presented, along with a survey of representative applications.

scholarly journals Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning

2020 ◽  
Vol 34 (04) ◽  
pp. 3741-3748
Author(s):  
Kristopher De Asis ◽  
Alan Chan ◽  
Silviu Pitis ◽  
Richard Sutton ◽  
Daniel Graves
Keyword(s):  
Reinforcement Learning ◽  
Function Approximation ◽  
Value Function ◽  
Temporal Difference ◽  
Value Functions ◽  
Difference Methods ◽  
Td Methods ◽  
The Stability ◽  
The Value Function ◽  
Temporal Difference Methods

We explore fixed-horizon temporal difference (TD) methods, reinforcement learning algorithms for a new kind of value function that predicts the sum of rewards over a fixed number of future time steps. To learn the value function for horizon h, these algorithms bootstrap from the value function for horizon h−1, or some shorter horizon. Because no value function bootstraps from itself, fixed-horizon methods are immune to the stability problems that plague other off-policy TD methods using function approximation (also known as “the deadly triad”). Although fixed-horizon methods require the storage of additional value functions, this gives the agent additional predictive power, while the added complexity can be substantially reduced via parallel updates, shared weights, and n-step bootstrapping. We show how to use fixed-horizon value functions to solve reinforcement learning problems competitively with methods such as Q-learning that learn conventional value functions. We also prove convergence of fixed-horizon temporal difference methods with linear and general function approximation. Taken together, our results establish fixed-horizon TD methods as a viable new way of avoiding the stability problems of the deadly triad.

scholarly journals A Tale of Two-Timescale Reinforcement Learning with the Tightest Finite-Time Bound

2020 ◽  
Vol 34 (04) ◽  
pp. 3701-3708
Author(s):  
Gal Dalal ◽  
Balazs Szorenyi ◽  
Gugan Thoppe
Keyword(s):  
Reinforcement Learning ◽  
Convergence Rate ◽  
Policy Evaluation ◽  
Finite Time ◽  
High Probability ◽  
Temporal Difference ◽  
Time Analysis ◽  
Difference Methods ◽  
Temporal Difference Methods ◽  
Two Timescale Stochastic Approximation

Policy evaluation in reinforcement learning is often conducted using two-timescale stochastic approximation, which results in various gradient temporal difference methods such as GTD(0), GTD2, and TDC. Here, we provide convergence rate bounds for this suite of algorithms. Algorithms such as these have two iterates, θn and wn, which are updated using two distinct stepsize sequences, αn and βn, respectively. Assuming αn = n−α and βn = n−β with 1 > α > β > 0, we show that, with high probability, the two iterates converge to their respective solutions θ* and w* at rates given by ∥θn - θ*∥ = Õ(n−α/2) and ∥wn - w*∥ = Õ(n−β/2); here, Õ hides logarithmic terms. Via comparable lower bounds, we show that these bounds are, in fact, tight. To the best of our knowledge, ours is the first finite-time analysis which achieves these rates. While it was known that the two timescale components decouple asymptotically, our results depict this phenomenon more explicitly by showing that it in fact happens from some finite time onwards. Lastly, compared to existing works, our result applies to a broader family of stepsizes, including non-square summable ones.

scholarly journals Temporal Uncertainty During Overshadowing

2011 ◽  
pp. 46-55
Author(s):  
Dómhnall J. Jennings ◽  
Eduardo Alonso ◽  
Esther Mondragón ◽  
Charlotte Bonardi
Keyword(s):  
Machine Learning ◽  
Reinforcement Learning ◽  
Associative Learning ◽  
Learning Theories ◽  
Temporal Difference ◽  
Temporal Uncertainty ◽  
Learning Problem ◽  
Difference Model ◽  
Distribution Form ◽  
Temporal Properties

Standard associative learning theories typically fail to conceptualise the temporal properties of a stimulus, and hence cannot easily make predictions about the effects such properties might have on the magnitude of conditioning phenomena. Despite this, in intuitive terms we might expect that the temporal properties of a stimulus that is paired with some outcome to be important. In particular, there is no previous research addressing the way that fixed or variable duration stimuli can affect overshadowing. In this chapter we report results which show that the degree of overshadowing depends on the distribution form - fixed or variable - of the overshadowing stimulus, and argue that conditioning is weaker under conditions of temporal uncertainty. These results are discussed in terms of models of conditioning and timing. We conclude that the temporal difference model, which has been extensively applied to the reinforcement learning problem in machine learning, accounts for the key findings of our study.

Sign in / Sign up

Export Citation Format

Share Document