scholarly journals Contrasting temporal difference and opportunity cost reinforcement learning in an empirical money-emergence paradigm

2018 ◽  
Vol 115 (49) ◽  
pp. E11446-E11454 ◽  
Author(s):  
Germain Lefebvre ◽  
Aurélien Nioche ◽  
Sacha Bourgeois-Gironde ◽  
Stefano Palminteri
Keyword(s):  
Reinforcement Learning ◽  
Human Subjects ◽  
Optimal Strategies ◽  
Theoretic Approach ◽  
Monetary Search ◽  
Commodity Money ◽  
Theoretical Predictions ◽  
Monetary Equilibrium ◽  
Reinforcement Learning Models ◽  
Classical Economic

Money is a fundamental and ubiquitous institution in modern economies. However, the question of its emergence remains a central one for economists. The monetary search-theoretic approach studies the conditions under which commodity money emerges as a solution to override frictions inherent to interindividual exchanges in a decentralized economy. Although among these conditions, agents’ rationality is classically essential and a prerequisite to any theoretical monetary equilibrium, human subjects often fail to adopt optimal strategies in tasks implementing a search-theoretic paradigm when these strategies are speculative, i.e., involve the use of a costly medium of exchange to increase the probability of subsequent and successful trades. In the present work, we hypothesize that implementing such speculative behaviors relies on reinforcement learning instead of lifetime utility calculations, as supposed by classical economic theory. To test this hypothesis, we operationalized the Kiyotaki and Wright paradigm of money emergence in a multistep exchange task and fitted behavioral data regarding human subjects performing this task with two reinforcement learning models. Each of them implements a distinct cognitive hypothesis regarding the weight of future or counterfactual rewards in current decisions. We found that both models outperformed theoretical predictions about subjects’ behaviors regarding the implementation of speculative strategies and that the latter relies on the degree of the opportunity costs consideration in the learning process. Speculating about the marketability advantage of money thus seems to depend on mental simulations of counterfactual events that agents are performing in exchange situations.

Bayes factors for reinforcement-learning models of the Iowa gambling task.

Decision ◽  
2016 ◽  
Vol 3 (2) ◽  
pp. 115-131 ◽  
Author(s):  
Helen Steingroever ◽  
Ruud Wetzels ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Reinforcement Learning ◽  
Iowa Gambling Task ◽  
Bayes Factors ◽  
Gambling Task ◽  
Learning Models ◽  
Reinforcement Learning Models

scholarly journals Individual differences in experienced and observational decision-making illuminate interactions between reinforcement learning and declarative memory

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Batel Yifrah ◽  
Ayelet Ramaty ◽  
Genela Morris ◽  
Avi Mendelsohn
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Declarative Memory ◽  
Contextual Information ◽  
Memory Performance ◽  
Relevant Information ◽  
Subjective Memory ◽  
Types Of Information ◽  
Reinforcement Learning Models ◽  
Implicit And Explicit

AbstractDecision making can be shaped both by trial-and-error experiences and by memory of unique contextual information. Moreover, these types of information can be acquired either by means of active experience or by observing others behave in similar situations. The interactions between reinforcement learning parameters that inform decision updating and memory formation of declarative information in experienced and observational learning settings are, however, unknown. In the current study, participants took part in a probabilistic decision-making task involving situations that either yielded similar outcomes to those of an observed player or opposed them. By fitting alternative reinforcement learning models to each subject, we discerned participants who learned similarly from experience and observation from those who assigned different weights to learning signals from these two sources. Participants who assigned different weights to their own experience versus those of others displayed enhanced memory performance as well as subjective memory strength for episodes involving significant reward prospects. Conversely, memory performance of participants who did not prioritize their own experience over others did not seem to be influenced by reinforcement learning parameters. These findings demonstrate that interactions between implicit and explicit learning systems depend on the means by which individuals weigh relevant information conveyed via experience and observation.

scholarly journals Test-retest reliability of canonical reinforcement learning models

2019 ◽  
Author(s):  
Laura Weidinger ◽  
Andrea Gradassi ◽  
Lucas Molleman ◽  
Wouter van den Bos
Keyword(s):  
Reinforcement Learning ◽  
Learning Models ◽  
Retest Reliability ◽  
Reinforcement Learning Models ◽  
Test Retest Reliability

scholarly journals The rational use of causal inference to guide reinforcement learning strengthens with age

2019 ◽  
Author(s):  
Alexandra O. Cohen ◽  
Kate Nussenbaum ◽  
Hayley Dorfman ◽  
Samuel J. Gershman ◽  
Catherine A. Hartley
Keyword(s):  
Reinforcement Learning ◽  
Causal Structure ◽  
Learning Task ◽  
Negative Events ◽  
Shape Learning ◽  
Adolescents And Adults ◽  
Bayesian Reinforcement Learning ◽  
External Causes ◽  
Reinforcement Learning Models ◽  
Best Fit

Beliefs about the controllability of positive or negative events in the environment can shape learning throughout the lifespan. Previous research has shown that adults’ learning is modulated by beliefs about the causal structure of the environment such that they will update their value estimates to a lesser extent when the outcomes can be attributed to hidden causes. The present study examined whether external causes similarly influenced outcome attributions and learning across development. Ninety participants, ages 7 to 25 years, completed a reinforcement learning task in which they chose between two options with fixed reward probabilities. Choices were made in three distinct environments in which different hidden agents occasionally intervened to generate positive, negative, or random outcomes. Participants’ beliefs about hidden-agent intervention aligned well with the true probabilities of positive, negative, or random outcome manipulation in each of the three environments. Computational modeling of the learning data revealed that while the choices made by both adults (ages 18 - 25) and adolescents (ages 13 - 17) were best fit by Bayesian reinforcement learning models that incorporate beliefs about hidden agent intervention, those of children (ages 7 - 12) were best fit by a one learning rate model that updates value estimates based on choice outcomes alone. Together, these results suggest that while children demonstrate explicit awareness of the causal structure of the task environment they do not implicitly use beliefs about the causal structure of the environment to guide reinforcement learning in the same manner as adolescents and adults.

Off-policy integral reinforcement learning algorithm in dealing with nonzero sum game for nonlinear distributed parameter systems

2020 ◽  
Vol 42 (15) ◽  
pp. 2919-2928
Author(s):  
He Ren ◽  
Jing Dai ◽  
Huaguang Zhang ◽  
Kun Zhang
Keyword(s):  
Differential Equation ◽  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Control Strategies ◽  
Iteration Process ◽  
Optimal Strategies ◽  
Distributed Parameter Systems ◽  
Adaptive Dynamic Programming ◽  
Distributed Parameter ◽  
Value Functions

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

scholarly journals A Game Theoretic Approach to Swarm Robotics

2006 ◽  
Vol 3 (3) ◽  
pp. 131-142 ◽  
Author(s):  
S. N. Givigi ◽  
H. M. Schwartz
Keyword(s):  
Game Theory ◽  
Reinforcement Learning ◽  
Personality Traits ◽  
Evolutionary Psychology ◽  
Learning Process ◽  
Swarm Robotics ◽  
Theoretic Approach ◽  
Intelligent Robots ◽  
Game Theoretic ◽  
Game Theoretic Approach

In this article, we discuss some techniques for achieving swarm intelligent robots through the use of traits of personality. Traits of personality are characteristics of each robot that, altogether, define the robot's behaviours. We discuss the use of evolutionary psychology to select a set of traits of personality that will evolve due to a learning process based on reinforcement learning. The use of Game Theory is introduced, and some simulations showing its potential are reported.

scholarly journals ERLP: Ensembles of Reinforcement Learning Policies (Student Abstract)

2020 ◽  
Vol 34 (10) ◽  
pp. 13905-13906
Author(s):  
Rohan Saphal ◽  
Balaraman Ravindran ◽  
Dheevatsa Mudigere ◽  
Sasikanth Avancha ◽  
Bharat Kaul
Keyword(s):  
Reinforcement Learning ◽  
State Of The Art ◽  
Multiple Models ◽  
Model Parameters ◽  
Continuous Control ◽  
Sample Complexity ◽  
Local Minima ◽  
Single Model ◽  
Learning Policies ◽  
Reinforcement Learning Models

Reinforcement learning algorithms are sensitive to hyper-parameters and require tuning and tweaking for specific environments for improving performance. Ensembles of reinforcement learning models on the other hand are known to be much more robust and stable. However, training multiple models independently on an environment suffers from high sample complexity. We present here a methodology to create multiple models from a single training instance that can be used in an ensemble through directed perturbation of the model parameters at regular intervals. This allows training a single model that converges to several local minima during the optimization process as a result of the perturbation. By saving the model parameters at each such instance, we obtain multiple policies during training that are ensembled during evaluation. We evaluate our approach on challenging discrete and continuous control tasks and also discuss various ensembling strategies. Our framework is substantially sample efficient, computationally inexpensive and is seen to outperform state of the art (SOTA) approaches

Sign in / Sign up

Export Citation Format

Share Document