scholarly journals Entropy-based metrics for predicting choice behavior based on local response to reward

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Ethan Trepka ◽  
Mehran Spitmaan ◽  
Bilal A. Bari ◽  
Vincent D. Costa ◽  
Jeremiah Y. Cohen ◽  
...  
Keyword(s):  
Choice Behavior ◽  
Matching Law ◽  
Neural Mechanisms ◽  
Local Response ◽  
Choice Data ◽  
Dynamic Learning ◽  
Model Free ◽  
Learning Tasks ◽  
Adaptive Choice ◽  
Trial Basis

AbstractFor decades, behavioral scientists have used the matching law to quantify how animals distribute their choices between multiple options in response to reinforcement they receive. More recently, many reinforcement learning (RL) models have been developed to explain choice by integrating reward feedback over time. Despite reasonable success of RL models in capturing choice on a trial-by-trial basis, these models cannot capture variability in matching behavior. To address this, we developed metrics based on information theory and applied them to choice data from dynamic learning tasks in mice and monkeys. We found that a single entropy-based metric can explain 50% and 41% of variance in matching in mice and monkeys, respectively. We then used limitations of existing RL models in capturing entropy-based metrics to construct more accurate models of choice. Together, our entropy-based metrics provide a model-free tool to predict adaptive choice behavior and reveal underlying neural mechanisms.

scholarly journals Novel entropy-based metrics for predicting choice behavior based on local response to reward

2021 ◽  
Author(s):  
Ethan Trepka ◽  
Mehran Spitmaan ◽  
Bilal A Bari ◽  
Vincent D Costa ◽  
Jeremiah Y Cohen ◽  
...  
Keyword(s):  
Choice Behavior ◽  
Matching Law ◽  
Neural Mechanisms ◽  
Local Response ◽  
Choice Data ◽  
Dynamic Learning ◽  
Model Free ◽  
Learning Tasks ◽  
Adaptive Choice ◽  
Trial Basis

For decades, behavioral scientists have used the matching law to quantify how animals distribute their choices between multiple options in response to reinforcement they receive. More recently, many reinforcement learning (RL) models have been developed to explain choice by integrating reward feedback over time. Despite reasonable success of RL models in capturing choice on a trial-by-trial basis, these models cannot capture variability in matching. To address this, we developed novel metrics based on information theory and applied them to choice data from dynamic learning tasks in mice and monkeys. We found that a single entropy-based metric can explain 50% and 41% of variance in matching in mice and monkeys, respectively. We then used limitations of existing RL models in capturing entropy-based metrics to construct a more accurate model of choice. Together, our novel entropy-based metrics provide a powerful, model-free tool to predict adaptive choice behavior and reveal underlying neural mechanisms.

Neural mechanisms for model-free and model-based reinforcement strategies in humans performing a multi-step navigation task

2010 ◽  
Vol 68 ◽  
pp. e285-e286
Author(s):  
Alan Fermin ◽  
Takehiko Yoshida ◽  
Makoto Ito ◽  
Junichiro Yoshimoto ◽  
Kenji Doya
Keyword(s):  
Neural Mechanisms ◽  
Navigation Task ◽  
Model Based ◽  
Model Free ◽  
Reinforcement Strategies

From biophysics to cognition: reward-dependent adaptive choice behavior

2008 ◽  
Vol 18 (2) ◽  
pp. 209-216 ◽  
Author(s):  
Alireza Soltani ◽  
Xiao-Jing Wang
Keyword(s):  
Choice Behavior ◽  
Adaptive Choice

scholarly journals A simple model for learning in volatile environments

2019 ◽  
Author(s):  
Payam Piray ◽  
Nathaniel D. Daw
Keyword(s):  
Kalman Filter ◽  
State Space Model ◽  
Choice Data ◽  
Exact Inference ◽  
Second Order Statistics ◽  
Decision Neuroscience ◽  
Learning Tasks ◽  
Neuroscience Research ◽  
Proposed Model ◽  
Psychological Models

AbstractSound principles of statistical inference dictate that uncertainty shapes learning. In this work, we revisit the question of learning in volatile environments, in which both the first and second-order statistics of observations dynamically evolve over time. We propose a new model, the volatile Kalman filter (VKF), which is based on a tractable state-space model of uncertainty and extends the Kalman filter algorithm to volatile environments. The proposed model is algorithmically simple and encompasses the Kalman filter as a special case. Specifically, in addition to the error-correcting rule of Kalman filter for learning observations, the VKF learns volatility according to a second error-correcting rule. These dual updates echo and contextualize classical psychological models of learning, in particular hybrid accounts of Pearce-Hall and Rescorla-Wagner. At the computational level, compared with existing models, the VKF gives up some flexibility in the generative model to enable a more faithful approximation to exact inference. When fit to empirical data, the VKF is better behaved than alternatives and better captures human choice data in two independent datasets of probabilistic learning tasks. The proposed model provides a coherent account of learning in stable or volatile environments and has implications for decision neuroscience research.

scholarly journals Putting value in context: A role for context memory in decisions for reward

2015 ◽  
Author(s):  
Aaron M. Bornstein ◽  
Kenneth A. Norman
Keyword(s):  
Reinforcement Learning ◽  
Choice Behavior ◽  
Model Free ◽  
Episodic Memories ◽  
Made In ◽  
Context Memory

AbstractHow does experience inform decisions? In episodic sampling, decisions are guided by a few episodic memories of past choices. This process can yield choice patterns similar to model-free Reinforcement Learning (RL); however, samples can vary from trial to trial, causing decisions to vary. Here, we show that context retrieved during episodic sampling can cause choice behavior to deviate sharply from the predictions of RL. Specifically, we show that, when a given memory is sampled, choices (in the present) are influenced by the properties of other decisions made in the same context as the sampled event. This effect is mediated by fMRI measures of context retrieval on each trial, suggesting a mechanism whereby cues trigger retrieval of context, which then triggers retrieval of other decisions from that context. This result establishes a new avenue by which experience can guide choice, and as such has broad implications for the study of decisions.

scholarly journals An Analysis of Decision Under Risk in Rats

2018 ◽  
Author(s):  
Christine M. Constantinople ◽  
Alex T. Piet ◽  
Carlos D. Brody
Keyword(s):  
Choice Behavior ◽  
Neural Substrates ◽  
Neural Mechanisms ◽  
Reference Dependence ◽  
Decision Making Under Risk ◽  
Internal Reference ◽  
Decision Under Risk ◽  
Psychological Phenomena ◽  
Human Choice ◽  
Behavioral Economic Theory

AbstractProspect Theory is the predominant behavioral economic theory describing decision-making under risk. It accounts for near universal aspects of human choice behavior whose prevalence may reflect fundamental neural mechanisms. We now apply Prospect Theory’s framework to rodents, using a task in which rats chose between guaranteed and probabilistic rewards. Like humans, rats distorted probabilities and showed diminishing marginal sensitivity, in which they were less sensitive to differences in larger rewards. They exhibited reference dependence, in which the valence of outcomes (gain or loss) was determined by an internal reference point reflecting reward history. The similarities between rats and humans suggest conserved neural substrates, and enable application of powerful molecular/circuit tools to study mechanisms of psychological phenomena from behavioral economics.

scholarly journals Directed exploration in the Iowa Gambling Task: model-free and model-based analyses in a large dataset of young and old healthy participants

2018 ◽  
Author(s):  
Romain Ligneul
Keyword(s):  
Computational Models ◽  
Iowa Gambling Task ◽  
Large Population ◽  
Gambling Task ◽  
Choice Data ◽  
Parameter Recovery ◽  
Model Free ◽  
Multiple Metrics ◽  
User Friendly ◽  
Healthy Participants

AbstractThe Iowa Gambling Task (IGT) is one of the most common paradigms used to assess decision-making and executive functioning in neurological and psychiatric disorders. Several reinforcement-learning (RL) models were recently proposed to refine the qualitative and quantitative inferences that can be made about these processes based on IGT data. Yet, these models do not account for the complex exploratory patterns which characterize participants’ behavior in the task. Using a dataset of more than 500 subjects, we demonstrate the existence of such patterns and we describe a new computational architecture (Explore-Exploit) disentangling exploitation, random exploration and directed exploration in this large population of participants. The EE architecture provided a better fit to the choice data on multiple metrics. Parameter recovery and simulation analyses confirmed the superiority of the EE scheme over alternative schemes. Furthermore, using the EE model, we were able to replicate the reduction in directed exploration across lifespan, as previously reported in other paradigms. Finally, we provide a user-friendly toolbox enabling researchers to easily fit computational models on the IGT data, hence promoting reanalysis of the numerous datasets acquired in various populations of patients.

scholarly journals Dopamine Enhances Model-Based over Model-Free Choice Behavior

Neuron ◽  
2012 ◽  
Vol 75 (3) ◽  
pp. 418-424 ◽  
Author(s):  
Klaus Wunderlich ◽  
Peter Smittenaar ◽  
Raymond J. Dolan
Keyword(s):  
Choice Behavior ◽  
Free Choice ◽  
Model Based ◽  
Model Free
Sign in / Sign up

Export Citation Format

Share Document