Within- and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory

Learning from rewards and punishments is essential to survival and facilitates flexible human behavior. It is widely appreciated that multiple cognitive and reinforcement learning systems contribute to decision-making, but the nature of their interactions is elusive. Here, we leverage methods for extracting trial-by-trial indices of reinforcement learning (RL) and working memory (WM) in human electro-encephalography to reveal single-trial computations beyond that afforded by behavior alone. Neural dynamics confirmed that increases in neural expectation were predictive of reduced neural surprise in the following feedback period, supporting central tenets of RL models. Within- and cross-trial dynamics revealed a cooperative interplay between systems for learning, in which WM contributes expectations to guide RL, despite competition between systems during choice. Together, these results provide a deeper understanding of how multiple neural systems interact for learning and decision-making and facilitate analysis of their disruption in clinical populations.

Download Full-text

Within and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory

10.1101/184812 ◽

2017 ◽

Cited By ~ 4

Author(s):

Anne GE Collins ◽

Michael J Frank

Keyword(s):

Decision Making ◽

Working Memory ◽

Reinforcement Learning ◽

Human Behavior ◽

Learning Systems ◽

Neural Systems ◽

Single Trial ◽

Neural Signals ◽

Human Eeg ◽

Habit Learning

AbstractLearning from rewards and punishments is essential to survival, and facilitates flexible human behavior. It is widely appreciated that multiple cognitive and reinforcement learning systems contribute to behavior, but the nature of their interactions is elusive. Here, we leverage novel methods for extracting trial-by-trial indices of reinforcement learning (RL) and working memory (WM) in human electroencephalography to reveal single trial computations beyond that afforded by behavior alone. Within-trial dynamics confirmed that increases in neural expectation were predictive of reduced neural surprise in the following feedback period, supporting central tenets of RL models. Cross-trial dynamics revealed a cooperative interplay between systems for learning, in which WM contributes expectations to guide RL, despite competition between systems during choice. Together, these results provide a deeper understanding of how multiple neural systems interact for learning and decision making, and facilitate analysis of their disruption in clinical populations.One sentence summaryDecoding of dynamical neural signals in humans reveals cooperation between cognitive and habit learning systems.

Download Full-text

Linking Nonlinear Neural Dynamics to Single-Trial Human Behavior

Multiscale Analysis and Nonlinear Dynamics ◽

10.1002/9783527671632.ch09 ◽

2013 ◽

pp. 217-232

Author(s):

Michael X Cohen ◽

Bradley Voytek

Keyword(s):

Human Behavior ◽

Neural Dynamics ◽

Single Trial

Download Full-text

Coordination of adaptive working memory and reinforcement learning systems explaining choice and reaction time in a human experiment

BMC Neuroscience ◽

10.1186/1471-2202-15-s1-p156 ◽

2014 ◽

Vol 15 (S1) ◽

Author(s):

Guillaume D Viejo ◽

Mehdi Khamassi ◽

Andrea Brovelli ◽

Benoît Girard

Keyword(s):

Working Memory ◽

Reinforcement Learning ◽

Reaction Time ◽

Learning Systems

Download Full-text

Geometry of neural computation unifies working memory and planning

10.1101/2021.02.01.429156 ◽

2021 ◽

Author(s):

Daniel B. Ehrlich ◽

John D. Murray

Keyword(s):

Neural Networks ◽

Decision Making ◽

Working Memory ◽

Prefrontal Cortex ◽

Human Behavior ◽

Real World ◽

Recurrent Neural Networks ◽

Neural Computation ◽

Neural Data ◽

The Brain

Real-world tasks require coordination of working memory, decision making, and planning, yet these cognitive functions have disproportionately been studied as independent modular processes in the brain. Here we propose that contingency representations, defined as mappings for how future behaviors depend on upcoming events, can unify working memory and planning computations. We designed a task capable of disambiguating distinct types of representations. Our experiments revealed that human behavior is consistent with contingency representations, and not with traditional sensory models of working memory. In task-optimized recurrent neural networks we investigated possible circuit mechanisms for contingency representations and found that these representations can explain neurophysiological observations from prefrontal cortex during working memory tasks. Finally, we generated falsifiable predictions for neural data to identify contingency representations in neural data and to dissociate different models of working memory. Our findings characterize a neural representational strategy that can unify working memory, planning, and context-dependent decision making.

Download Full-text

Efficient reinforcement learning of a reservoir network model of parametric working memory achieved with a cluster population winner-take-all readout mechanism

Journal of Neurophysiology ◽

10.1152/jn.00378.2015 ◽

2015 ◽

Vol 114 (6) ◽

pp. 3296-3305 ◽

Cited By ~ 4

Author(s):

Zhenbo Cheng ◽

Zhidong Deng ◽

Xiaolin Hu ◽

Bo Zhang ◽

Tianming Yang

Keyword(s):

Decision Making ◽

Working Memory ◽

Prefrontal Cortex ◽

Reinforcement Learning ◽

Delay Period ◽

Information Storage ◽

Specific Information ◽

Stable States ◽

Winner Take All ◽

Take All

The brain often has to make decisions based on information stored in working memory, but the neural circuitry underlying working memory is not fully understood. Many theoretical efforts have been focused on modeling the persistent delay period activity in the prefrontal areas that is believed to represent working memory. Recent experiments reveal that the delay period activity in the prefrontal cortex is neither static nor homogeneous as previously assumed. Models based on reservoir networks have been proposed to model such a dynamical activity pattern. The connections between neurons within a reservoir are random and do not require explicit tuning. Information storage does not depend on the stable states of the network. However, it is not clear how the encoded information can be retrieved for decision making with a biologically realistic algorithm. We therefore built a reservoir-based neural network to model the neuronal responses of the prefrontal cortex in a somatosensory delayed discrimination task. We first illustrate that the neurons in the reservoir exhibit a heterogeneous and dynamical delay period activity observed in previous experiments. Then we show that a cluster population circuit decodes the information from the reservoir with a winner-take-all mechanism and contributes to the decision making. Finally, we show that the model achieves a good performance rapidly by shaping only the readout with reinforcement learning. Our model reproduces important features of previous behavior and neurophysiology data. We illustrate for the first time how task-specific information stored in a reservoir network can be retrieved with a biologically plausible reinforcement learning training scheme.

Download Full-text

Plucking a string or playing a G? Degree of choice abstraction impacts human reinforcement learning

10.1101/2021.08.25.457707 ◽

2021 ◽

Author(s):

Milena Rmus ◽

Amy Zou ◽

Anne G.E. Collins

Keyword(s):

Working Memory ◽

Reinforcement Learning ◽

Learning Process ◽

Information Integration ◽

Index Finger ◽

Learning Systems ◽

Response Type ◽

Stimulus Response ◽

Fast Learning ◽

Motor Actions

AbstractIn reinforcement learning (RL) experiments, participants learn to make rewarding choices in response to different stimuli; RL models use outcomes to estimate stimulus-response values which change incrementally. RL models consider any response type indiscriminately, ranging from less abstract choices (e.g. pressing a key with the index finger), to more abstract choices that can be executed in a number of ways (e.g. getting dinner at the restaurant). But does the learning process vary as a function of how abstract the choices are? In Experiment 1, we show that choice abstraction impacts learning: participants were slower and less accurate in learning to select a more abstract choice. Using computational modeling, we show that two mechanisms contribute to this. First, the values of motor actions interfered with the values of more abstract responses, resulting in more incorrect choices; second, information integration for relevant abstract choices was slower. In Experiment 2, we replicate the findings from Experiment 1, and further extend the results by investigating whether slowed learning is attributable to working memory (WM) or RL contributions. We find that the impairment in more abstract/flexible choices is driven primarily by a weaker contribution of WM. We conclude that defining a more abstract choice space used by multiple learning systems recruits limited executive resources, limiting how much such processes then contribute to fast learning.

Download Full-text

Context-dependent outcome encoding in human reinforcement learning

10.31234/osf.io/4qh2d ◽

2021 ◽

Author(s):

Stefano Palminteri ◽

Maël Lebreton

Keyword(s):

Decision Making ◽

Reinforcement Learning ◽

Human Behavior ◽

Context Dependence ◽

Economic Decision ◽

Economic Decision Making ◽

Subjective Value ◽

Objective Value ◽

Context Dependent ◽

Subjective Context

A wealth of evidence in perceptual and economic decision-making research suggests that the subjective value of one option is determined by other available options (i.e. the context). A series of studies provides evidence that the same coding principles apply to situations where decisions are shaped by past outcomes, i.e. in reinforcement-learning situations. In bandit tasks, human behavior is explained by models assuming that individuals do not learn the objective value of an outcome, but rather its subjective, context-dependent representation. We argue that, while such outcome context-dependence may be informationally or ecologically optimal, it concomitantly undermines the capacity to generalize value-based knowledge to new contexts – sometimes creating apparent decision paradoxes.

Download Full-text