Learning at Variable Attentional Load Requires Cooperation of Working Memory, Meta-learning and Attention-augmented Reinforcement Learning

AbstractFlexible learning of changing reward contingencies can be realized with different strategies. A fast learning strategy involves using working memory of recently rewarded objects to guide choices. A slower learning strategy uses prediction errors to gradually update value expectations to improve choices. How the fast and slow strategies work together in scenarios with real-world stimulus complexity is not well known. Here, we disentangle their relative contributions in rhesus monkeys while they learned the relevance of object features at variable attentional load. We found that learning behavior across six subjects is consistently best predicted with a model combining (i) fast working memory (ii) slower reinforcement learning from differently weighted positive and negative prediction errors, as well as (iii) selective suppression of non-chosen feature values and (iv) a meta-learning mechanism adjusting exploration rates based on a memory trace of recent errors. These mechanisms cooperate differently at low and high attentional loads. While working memory was essential for efficient learning at lower attentional loads, enhanced weighting of negative prediction errors and meta-learning were essential for efficient learning at higher attentional loads. Together, these findings pinpoint a canonical set of learning mechanisms and demonstrate how they cooperate when subjects flexibly adjust to environments with variable real-world attentional demands.Significance statementLearning which visual features are relevant for achieving our goals is challenging in real-world scenarios with multiple distracting features and feature dimensions. It is known that in such scenarios learning benefits significantly from attentional prioritization. Here we show that beyond attention, flexible learning uses a working memory system, a separate learning gain for avoiding negative outcomes, and a meta-learning process that adaptively increases exploration rates whenever errors accumulate. These subcomponent processes of cognitive flexibility depend on distinct learning signals that operate at varying timescales, including the most recent reward outcome (for working memory), memories of recent outcomes (for adjusting exploration), and reward prediction errors (for attention augmented reinforcement learning). These results illustrate the specific mechanisms that cooperate during cognitive flexibility.

Download Full-text

The Tortoise and the Hare: Interactions between Reinforcement Learning and Working Memory

Journal of Cognitive Neuroscience ◽

10.1162/jocn_a_01238 ◽

2018 ◽

Vol 30 (10) ◽

pp. 1422-1432 ◽

Cited By ~ 16

Author(s):

Anne G. E. Collins

Keyword(s):

Working Memory ◽

Reinforcement Learning ◽

Computational Modeling ◽

Behavioral Experiment ◽

Prediction Errors ◽

Multiple Systems ◽

Memory Interference ◽

Neural Representations ◽

The Cost

Learning to make rewarding choices in response to stimuli depends on a slow but steady process, reinforcement learning, and a fast and flexible, but capacity-limited process, working memory. Using both systems in parallel, with their contributions weighted based on performance, should allow us to leverage the best of each system: rapid early learning, supplemented by long-term robust acquisition. However, this assumes that using one process does not interfere with the other. We use computational modeling to investigate the interactions between the two processes in a behavioral experiment and show that working memory interferes with reinforcement learning. Previous research showed that neural representations of reward prediction errors, a key marker of reinforcement learning, were blunted when working memory was used for learning. We thus predicted that arbitrating in favor of working memory to learn faster in simple problems would weaken the reinforcement learning process. We tested this by measuring performance in a delayed testing phase where the use of working memory was impossible, and thus participant choices depended on reinforcement learning. Counterintuitively, but confirming our predictions, we observed that associations learned most easily were retained worse than associations learned slower: Using working memory to learn quickly came at the cost of long-term retention. Computational modeling confirmed that this could only be accounted for by working memory interference in reinforcement learning computations. These results further our understanding of how multiple systems contribute in parallel to human learning and may have important applications for education and computational psychiatry.

Download Full-text

Theoretical and Empirical Analysis of a Spatial EA Parallel Boosting Algorithm

Evolutionary Computation ◽

10.1162/evco_a_00202 ◽

2018 ◽

Vol 26 (1) ◽

pp. 43-66 ◽

Cited By ~ 1

Author(s):

Uday Kamath ◽

Carlotta Domeniconi ◽

Kenneth De Jong

Keyword(s):

Real World ◽

Learning Algorithm ◽

Learning Algorithms ◽

Real World Data ◽

Meta Level ◽

Meta Learning ◽

Robustness To Noise ◽

Boosting Algorithm ◽

Efficient Learning ◽

Empirical Analyses

Many real-world problems involve massive amounts of data. Under these circumstances learning algorithms often become prohibitively expensive, making scalability a pressing issue to be addressed. A common approach is to perform sampling to reduce the size of the dataset and enable efficient learning. Alternatively, one customizes learning algorithms to achieve scalability. In either case, the key challenge is to obtain algorithmic efficiency without compromising the quality of the results. In this article we discuss a meta-learning algorithm (PSBML) that combines concepts from spatially structured evolutionary algorithms (SSEAs) with concepts from ensemble and boosting methodologies to achieve the desired scalability property. We present both theoretical and empirical analyses which show that PSBML preserves a critical property of boosting, specifically, convergence to a distribution centered around the margin. We then present additional empirical analyses showing that this meta-level algorithm provides a general and effective framework that can be used in combination with a variety of learning classifiers. We perform extensive experiments to investigate the trade-off achieved between scalability and accuracy, and robustness to noise, on both synthetic and real-world data. These empirical results corroborate our theoretical analysis, and demonstrate the potential of PSBML in achieving scalability without sacrificing accuracy.

Download Full-text

Asymmetric learning facilitates human inference of transitive relations

10.1101/2021.04.03.437766 ◽

2021 ◽

Author(s):

Simon Ciranka ◽

Juan Linde-Domingo ◽

Ivan Padezhki ◽

Clara Wicharz ◽

Charley M Wu ◽

...

Keyword(s):

Reinforcement Learning ◽

Learning Strategy ◽

Cognitive Distortions ◽

Relational Learning ◽

Empirical Support ◽

Transitive Inference ◽

Learning Mechanisms ◽

Previous Evidence ◽

Asymmetric Learning ◽

Intense Research

Humans and other animals are capable of inferring never-experienced relations (e.g., A>C) from other relational observations (e.g., A>B and B>C). The processes behind such transitive inference are subject to intense research. Here, we demonstrate a new aspect of relational learning, building on previous evidence that transitive inference can be accomplished through simple reinforcement learning mechanisms. We show in simulations that inference of novel relations benefits from an asymmetric learning policy, where observers update only their belief about the winner (or loser) in a pair. Across 4 experiments (n=145), we find substantial empirical support for such asymmetries in inferential learning. The learning policy favoured by our simulations and experiments gives rise to a compression of values which is routinely observed in psychophysics and behavioural economics. In other words, a seemingly biased learning strategy that yields well-known cognitive distortions can be beneficial for transitive inferential judgments.

Download Full-text

Trace Imbalance in Reinforcement and Punishment Systems Can Mis-reinforce Implicit Choices Leading to Anxiety

10.1101/2020.08.07.241588 ◽

2020 ◽

Author(s):

Yuki Sakai ◽

Yutaka Sakai ◽

Yoshinari Abe ◽

Jin Narumoto ◽

Saori C. Tanaka

Keyword(s):

Reinforcement Learning ◽

Memory Trace ◽

Healthy Population ◽

Prediction Errors ◽

Obsessive Compulsive ◽

Compulsive Disorder ◽

Learning Framework ◽

Negative Prediction ◽

Neural Signature ◽

Healthy Participants

AbstractNobody wants to experience anxiety. However, anxiety may be induced by our own implicit choices that are mis-reinforced by some imbalance in reinforcement learning. Here we focused on obsessive-compulsive disorder (OCD) as a candidate for implicitly learned anxiety. Simulations in the reinforcement learning framework showed that agents implicitly learn to become anxious when the memory trace signal for past actions decays differently for positive and negative prediction errors. In empirical data, we confirmed that OCD patients showed extremely imbalanced traces, which were normalized by serotonin enhancers. We also used fMRI to identify the neural signature of OCD and healthy participants with imbalanced traces. Beyond the spectrum of clinical phenotypes, these behavioral and neural characteristics can be generalized to variations in the healthy population.

Download Full-text

The tortoise and the hare: interactions between reinforcement learning and working memory

10.1101/234724 ◽

2017 ◽

Author(s):

Anne G.E. Collins

Keyword(s):

Working Memory ◽

Reinforcement Learning ◽

Computational Modeling ◽

Behavioral Experiment ◽

Prediction Errors ◽

Multiple Systems ◽

Memory Interference ◽

Neural Representations ◽

The Cost

AbstractLearning to make rewarding choices in response to stimuli depends on a slow but steady process, reinforcement learning, and a fast and flexible, but capacity limited process, working memory. Using both systems in parallel, with their contributions weighted based on performance, should allow us to leverage the best of each system: rapid early learning, supplemented by long term robust acquisition. However, this assumes that using one process does not interfere with the other. We use computational modeling to investigate the interactions between the two processes in a behavioral experiment, and show that working memory interferes with reinforcement learning. Previous research showed that neural representations of reward prediction errors, a key marker of reinforcement learning, were blunted when working memory was used for learning. We thus predicted that arbitrating in favor of working memory to learn faster in simple problems would weaken the reinforcement learning process. We tested this by measuring performance in a delayed testing phase where the use of working memory was impossible, and thus subject choices depended on reinforcement learning. Counter-intuitively, but confirming our predictions, we observed that associations learned most easily were retained worse than associations learned slower: using working memory to learn quickly came at the cost of long-term retention. Computational modeling confirmed that this could only be accounted for by working memory interference in reinforcement learning computations. These results further our understanding of how multiple systems contribute in parallel to human learning, and may have important applications for education and computational psychiatry.

Download Full-text

Learning Unmanned Aerial Vehicle Control for Autonomous Target Following

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2018/685 ◽

2018 ◽

Cited By ~ 2

Author(s):

Siyi Li ◽

Tianbo Liu ◽

Chi Zhang ◽

Dit-Yan Yeung ◽

Shaojie Shen

Keyword(s):

Neural Network ◽

Reinforcement Learning ◽

Unmanned Aerial Vehicle ◽

Learning Process ◽

Real World ◽

Model Free ◽

The Neural Network ◽

Policy Gradient ◽

Aerial Vehicle ◽

Efficient Learning

While deep reinforcement learning (RL) methods have achieved unprecedented successes in a range of challenging problems, their applicability has been mainly limited to simulation or game domains due to the high sample complexity of the trial-and-error learning process. However, real-world robotic applications often need a data-efficient learning process with safety-critical constraints. In this paper, we consider the challenging problem of learning unmanned aerial vehicle (UAV) control for tracking a moving target. To acquire a strategy that combines perception and control, we represent the policy by a convolutional neural network. We develop a hierarchical approach that combines a model-free policy gradient method with a conventional feedback proportional-integral-derivative (PID) controller to enable stable learning without catastrophic failure. The neural network is trained by a combination of supervised learning from raw images and reinforcement learning from games of self-play. We show that the proposed approach can learn a target following policy in a simulator efficiently and the learned behavior can be successfully transferred to the DJI quadrotor platform for real-world UAV control.

Download Full-text

Valence biases factual and counterfactual learning in opposite directions

10.1101/090654 ◽

2016 ◽

Cited By ~ 1

Author(s):

Stefano Palminteri ◽

Germain Lefebvre ◽

Emma J. Kilford ◽

Sarah-Jayne Blakemore

Keyword(s):

Reinforcement Learning ◽

Computational Model ◽

Prediction Error ◽

Prediction Errors ◽

Learning Tasks ◽

Negative Prediction ◽

Account Information ◽

Factual Learning

AbstractPrevious studies suggest that factual learning, that is, learning from obtained outcomes, is biased, such that participants preferentially take into account positive, as compared to negative, prediction errors. However, whether or not the prediction error valence also affects counterfactual learning, that is, learning from forgone outcomes, is unknown. To address this question, we analysed the performance of two cohorts of participants on reinforcement learning tasks using a computational model that was adapted to test if prediction error valance influences learning. Concerning factual learning, we replicated previous findings of a valence-induced bias, whereby participants learned preferentially from positive, relative to negative, prediction errors. In contrast, for counterfactual learning, we found the opposite valence-induced bias: negative prediction errors were preferentially taken into account relative to positive ones. When considering valence-induced bias in the context of both factual and counterfactual learning, it appears that people tend to preferentially take into account information that confirms their current choice

Download Full-text