Inter-Task Similarity for Lifelong Reinforcement Learning in Heterogeneous Tasks

Reinforcement learning (RL) is a learning paradigm in which an agent interacts with the environment it inhabits to learn in a trial-and-error way. By letting the agent acquire knowledge from its own experience, RL has been successfully applied to complex domains such as robotics. However, for non-trivial problems, training an RL agent can take very long periods of time. Lifelong machine learning (LML) is a learning setting in which the agent learns to solve tasks sequentially, by leveraging knowledge accumulated from previously solved tasks to learn better/faster in a new one. Most LML works heavily rely on the assumption that tasks are similar to each other. However, this may not be true for some domains with a high degree of task-diversity that could benefit from adopting a lifelong learning approach, e.g., service robotics. Therefore, in this research we will address the problem of learning to solve a sequence of RL heterogeneous tasks (i.e., tasks that differ in their state-action space).

Learning Rates Are Not All the Same: The Interpretation of Computational Model Parameters Depends on the Context

Reinforcement Learning (RL) has revolutionized the cognitive and brain sciences, explaining behavior from simple conditioning to problem solving, across the life span, and anchored in brain function. However, discrepancies in results are increasingly apparent between studies, particularly in the developmental literature. To better understand these, we investigated to which extent parameters generalize between tasks and models, and capture specific and uniquely interpretable (neuro)cognitive processes. 291 participants aged 8-30 years completed three learning tasks in a single session, and were fitted using state-of-the-art RL models. RL decision noise/exploration parameters generalized well between tasks, decreasing between ages 8-17. Learning rates for negative feedback did not generalize, and learning rates for positive feedback showed intermediate generalizability, dependent on task similarity. These findings can explain discrepancies in the existing literature. Future research therefore needs to carefully consider task characteristics when relating findings across studies, and develop strategies to computationally model how context impacts behavior.

The role of task similarity for ego depletion: a registered report

Ego depletion refers to decrements in self-control performance resulting from prior use of self-control. The ego depletion effect has received much research attention, but the more recent literature reports small or null effects. This registered report examined the moderating effect of task similarity on the ego depletion effect. We predicted a crossover interaction between type of primary and secondary task such that engaging in a demanding self-control task should lead to better performance in the secondary task when it is similar to the primary task (facilitation effect) but worse performance when it is dissimilar (ego depletion). In a preregistered pilot study, N = 80 participants first completed either a visual stop-signal task (SST) or a simple lexical categorization task. They proceeded with one of four tasks classified as increasingly dissimilar based on their underlying operations and executive functions: (1) auditory stop-signal task, (2) Stroop task, (3) Eriksen flanker task, and (4) unsolvable anagrams. Both the pilot study and a high-powered registered replication (N = 300) revealed the predicted interaction effect. However, evidence for facilitation from similar tasks was stronger than evidence for depletion from dissimilar tasks. Together, these findings highlight the important role of task similarity for the study of ego depletion and related phenomena.