Learning compositional sequences with multiple time scales through a hierarchical network of spiking neurons

Sequential behaviour is often compositional and organised across multiple time scales: a set of individual elements developing on short time scales (motifs) are combined to form longer functional sequences (syntax). Such organisation leads to a natural hierarchy that can be used advantageously for learning, since the motifs and the syntax can be acquired independently. Despite mounting experimental evidence for hierarchical structures in neuroscience, models for temporal learning based on neuronal networks have mostly focused on serial methods. Here, we introduce a network model of spiking neurons with a hierarchical organisation aimed at sequence learning on multiple time scales. Using biophysically motivated neuron dynamics and local plasticity rules, the model can learn motifs and syntax independently. Furthermore, the model can relearn sequences efficiently and store multiple sequences. Compared to serial learning, the hierarchical model displays faster learning, more flexible relearning, increased capacity, and higher robustness to perturbations. The hierarchical model redistributes the variability: it achieves high motif fidelity at the cost of higher variability in the between-motif timings.

Download Full-text

Learning compositional sequences with multiple time scales through a hierarchical network of spiking neurons

10.1101/2020.09.08.287748 ◽

2020 ◽

Author(s):

Amadeus Maes ◽

Mauricio Barahona ◽

Claudia Clopath

Keyword(s):

Time Scales ◽

Hierarchical Model ◽

Hierarchical Structures ◽

Spiking Neurons ◽

Multiple Time Scales ◽

Multiple Time ◽

The Cost ◽

Multiple Sequences ◽

Short Time ◽

Temporal Learning

ABSTRACTSequential behaviour is often compositional and organised across multiple time scales: a set of individual elements developing on short time scales (motifs) are combined to form longer functional sequences (syntax). Such organisation leads to a natural hierarchy that can be used advantageously for learning, since the motifs and the syntax can be acquired independently. Despite mounting experimental evidence for hierarchical structures in neuroscience, models for temporal learning based on neuronal networks have mostly focused on serial methods. Here, we introduce a network model of spiking neurons with a hierarchical organisation aimed at sequence learning on multiple time scales. Using biophysically motivated neuron dynamics and local plasticity rules, the model can learn motifs and syntax independently. Furthermore, the model can relearn sequences efficiently and store multiple sequences. Compared to serial learning, the hierarchical model displays faster learning, more flexible relearning, increased capacity, and higher robustness to perturbations. The hierarchical model redistributes the variability: it achieves high motif fidelity at the cost of higher variability in the between-motif timings.

Download Full-text

Meta-control of the exploration-exploitation dilemma emerges from probabilistic inference over a hierarchy of time scales

10.1101/847566 ◽

2019 ◽

Author(s):

Dimitrije Marković ◽

Thomas Goschke ◽

Stefan J. Kiebel

Keyword(s):

Cognitive Control ◽

Time Scales ◽

Probabilistic Inference ◽

Multiple Time Scales ◽

Multiple Time ◽

Dynamic Task ◽

Wide Range ◽

Hidden States ◽

Short Time ◽

Exploration Exploitation

AbstractCognitive control is typically understood as a set of mechanisms which enable humans to reach goals that require integrating the consequences of actions over longer time scales. Importantly, using routine beheavior or making choices beneficial only at a short time scales would prevent one from attaining these goals. During the past two decades, researchers have proposed various computational cognitive models that successfully account for behaviour related to cognitive control in a wide range of laboratory tasks. As humans operate in a dynamic and uncertain environment, making elaborate plans and integrating experience over multiple time scales is computationally expensive, the specific question of how uncertain consequences at different time scales are integrated into adaptive decisions remains poorly understood. Here, we propose that precisely the problem of integrating experience and forming elaborate plans over multiple time scales is a key component for better understanding how human agents solve cognitive control dilemmas such as the exploration-exploitation dilemma. In support of this conjecture, we present a computational model of probabilistic inference over hidden states and actions, which are represented as a hierarchy of time scales. Simulations of goal-reaching agents instantiating the model in an uncertain and dynamic task environment show how the exploration-exploitation dilemma may be solved by inferring meta-control states which adapt behaviour to changing contexts.

Download Full-text

Deep Reinforcement Learning with Hierarchical Structures

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2021/681 ◽

2021 ◽

Author(s):

Siyuan Li

Keyword(s):

Reinforcement Learning ◽

Time Scales ◽

Hierarchical Structures ◽

Multiple Time Scales ◽

Multiple Time ◽

Top Down ◽

Bottom Up ◽

Hierarchical Reinforcement Learning ◽

Future Work

Hierarchical reinforcement learning (HRL), which enables control at multiple time scales, is a promising paradigm to solve challenging and long-horizon tasks. In this paper, we briefly introduce our work in bottom-up and top-down HRL and outline the directions for future work.

Download Full-text

Meta-control of the exploration-exploitation dilemma emerges from probabilistic inference over a hierarchy of time scales

Cognitive Affective & Behavioral Neuroscience ◽

10.3758/s13415-020-00837-x ◽

2020 ◽

Author(s):

Dimitrije Marković ◽

Thomas Goschke ◽

Stefan J. Kiebel

Keyword(s):

Cognitive Control ◽

Time Scales ◽

Multiple Time Scales ◽

Multiple Time ◽

Single Action ◽

Wide Range ◽

Control Concept ◽

Novel Concept ◽

Short Time ◽

Exploration Exploitation

AbstractCognitive control is typically understood as a set of mechanisms that enable humans to reach goals that require integrating the consequences of actions over longer time scales. Importantly, using routine behaviour or making choices beneficial only at short time scales would prevent one from attaining these goals. During the past two decades, researchers have proposed various computational cognitive models that successfully account for behaviour related to cognitive control in a wide range of laboratory tasks. As humans operate in a dynamic and uncertain environment, making elaborate plans and integrating experience over multiple time scales is computationally expensive. Importantly, it remains poorly understood how uncertain consequences at different time scales are integrated into adaptive decisions. Here, we pursue the idea that cognitive control can be cast as active inference over a hierarchy of time scales, where inference, i.e., planning, at higher levels of the hierarchy controls inference at lower levels. We introduce the novel concept of meta-control states, which link higher-level beliefs with lower-level policy inference. Specifically, we conceptualize cognitive control as inference over these meta-control states, where solutions to cognitive control dilemmas emerge through surprisal minimisation at different hierarchy levels. We illustrate this concept using the exploration-exploitation dilemma based on a variant of a restless multi-armed bandit task. We demonstrate that beliefs about contexts and meta-control states at a higher level dynamically modulate the balance of exploration and exploitation at the lower level of a single action. Finally, we discuss the generalisation of this meta-control concept to other control dilemmas.

Download Full-text

SOME REMARKS ON THE MULTIPLE TIME SCALES OF DIFFUSION IN MINERALS AND MELTS

10.1130/abs/2018am-323387 ◽

2018 ◽

Author(s):

Yan Liang ◽

◽

Daniele J. Cherniak ◽

Chenguang Sun

Keyword(s):

Time Scales ◽

Multiple Time Scales ◽

Multiple Time

Download Full-text

Robust optimal scheduling of integrated energy systems based on hybrid energy storage and multiple time scales

2020 IEEE Sustainable Power and Energy Conference (iSPEC) ◽

10.1109/ispec50848.2020.9351230 ◽

2020 ◽

Author(s):

Mingfang Lu ◽

Xianshan Li ◽

Fei Li

Keyword(s):

Energy Storage ◽

Time Scales ◽

Energy Systems ◽

Optimal Scheduling ◽

Multiple Time Scales ◽

Multiple Time ◽

Hybrid Energy ◽

Hybrid Energy Storage ◽

Integrated Energy Systems

Download Full-text

Are We Doing ‘Systems’ Research? An Assessment of Methods for Climate Change Adaptation to Hydrohazards in a Complex World

Sustainability ◽

10.3390/su11041163 ◽

2019 ◽

Vol 11 (4) ◽

pp. 1163 ◽

Cited By ~ 4

Author(s):

Melissa Bedinger ◽

Lindsay Beevers ◽

Lila Collet ◽

Annie Visser

Keyword(s):

Climate Change ◽

Human Nature ◽

Climate Change Adaptation ◽

Time Scales ◽

Spatial Scales ◽

Local Context ◽

Multiple Time Scales ◽

Future Research ◽

Multiple Time ◽

Systems Research

Climate change is a product of the Anthropocene, and the human–nature system in which we live. Effective climate change adaptation requires that we acknowledge this complexity. Theoretical literature on sustainability transitions has highlighted this and called for deeper acknowledgment of systems complexity in our research practices. Are we heeding these calls for ‘systems’ research? We used hydrohazards (floods and droughts) as an example research area to explore this question. We first distilled existing challenges for complex human–nature systems into six central concepts: Uncertainty, multiple spatial scales, multiple time scales, multimethod approaches, human–nature dimensions, and interactions. We then performed a systematic assessment of 737 articles to examine patterns in what methods are used and how these cover the complexity concepts. In general, results showed that many papers do not reference any of the complexity concepts, and no existing approach addresses all six. We used the detailed results to guide advancement from theoretical calls for action to specific next steps. Future research priorities include the development of methods for consideration of multiple hazards; for the study of interactions, particularly in linking the short- to medium-term time scales; to reduce data-intensivity; and to better integrate bottom–up and top–down approaches in a way that connects local context with higher-level decision-making. Overall this paper serves to build a shared conceptualisation of human–nature system complexity, map current practice, and navigate a complexity-smart trajectory for future research.

Download Full-text