Exploring Cognition with Brain–Machine Interfaces

Traditional brain–machine interfaces decode cortical motor commands to control external devices. These commands are the product of higher-level cognitive processes, occurring across a network of brain areas, that integrate sensory information, plan upcoming motor actions, and monitor ongoing movements. We review cognitive signals recently discovered in the human posterior parietal cortex during neuroprosthetic clinical trials. These signals are consistent with small regions of cortex having a diverse role in cognitive aspects of movement control and body monitoring, including sensorimotor integration, planning, trajectory representation, somatosensation, action semantics, learning, and decision making. These variables are encoded within the same population of cells using structured representations that bind related sensory and motor variables, an architecture termed partially mixed selectivity. Diverse cognitive signals provide complementary information to traditional motor commands to enable more natural and intuitive control of external devices.

Multi-task representations in human cortex transform along a sensory-to-motor hierarchy

Human cognition recruits diverse neural processes, yet the organizing computational and functional architectures remain unclear. Here, we characterized the geometry and topography of multi-task representations across human cortex using functional MRI during 26 cognitive tasks in the same subjects. We measured the representational similarity across tasks within a region, and the alignment of representations between regions. We found a cortical topography of representational alignment following a hierarchical sensory-association-motor gradient, revealing compression-then-expansion of multi-task dimensionality along this gradient. To investigate computational principles of multi-task representations, we trained multi-layer neural network models to transform empirical visual to motor representations. Compression-then-expansion organization in models emerged exclusively in a training regime where internal representations are highly optimized for sensory-to-motor transformation, and not under generic signal propagation. This regime produces hierarchically structured representations similar to empirical cortical patterns. Together, these results reveal computational principles that organize multi-task representations across human cortex to support flexible cognition.

Automatic Melody Harmonization via Reinforcement Learning by Exploring Structured Representations for Melody Sequences

We present a novel reinforcement learning architecture that learns a structured representation for use in symbolic melody harmonization. Probabilistic models are predominant in melody harmonization tasks, most of which only treat melody notes as independent observations and do not take note of substructures in the melodic sequence. To fill this gap, we add substructure discovery as a crucial step in automatic chord generation. The proposed method consists of a structured representation module that generates hierarchical structures for the symbolic melodies, a policy module that learns to break a melody into segments (whose boundaries concur with chord changes) and phrases (the subunits in segments), and a harmonization module that generates chord sequences for each segment. We formulate the structure discovery process as a sequential decision problem with a policy gradient RL method selecting the boundary of each segment or phrase to obtain an optimized structure. We conduct experiments on our preprocessed HookTheory Lead Sheet Dataset, which has 17,979 melody/chord pairs. The results demonstrate that our proposed method can learn task-specific representations and, thus, yield competitive results compared with state-of-the-art baselines.

GM-MLIC: Graph Matching based Multi-Label Image Classification

Multi-Label Image Classification (MLIC) aims to predict a set of labels that present in an image. The key to deal with such problem is to mine the associations between image contents and labels, and further obtain the correct assignments between images and their labels. In this paper, we treat each image as a bag of instances, and reformulate the task of MLIC as a instance-label matching selection problem. To model such problem, we propose a novel deep learning framework named Graph Matching based Multi-Label Image Classification (GM-MLIC), where Graph Matching (GM) scheme is introduced owing to its excellent capability of excavating the instance and label relationship. Specifically, we first construct an instance spatial graph and a label semantic graph respectively, and then incorporate them into a constructed assignment graph by connecting each instance to all labels. Subsequently, the graph network block is adopted to aggregate and update all nodes and edges state on the assignment graph to form structured representations for each instance and label. Our network finally derives a prediction score for each instance-label correspondence and optimizes such correspondence with a weighted cross-entropy loss. Extensive experiments conducted on various datasets demonstrate the superiority of our proposed method.

Balancing Flexibility and Interference in Working Memory

Working memory is central to cognition, flexibly holding the variety of thoughts needed for complex behavior. Yet, despite its importance, working memory has a severely limited capacity, holding only three to four items at once. In this article, I review experimental and computational evidence that the flexibility and limited capacity of working memory reflect the same underlying neural mechanism. I argue that working memory relies on interactions between high-dimensional, integrative representations in the prefrontal cortex and structured representations in the sensory cortex. Together, these interactions allow working memory to flexibly maintain arbitrary representations. However, the distributed nature of working memory comes at the cost of causing interference between items in memory, resulting in a limited capacity. Finally, I discuss several mechanisms used by the brain to reduce interference and maximize the effective capacity of working memory. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Rich and lazy learning of task representations in brains and neural networks

How do neural populations code for multiple, potentially conflicting tasks? Here, we used computational simulations involving neural networks to define “lazy” and “rich” coding solutions to this multitasking problem, which trade off learning speed for robustness. During lazy learning the input dimensionality is expanded by random projections to the network hidden layer, whereas in rich learning hidden units acquire structured representations that privilege relevant over irrelevant features. For context-dependent decision-making, one rich solution is to project task representations onto low-dimensional and orthogonal manifolds. Using behavioural testing and neuroimaging in humans, and analysis of neural signals from macaque prefrontal cortex, we report evidence for neural coding patterns in biological brains whose dimensionality and neural geometry are consistent with the rich learning regime.

On the nature of the lexicon: The status of rich lexical meanings

Both in linguistics and in psycholinguistics there is some debate about how rich or thin lexico-semantic representations are. Traditionally, in formal semantics but also in philosophy of language as well as in cognitive pragmatics, lexical meanings have been thought to be simple stable denotations or functions. In this paper, we present and discuss a number of interpretational phenomena of which the analysis proposed in the literature makes crucial use of rich meanings. The phenomena in question are cases where the assignment of truth-conditional contents to utterances seems to follow rules that do not operate on simple stable denotations or any other kind of ‘thin’ meanings but where composition takes rich structured representations as input. We also discuss problems for such accounts, which are mostly based on the inability of extant rich meanings accounts to explain many other interpretational phenomena, and we discuss the solutions that have been proposed to solve them. Furthermore, we address the discussion whether the informationally rich meanings are part of semantics, and more specifically part of the lexicon, or whether this information should be ascribed to more general world knowledge.

Plant Blindness

“Plant blindness” is the phrase introduced in an influential 1999 publication by James Wandersee and Elisabeth Schussler in connection with zoocentrism, initially in the context of biological education in the United States, but later addressed by researchers in a diversity of cultures. Wandersee and Schussler were much influenced by the psychology of perception and how it appeared to account for a general insensitivity to plants in the environment and dwindling understanding of the fundamental importance of plants for human survival and global ecology. The roots of plant blindness have been intensively analysed. Some studies conclude that it is an intrinsic trait, hardwired into human physiology and psychology. Others point to the consequences of historic trends in industrialization and urbanization and the progressive disconnection of people from the natural environment and primary sources of food, feed, fiber, and fuel. Much of the plant blindness literature confronts the need to remedy what it terms a specific condition, particularly at a time of climate and biodiversity crisis. Perhaps one of the challenges in this work is that those seeking to counteract plant blindness through education are often scientists or science educators who frequently perceive plant blindness as an ontological condition, which can be overcome by scientifically structured representations of plants using controlled vocabularies. But for those outside these communities plants are part of a worldview that is far more epistemological and thus the way plants enter, or fail to enter, an individual’s consciousness is constructed as a sociological event related to culture, experience, and environment. Understanding this is crucial if communicators and educators are to engage with the complexity of plant blindness effectively.