Evidence for, and challenges to, sensory recruitment models of visual working memory

Visual working memory refers to the ability to temporarily hold information in mind in the service of behavior. Often, it is not sufficient to hold an abstract idea in mind to achieve our goals. Rather, we must maintain vivid sensory details. For example, when buying a spool of thread to repair a much-loved shirt, holding an abstract category in mind is not sufficient to buy the correct color (e.g. ‘blue’)—instead, you need a precise visual memory of the color (e.g., a particular gray-ish shade of blue). One proposal for how we maintain vivid, detailed information in mind is the sensory recruitment hypothesis. Sensory recruitment proposes that neural circuits already specialized for encoding sensory details during perception are likewise recruited to help maintain this information in working memory. In this review, we recount evidence that is consistent with a key role for early visual cortex in supporting visual working memory, we discuss key debates about the role of early sensory activity in supporting memory maintenance, and we outline a framework in which sensory codes are one part of a flexible, multi-level working memory representation.

Why brain-controlled neuroprosthetics matter: mechanisms underlying electrical stimulation of muscles and nerves in rehabilitation

Delivering short trains of electric pulses to the muscles and nerves can elicit action potentials resulting in muscle contractions. When the stimulations are sequenced to generate functional movements, such as grasping or walking, the application is referred to as functional electrical stimulation (FES). Implications of the motor and sensory recruitment of muscles using FES go beyond simple contraction of muscles. Evidence suggests that FES can induce short- and long-term neurophysiological changes in the central nervous system by varying the stimulation parameters and delivery methods. By taking advantage of this, FES has been used to restore voluntary movement in individuals with neurological injuries with a technique called FES therapy (FEST). However, long-lasting cortical re-organization (neuroplasticity) depends on the ability to synchronize the descending (voluntary) commands and the successful execution of the intended task using a FES. Brain-computer interface (BCI) technologies offer a way to synchronize cortical commands and movements generated by FES, which can be advantageous for inducing neuroplasticity. Therefore, the aim of this review paper is to discuss the neurophysiological mechanisms of electrical stimulation of muscles and nerves and how BCI-controlled FES can be used in rehabilitation to improve motor function.

Complex naturalistic stimuli are prioritized by attention when maintained in visual working memory

Recent studies suggest that a stimulus actively maintained in working memory (WM) automatically captures visual attention when subsequently perceived. Such a WM-guidance effect has been consistently observed for stimuli defined by simple features, such as colour or orientation, but studies using more complex stimuli provide inconclusive results. Therefore, we investigated whether the WM-guidance effect occurs also for naturalistic stimuli, whose identity is defined by multiple features and relations among them, specifically for faces and houses. The experiment consisted of multiple blocks in which participants (N = 28) either memorized or merely saw (WM or exposure condition, respectively) a template stimulus and then performed several dot-probe trials, with pairs of stimuli (template and control) presented laterally as distractors and followed by a target-asterisk. Evidence for attentional prioritization of the memorized stimuli was found in the reaction-times (RT) analysis, but not in the analysis of the N2pc ERP component, which raises questions concerning the attentional mechanism involved. Further, in an exploratory ERP analysis we found evidence for a very early (100-200 ms post-stimulus) prioritization specific to the memorized faces, which is in line with the sensory recruitment theory of WM.

Modulation of the pupillary response by the content of visual working memory

Studies of selective attention during perception have revealed modulation of the pupillary response according to the brightness of task-relevant (attended) vs. -irrelevant (unattended) stimuli within a visual display. As a strong test of top-down modulation of the pupil response by selective attention, we asked whether changes in pupil diameter follow internal shifts of attention to memoranda of visual stimuli of different brightness maintained in working memory, in the absence of any visual stimulation. Across 3 studies, we reveal dilation of the pupil when participants orient attention to the memorandum of a dark grating relative to that of a bright grating. The effect occurs even when the attention-orienting cue is independent of stimulus brightness, and even when stimulus brightness is merely incidental and not required for the working-memory task of judging stimulus orientation. Furthermore, relative dilation and constriction of the pupil occurred dynamically and followed the changing temporal expectation that 1 or the other stimulus would be probed across the retention delay. The results provide surprising and consistent evidence that pupil responses are under top-down control by cognitive factors, even when there is no direct adaptive gain for such modulation, since no visual stimuli were presented or anticipated. The results also strengthen the view of sensory recruitment during working memory, suggesting even activation of sensory receptors. The thought-provoking corollary to our findings is that the pupils provide a reliable measure of what is in the focus of mind, thus giving a different meaning to old proverbs about the eyes being a window to the mind.

Visual objects interact differently during encoding and memory maintenance

The storage mechanisms of working memory are the matter of an ongoing debate. The sensory recruitment hypothesis states that memory maintenance and perceptual encoding rely on the same neural substrate. This suggests that the same cortical mechanisms that shape object perception also apply to maintained memory content. We tested this prediction using the Direction Illusion, i.e., the mutual repulsion of two concurrently visible motion directions. Participants memorized the directions of two random dot patterns for later recall. In Experiments 1 and 2, we varied the temporal separation of spatially distinct stimuli to manipulate perceptual concurrency, while keeping concurrency within working memory constant. We observed mutual motion repulsion only under simultaneous stimulus presentation, but proactive repulsion and retroactive attraction under immediate stimulus succession. At inter-stimulus intervals of 0.5 and 2 s, however, proactive repulsion vanished, while the retroactive attraction remained. In Experiment 3, we presented both stimuli at the same spatial position and observed a reappearance of the repulsion effect. Our results indicate that the repulsive mechanisms that shape object perception across space fade during the transition from a perceptual representation to a consolidated memory content. This suggests differences in the underlying structure of perceptual and mnemonic representations. The persistence of local interactions, however, indicates different mechanisms of spatially global and local feature interactions.

Reduced visual and frontal cortex activation during visual working memory in grapheme-colour synaesthetes relative to young and older adults

The sensory recruitment model envisages visual working memory (VWM) as an emergent property that is encoded and maintained in sensory (visual) regions. The model implies that enhanced sensory-perceptual functions, as in synaesthesia, entail a dedicated VWM-system, showing reduced visual cortex activity as a result of neural specificity. By contrast, sensory-perceptual decline, as in old age, is expected to show enhanced visual cortex activity as a result of neural broadening. To test this model, young grapheme-colour synaesthetes, older adults and young controls engaged in a delayed pair-associative retrieval and a delayed matching-to-sample task, consisting of achromatic fractal stimuli that do not induce synaesthesia. While a previous analysis of this dataset (Pfeifer et al., 2016) has focused on cued retrieval and recognition of pair-associates (i.e. long-term memory), the current study focuses on visual working memory and considers, for the first time, the crucial delay period in which no visual stimuli are present, but working memory processes are engaged. Participants were trained to criterion and demonstrated comparable behavioural performance on VWM tasks. Whole-brain and region-of-interest-analyses revealed significantly lower activity in synaesthetes’ middle frontal gyrus and visual regions (cuneus, inferior temporal cortex) respectively, suggesting greater neural efficiency relative to young and older adults in both tasks. The results support the sensory recruitment model and can explain age and individual WM-differences based on neural specificity in visual cortex.