Sensory Substitution: Unfulfilled Promises and Fundamental Limitations

Many of the most attention-grabbing claims concerning the uptake of sensory substitution devices in the last 50 years have, noticeably, not come to pass. I highlight a number of the fundamental limitations (some acknowledged, others not) that may have prevented the development and uptake of these devices amongst individuals suffering from sensory loss. First and foremost, it may simply be impossible to fully substitute for the loss of vision (the sense most substituted for) given the imbalance in neural cortical resources given to processing information in the various senses. Second, the inability to substitute for the hedonic attributes of a given modality constitutes an important, if currently under-acknowledged, problem. Most researchers tend to focus their efforts on the substitution of the sensory-discriminative (primarily spatial) aspects of stimulation instead. Third, I highlight the technological limitations associated with providing useful substitution devices for those who have lost their sense of taste or smell, senses which, theoretically, should be far easier to substitute for. Another factor that may have limited the uptake of these devices—aesthetic concerns about the appearance of users wearing them—is, I believe, likely to disappear, as a range of other augmented-perception technologies become more widely accepted.

Sensory Substitution Devices as Advanced Sensory Tools

There has been considerable effort devoted towards understanding sensory substitution devices in terms of their relationship to canonical sensory modalities. The approach taken in this essay is rather different, although complementary, in that we seek to define a broad conceptual space of ‘sensory tools’ in which sensory substitution devices can be situated. Such devices range from telescopes, to cochlear implants, to attempts to create a magnetic sense. One feature of these devices is that they operate at the level of ‘raw’ sensory information. As such, systems such as Braille which operate at a symbolic/conceptual level do not count as a sensory tool (or a sensory substitution device) and nor would a device such as CCTV which, although capturing raw sensory information, would not meet a conventional definition of a tool. With this approach, we hope to avoid the circularity inherent in previous attempts at defining sensory substitution and provide a better starting point to explore the effects of sensory tools, more generally, on the functioning of the nervous system.

Rewired Animals and Sensory Substitution: The Cause Is Not Cortical Plasticity

Cortical plasticity is often invoked to explain changes in the quality or location of experience observed in rewired animals, in sensory substitution, in extension of the body through tool use, and in the rubber hand illusion. However this appeal to cortical plasticity may be misleading, because it suggests that the cortical areas that are plastic are themselves the loci of generation of experience. This would be an error, I claim, since cortical areas do not generate experience. Cortical areas participate in enabling the interaction of an agent with its environment, and the quality of this interaction constitutes the quality of experience. Thus it is not plasticity in itself, but the change in modes of interaction which plasticity allows, which gives rise to the change of experience observed in these studies.

Sensory Substitution and Perceptual Emergence

Designers of sensory substitution devices (SSDs) typically aim to capture a wide representational scope by requiring their devices to present in the substituting modality the full range of basic energy available to the substituted modality, and then using this information to code up any further representations needed. On this view, if we could build a device that represents to touch (say) the distal distribution of light intensity—the basic form of energy to which visual receptors are normally responsive—our device could, in principle, represent everything available to vision: colour, shape, form, motion, and so on. Unfortunately, I will argue, this simple idea fails. For perceptual modalities represent ‘emergent’ features —i.e. features whose exemplification is not fixed by the representation of the distribution of basic energy. Hence, an SSD whose basic representational vocabulary is limited to the distribution of such basic energy will leave things out. None of this shows that SSDs will inevitably fail to represent what sensory modalities normally represent. It does suggest, however, that if we want them to represent what sensory modalities normally represent, we will have to do more than preserve the representation of basic energy to which the substituted modalities are sensitive.

The Processing of What, Where, and How

Vision is the default sensory modality for normal spatial navigation in humans. Touch is restricted to providing information about peripersonal space, whereas detecting and avoiding obstacles in extrapersonal space is key for efficient navigation. Hearing is restricted to the detection of objects that emit noise, yet many obstacles such as walls are silent. Sensory substitution devices provide a means of translating distal visual information into a form that visually impaired individuals can process through either touch or hearing. Here we will review findings from various sensory substitution systems for the processing of visual information that can be classified as what (object recognition), where (localization), and how (perception for action) processing. Different forms of sensory substitution excel at some tasks more than others. Spatial navigation brings together these different forms of information and provides a useful model for comparing sensory substitution systems, with important implications for rehabilitation, neuroanatomy, and theories of cognition.

Sensory Substitution Devices and Behavioural Transference: A Commentary on Recent Work from the Lab of Amir Amedi

Sensory substitution devices (SSDs) are most familiar from their use with subjects who are deficient in a target modality (e.g. congenitally blind subjects), but there is no doubt that the use and potential value of SSDs extend to persons without such deficits. Recent work by Amedi and his team (in particular Levy-Tzedek et al. 2012) has begun to explore this. Their idea is that SSDs may facilitate behavioural transference (BT) across sense modalities. In this case, a motor skill learned through visual perception might be subsequently employed in response to auditory perception, using an SSD as a mediator. They infer from the existence of such BT that the learned skill is amodally represented. After a brief overview I identify ways to more fully test for BT within this experimental paradigm and argue that their conclusion about amodal representation is premature. Additionally, I argue that their preferred SSD (Eyemusic) is of limited value for the project. While my remarks are critical, my intention is to be constructive, particularly in light of the fact that Levy-Tzedek et al. (2012) is, I believe, the first output from Amedi’s lab concerning this line of research.

Disambiguating the Perceptual Assumption

Deroy and Auvray together with Ptito et al. have argued against what they dub ‘the perceptual assumption’, which they claim underlies all previous research into sensory substitution devices (SSDs). In this chapter, I argue that the perceptual assumption needs to be disambiguated in three distinct ways: (A) SSD use is best modelled as a known, ‘natural’ modality; (B) SSD use is best modelled as a unique sensory modality full stop; and (C) SSD use is best modelled as a perceptual process. Different theorists are variously committed to these distinct claims. More importantly, evaluating A, B, or C for rejection depends on distinct evidence of difference between SSD use and (A) each natural modality, (B) any modality, and (C) perceptual processing. I argue that even if the offered evidence of difference for A–C is granted, Auvray and Deroy’s advocated rejections are not entailed; it remains to be shown that the identified differences undermine the appropriate use of the corresponding models.

Sensory Substitution and the Transparency of Visual Experience

Sensory substitution devices make use of information in one sensory modality to deliver information usually provided by another. But when information usually presented visually is presented to a subject in an auditory or haptic way, is the resulting experience in any sense visual? Or does sensory substitution show that dimensions of experience—about the spatial layout of objects and properties in the environment—that were previously taken to be essentially visual can be experienced in other modalities too? I will consider this question by looking at whether a property such as the transparency of visual experience can be transferred to, and enhance, experience in other modalities.

Sensory Substitution and Augmentation: An Introduction

In this essay I outline the main questions and the debates about sensory substitution and augmentation devices. I describe the two most studied modern sensory substitution devices (TVSS and the vOICe) and one sensory augmentation device (the feelSpace belt). I discuss whether use of these devices gives rise to new sensory experiences of objects or just new perceptual judgements about objects. Then, on the assumption that new sensory experiences are being had, I consider what sensory modality is operative—the substituted or the substituting one, or another altogether. I examine the evidence concerning whether the experiences had in sensory substitution are of a two- or a three-dimensional world, and about the nature of those experiences with respect to whether colour is represented in them. I consider whether there are any limits to what information or what experiences can be given via sensory substitution. And I discuss whether the results from sensory substitution experiments can be used to support certain theories of perception at the expense of rivals. Furthermore, the practical use of sensory substitution and augmentation devices is considered. Finally, I provide a brief overview of the rest of the essays that this volume contains and the host of further interesting issues that the authors consider and address.

The Role of Noetic Feelings in Sensory Substitution

A central issue in the study of sensory substitution devices is whether and to what extent the original source of knowledge about the world that they seem to enable substitutes for ordinary perception. I argue that this issue divides into two sub-issues, which concern respectively the sensory and the affective aspects of the phenomenology of perception. The sensory aspect determines the representational contents of experience, while the affective aspect is constituted by noetic feelings such as familiarity and presence. A pessimistic view may be formulated according to which neither familiarity nor presence can be reproduced in sensory substitution. There are important phenomenological differences between ordinary perception and sensory substitution due to the massively parallel architecture of the former. However, drawing on a discussion of Capgras syndrome and derealization disorder, I show that we also enjoy higher-level, post-perceptual feelings of familiarity and presence. In the last part of the essay, I draw on the psychological literature on metacognition and sketch a fluency-based account of both these feelings, which leads to a more optimistic view about whether they can be enjoyed by trained users of sensory substitution devices.