Perception and Imagery

This chapter provides a general account of imagery that applies to both external senses such as vision and internal senses such as pain. It identifies five mental operations that occur in all kinds of imagery: intensification, focusing, combination, juxtaposition, and decomposition. Each of these operations results from neural mechanisms that are part of the Semantic Pointer Architecture, including storage, retrieval, neural representation, binding, competition, and transformation. There is abundant psychological and neural evidence that imagery is real and that the brain’s computations employ special patterns of neural representation that develop from sensory inputs. This development requires binding into semantic pointers that are susceptible to symbol-like manipulation that exploits the different sensory characters of visual, auditory, and other sorts of representation.

The Self

The self is a complex of mechanisms at multiple levels that include the molecular and the social. Semantic pointers are crucial to the self with respect to various phenomena, including how one represents oneself to oneself and to others, as well as in how one evaluates oneself. Also explained are operations that the self does to itself in efforts to achieve short-term goals such as self-control and long-term goals such as self-fulfillment. Semantic pointer explanations of images, concepts, and other mental representations are important for understanding how selves accomplish their goals. Representations of the self via semantic pointers can recursively be bound into semantic pointers for beliefs, desires, and intentions. Discussion of the social mechanisms relevant to the self begins to connect neural and mental mechanisms with discussions of social sciences and professions.

Language

Semantic pointers handle syntactic structure in a way that integrates with other key aspects of language, including semantics, pragmatics, and phonology. Semantic pointers plausibly provide the underlying neural mechanisms for Jackendoff’s parallel architecture and for other theories of language that go beyond Chomsky’s syntax-first approach. In particular, they show how the mental representation of a word can efficiently combine information about sound, meaning, and grammar to enable the organization of words into sentences. Semantic pointers cast the meanings of words and sentences as multidimensional, relying not just on the relations of words to other words but also on the relation of words to the world through sensory-motor operations, with further contributions from genetic and social processes. The Semantic Pointer Architecture also provides neural mechanisms for explaining complex linguistic phenomena such as conceptual blending and metaphor.

Action and Intention

Actions results from the same neural mechanisms that explain sensation, imagery, concepts, rules, analogies, emotions, and consciousness. Neural representations govern motor operations such as walking and talking. Action selection, however, goes beyond simple associations of perception and motor control, because of deliberations in humans using beliefs, desires, and intentions. The basic neural mechanisms of representation, binding into semantic pointers, and competition among pointers function to produce actions. Intentions are semantic pointers that bind representations of the relevant situation, doing, evaluation, and self. Intentions are embodied in that representing the situation includes perceptions, doing the action includes motor representations, and performing the evaluation is an emotional process that includes physiology. But intentions can also be transbodied, when representations for the situation, cognitive appraisal, and the self are abstracted by recursive bindings that far surpass sensory-motor inputs.

Consciousness

Progress is being made in understanding how brain mechanisms generate conscious experience. Simple conscious experiences such as sensations of colors, shapes, and sounds require only neural representations as patterns of firing that result from sensory inputs and internal processing. More complicated conscious experiences, such as awareness of reading in a chair in a room, require the amalgamation of sensations and images into more complex representations through binding into semantic pointers. Recursive binding—bindings of bindings of bindings—can produce the most complicated kinds of conscious experience of which humans are capable, taking people from feelings to awareness to self-awareness. Consciousness is limited because recursive binding and competition among the resulting semantic pointers depend on processing by many neurons.

Emotions

Emotions serve not only to stand for things in the world but also to indicate their value. Decision, action, and many kinds of problem solving require determining how the world should be, not just how it is. Humans and other animals evolved with emotions as part of their innate biological machinery to guide action and inference. Emotions are patterns of neural firing that result from binding three different factors that are complementary rather than conflictive. A verbal or sensory representation of a situation can be bound both with a representation of the physiological states that the situation elicited and a cognitive appraisal of the import of the situation. Cognitive appraisal can also incorporate social factors because of the contributions of social goals and the culturally established associations of emotional words.

Analogies

Analogies contribute to many kinds of human thinking, including problem solving, decision making, explanation, persuasion, and entertainment. An analogy is a systematic comparison between a source analog and a target analog, where information about the source is used to generate inferences about the target. The major stages of analogical thinking are (a) obtaining a source analog by memory retrieval or other means, (b) mapping the source to the target, (c) adapting the source to inform the target, and (d) learning by generalizing source and target into a schema. Most theories of analogy have used verbal representations, but a much broader appreciation of analogical thinking can be gained with semantic pointers. Analogies often use words, but they can also operate with visual, auditory, and other sensory modalities, all of which can contribute to all stages of analogy.

How Brains Make Minds

Brains make minds because mental representations and processes are performed by neural mechanisms. Mental representations work by patterns of firing in neural groups. More complicated representations that go beyond sensory experience can be formed by binding representations together, combining patterns of firing into new ones. In particular, binding can produce semantic pointers that coalesce and compress different kinds of information, including sensory, motor, emotional and verbal information. Semantic pointers retain connections to sensory and motor experience while also acquiring the autonomy that is usually attributed to symbols. Eliasmith’s semantic pointer hypothesis shows how neural cells can interact to produce high-level thinking. Different representations compete with each other to provide accounts of what is going on in the world through a parallel process of satisfaction of multiple constraints. Neural networks can learn by changing the synaptic connections between neurons.

Rules

Rules are mental representations of the form If condition, then action, where matching the condition leads to execution of the action. Chaining rules together makes possible solution of complex problems, such as figuring out how to get from one city to another. Mental rules of this sort are also important for explaining people’s ability to generate and comprehend language. Semantic pointers provide a valuable supplement to conventional theories of rules in two ways. First, they show how rules as mental representations can also be neural representations, through encoding and binding of if–then, the condition, and the action, all as patterns of firing. Second, semantic pointers show how conditions and actions can go beyond verbal information to incorporate all kinds of sensory information.

Creativity

Creativity results from neural processes that include binding of representations, generation of new concepts and rules, and the application of analogies. The Semantic Pointer Architecture accommodates the full range of multimodal representations needed for creativity in the domains of scientific discovery, technological invention, artistic imagination, and social innovation. New semantic pointers can be generated by convolution-based bindings in ways that produce new and useful images, concepts, rules, and analogies. Procedural creativity is the generation of new methods expressed as rules. The pragmatic focus of creative problem solving and the evaluation of the goal relevance of new products can be carried out by emotions. Competition among semantic pointers explains how the realization that one might have done something creative enters consciousness.