Independent Multisensory Integration and Crossmodal Attention Processing: Evidence from Audiovisual Gender Congruency Tasks

Multisensory integration and crossmodal attention are two of the basic mechanisms in processing multisensory inputs, and they are usually mixed. Whether these two processes are dependent or independent remains controversial. To examine the relationship between multisensory integration and crossmodal attention, we adopted modified multilevel audiovisual gender judgment paradigms and evaluated the congruency effects in reaction time (RT) and the inverse effectiveness (IE) effects. If they were dependent, the occurrence of one effect would be accompanied with that of the other. Using both morphed faces and voices, we first performed a speeded classification task, in which participants were either asked to attend to faces (experiment 1a) or attend to voices (experiment 1b); then, we performed an unspeeded rating task with faces as the targets (experiment 2). We observed both a congruency effect in RT and an IE effect in experiment 1a, a congruency effect in RT alone in experiment 1b, and an IE effect alone in experiment 2. These results indicate that the two processes are independent of each other.

Semantic Crosstalk in Timbre Perception

Many adjectives for musical timbre reflect cross-modal correspondence, particularly with vision and touch (e.g., “dark–bright,” “smooth–rough”). Although multisensory integration between visual/tactile processing and hearing has been demonstrated for pitch and loudness, timbre is not well understood as a locus of cross-modal mappings. Are people consistent in these semantic associations? Do cross-modal terms reflect dimensional interactions in timbre processing? Here I designed two experiments to investigate crosstalk between timbre semantics and perception through the use of Stroop-type speeded classification. Experiment 1 found that incongruent pairings of instrument timbres and written names caused significant Stroop-type interference relative to congruent pairs, indicating bidirectional crosstalk between semantic and auditory modalities. Pre-Experiment 2 asked participants to rate natural and synthesized timbres on semantic differential scales capturing luminance (brightness) and texture (roughness) associations, finding substantial consistency for a number of timbres. Acoustic correlates of these associations were also assessed, indicating an important role for high-frequency energy in the intensity of cross-modal ratings. Experiment 2 used timbre adjectives and sound stimuli validated in the previous experiment in two variants of a semantic-auditory Stroop-type task. Results of linear mixed-effects modeling of reaction time and accuracy showed slight interference in semantic processing when adjectives were paired with cross-modally incongruent instrument timbres (e.g., the word “smooth” with a “rough” timbre). Taken together, I conclude by suggesting that semantic crosstalk in timbre processing may be partially automatic and could reflect weak synesthetic congruency between interconnected sensory domains.

Pitch-verticality and pitch-size cross-modal interactions

Two studies were conducted on cross-modal matching between pitch and sound source localization on the vertical axis, and pitch and size. In the first study 100 Hz, 200 Hz, 600 Hz, and 800 Hz tones were emitted by a loudspeaker positioned 60 cm above or below to the participant’s ear level. Using a speeded classification task, 30 participants had to indicate the sound source in 160 trials. Both reaction times and errors were analyzed. The results showed that in the congruent condition of high-pitched tones emitted from the upper loudspeaker, reaction times were significantly faster and the number of errors was significantly lower. Pitch was mapped on the vertical axis for sound localization. A main effect for sound source direction was also found. Tones coming from the upper loudspeaker were recognized faster and more accurately. Males were faster than females in identifying sound source direction. In the second experiment, 20 participants had to match 21 tones varying in pitch with 9 circles differing in visual angle on 42 trials. The results showed a clear inverse linear association between log-spaced tone pitch and circle diameter.

Interactions Between Auditory Elevation, Auditory Pitch and Visual Elevation During Multisensory Perception

Cross-modal correspondences refer to associations between apparently unrelated stimulus features in different senses. For example, high and low auditory pitches are associated with high and low visual elevations, respectively. Here we examined how this crossmodal correspondence between visual elevation and auditory pitch relates to auditory elevation. We used audiovisual combinations of high- or low-frequency bursts of white noise and a visual stimulus comprising a white circle. Auditory and visual stimuli could each occur at high or low elevations. These multisensory stimuli could be congruent or incongruent for three correspondence types: cross-modal featural (auditory pitch/visual elevation), within-modal featural (auditory pitch/auditory elevation) and cross-modal spatial (auditory and visual elevation). Participants performed a 2AFC speeded classification (high or low) task while attending to auditory pitch, auditory elevation, or visual elevation. We tested for modulatory interactions between the three correspondence types. Modulatory interactions were absent when discriminating visual elevation. However, the within-modal featural correspondence affected the cross-modal featural correspondence during discrimination of auditory elevation and pitch, while the reverse modulation was observed only during discrimination of auditory pitch. The cross-modal spatial correspondence modulated the other two correspondences only when auditory elevation was being attended, was modulated by the cross-modal featural correspondence only during attention to auditory pitch, and was modulated by the within-modal featural correspondence while performing discrimination of either auditory elevation or pitch. We conclude that the cross-modal correspondence between auditory pitch and visual elevation interacts strongly with auditory elevation.

Processing Relationships Between Language-Being-Spoken and Other Speech Dimensions in Monolingual and Bilingual Listeners

While indexical information is implicated in many levels of language processing, little is known about the internal structure of the system of indexical dimensions, particularly in bilinguals. A series of three experiments using the speeded classification paradigm investigated the relationship between various indexical and non-linguistic dimensions of speech in processing. Namely, we compared the relationship between a lesser-studied indexical dimension relevant to bilinguals, which language is being spoken (in these experiments, either Mandarin Chinese or English), with: talker identity (Experiment 1), talker gender (Experiment 2), and amplitude of speech (Experiment 3). Results demonstrate that language-being-spoken is integrated in processing with each of the other dimensions tested, and that these processing dependencies seem to be independent of listeners’ bilingual status or experience with the languages tested. Moreover, the data reveal processing interference asymmetries, suggesting a processing hierarchy for indexical, non-linguistic speech features.

The semantic basis of taste-shape associations

Previous research shows that people systematically match tastes with shapes. Here, we assess the extent to which matched taste and shape stimuli share a common semantic space and whether semantically congruent versus incongruent taste/shape associations can influence the speed with which people respond to both shapes and taste words. In Experiment 1, semantic differentiation was used to assess the semantic space of both taste words and shapes. The results suggest a common semantic space containing two principal components (seemingly, intensity and hedonics) and two principal clusters, one including round shapes and the taste word “sweet,” and the other including angular shapes and the taste words “salty,” “sour,” and “bitter.” The former cluster appears more positively-valenced whilst less potent than the latter. In Experiment 2, two speeded classification tasks assessed whether congruent versus incongruent mappings of stimuli and responses (e.g., sweet with round versus sweet with angular) would influence the speed of participants’ responding, to both shapes and taste words. The results revealed an overall effect of congruence with congruent trials yielding faster responses than their incongruent counterparts. These results are consistent with previous evidence suggesting a close relation (or crossmodal correspondence) between tastes and shape curvature that may derive from common semantic coding, perhaps along the intensity and hedonic dimensions.

The semantic basis of taste-shape associations

Previous research shows that people systematically match tastes with shapes. Here, we assessed the extent to which matching taste and shape stimuli share a common semantic space and whether semantically congruent versus incongruent taste/shape associations can influence the speed with which people respond to both shapes and taste words. In Experiment 1, we used semantic differentiation to assess the semantic space of both taste words and shapes. The results suggest a common semantic space containing two principal components (seemingly potency and evaluation) and two principal clusters, one including round shapes and the taste word “sweet”, and the other including angular shapes and the taste words “salty”, “sour”, and “bitter”. The former cluster appears more positively-valenced whilst less potent than the latter. In Experiment 2, two speeded classification tasks assessed whether congruent versus incongruent mappings of stimuli and responses (e.g., sweet with round versus sweet with angular) would influence participants’ speed of responding, both to shapes and to taste words. The results revealed an overall effect of congruence that was driven mostly when the participants had to classify shapes with taste words as responses. These results are consistent with previous evidence suggesting a close relation (or crossmodal correspondence) between tastes and shape curvature that may derive from common semantic coding, perhaps along the sensory-discriminative and hedonic dimensions.

The semantic basis of taste-shape associations

Previous research shows that people systematically match tastes with shapes. Here, we assessed the extent to which matching taste and shape stimuli share a common semantic space and whether semantically congruent versus incongruent taste/shape associations can influence the speed with which people respond to both shapes and taste words. In Experiment 1, we used semantic differentiation to assess the semantic space of both taste words and shapes. The results suggest a common semantic space containing two principal components (seemingly potency and evaluation) and two principal clusters, one including round shapes and the taste word “sweet”, and the other including angular shapes and the taste words “salty”, “sour”, and “bitter”. The former cluster appears more positively-valenced whilst less potent than the latter. In Experiment 2, two speeded classification tasks assessed whether congruent versus incongruent mappings of stimuli and responses (e.g., sweet with round versus sweet with angular) would influence participants’ speed of responding, both to shapes and to taste words. The results revealed an overall effect of congruence that was driven mostly when the participants had to classify shapes with taste words as responses. These results are consistent with previous evidence suggesting a close relation (or crossmodal correspondence) between tastes and shape curvature that may derive from common semantic coding, perhaps along the sensory-discriminative and hedonic dimensions.