Crossmodal correspondences: Three ways to match content across sensory modalities

2012 ◽  
Author(s):  
Ophelia Deroy
Keyword(s):  
Crossmodal Correspondences ◽  
Sensory Modalities

Audiovisual crossmodal correspondences in Autism Spectrum Disorders (ASDs)

2012 ◽  
Vol 25 (0) ◽  
pp. 44
Author(s):  
Valeria Occelli ◽  
Gianluca Esposito ◽  
Paola Venuti ◽  
Peter Walker ◽  
Massimiliano Zampini
Keyword(s):  
Autism Spectrum ◽  
Crossmodal Correspondences ◽  
Association Pattern ◽  
Visual Patterns ◽  
Visual Inputs ◽  
Sensory Modalities ◽  
Crossmodal Correspondence ◽  
Matching Performance ◽  
Spectrum Disorders ◽  
Auditory Features

The label ‘crossmodal correspondences’ has been used to define the nonarbitrary associations that appear to exist between different basic physical stimulus attributes in different sensory modalities. For instance, it has been consistently shown in the neurotypical population that higher pitched sounds are more frequently matched with visual patterns which are brighter, smaller, and sharper than those associated to lower pitched sounds. Some evidence suggests that patients with ASDs tend not to show this crossmodal preferential association pattern (e.g., curvilinear shapes and labial/lingual consonants vs. rectilinear shapes and plosive consonants). In the present study, we compared the performance of children with ASDs (6–15 years) and matched neurotypical controls in a non-verbal crossmodal correspondence task. The participants were asked to indicate which of two bouncing visual patterns was making a centrally located sound. In intermixed trials, the visual patterns varied in either size, surface brightness, or shape, whereas the sound varied in pitch. The results showed that, whereas the neurotypical controls reliably matched the higher pitched sound to a smaller and brighter visual pattern, the performance of participants with ASDs was at chance level. In the condition where the visual patterns differed in shape, no inter-group difference was observed. Children’s matching performance cannot be attributed to intensity matching or difficulties in understanding the instructions, which were controlled. These data suggest that the tendency to associate congruent visual and auditory features vary as a function of the presence of ASDs, possibly pointing to poorer capabilities to integrate auditory and visual inputs in this population.

scholarly journals On the Relative Nature of (Pitch-Based) Crossmodal Correspondences

2019 ◽  
Vol 32 (3) ◽  
pp. 235-265 ◽  
Author(s):  
Charles Spence
Keyword(s):  
Empirical Investigation ◽  
Visual Stimuli ◽  
Crossmodal Correspondences ◽  
Clear Cut ◽  
Specific Focus ◽  
Sensory Modalities ◽  
One To One ◽  
Crossmodal Associations

Abstract This review deals with the question of the relative vs absolute nature of crossmodal correspondences, with a specific focus on those correspondences involving the auditory dimension of pitch. Crossmodal correspondences have been defined as the often-surprising crossmodal associations that people experience between features, attributes, or dimensions of experience in different sensory modalities, when either physically present, or else merely imagined. In the literature, crossmodal correspondences have often been contrasted with synaesthesia in that the former are frequently said to be relative phenomena (e.g., it is the higher-pitched of two sounds that is matched with the smaller of two visual stimuli, say, rather than there being a specific one-to-one crossmodal mapping between a particular pitch of sound and size of object). By contrast, in the case of synaesthesia, the idiosyncratic mapping between inducer and concurrent tends to be absolute (e.g., it is a particular sonic inducer that elicits a specific colour concurrent). However, a closer analysis of the literature soon reveals that the distinction between relative and absolute in the case of crossmodal correspondences may not be as clear-cut as some commentators would have us believe. Furthermore, it is important to note that the relative vs absolute question may receive different answers depending on the particular (class of) correspondence under empirical investigation.

scholarly journals Smelling sensations: olfactory crossmodal correspondences

2020 ◽  
Author(s):  
Ryan Joseph Ward ◽  
Sophie Wuerger ◽  
Alan Marshall
Keyword(s):  
Aromatic Compounds ◽  
Explicit Knowledge ◽  
Principal Component ◽  
Multisensory Perception ◽  
Crossmodal Correspondences ◽  
Cross Model ◽  
Sensory Modalities ◽  
Olfactory Stimuli ◽  
Emotional Dimensions ◽  
Musical Genres

Crossmodal correspondences are the associations between apparently distinct stimuli in different sensory modalities . These associations, albeit surprising, are generally shared in most of the population. Olfaction is ingrained in the fabric of our daily life and constitutes an integral part of our perceptual reality, with olfaction being more commonly used in the entertainment and analytical domains, it is crucial to uncover the robust correspondences underlying common aromatic compounds. Towards this end, we investigated an aggregate of crossmodal correspondences between ten olfactory stimuli and other modalities ( angularity of shapes, smoothness of texture, pleasantness, pitch, colours, musical genres and emotional dimensions ) using a large sample of 68 observers. We uncover the correspondences between these modalities and extent of these associations with respect to the explicit knowledge of the respective aromatic compound. The results revealed the robustness of prior studies, as well as, contributions towards olfactory integration between an aggregate of other dimensions. The knowledge of an odour's identity coupled with the multisensory perception of the odours indicates that these associations, for the most part, are relatively robust and do not rely on explicit knowledge of the odour. Through principal component analysis of the perceptual ratings, new cross-model mediations have been uncovered between odours and their intercorrelated sensory dimensions. Our results demonstrate a collective of associations between olfaction and other dimensions, potential cross modal mediations via exploratory factor analysis and the robustness of these correspondence with respect to the explicit knowledge of an odour. We anticipate the findings reported in this paper could be used as a psychophysical framework aiding in a collective of applications ranging from olfaction enhanced multimedia to marketing.

Crossmodal Correspondences: Four Challenges

2016 ◽  
Vol 29 (1-3) ◽  
pp. 29-48 ◽  
Author(s):  
Ophelia Deroy ◽  
Charles Spence
Keyword(s):  
Mental Imagery ◽  
Crossmodal Correspondences ◽  
Sensory Modalities ◽  
Behavioral Tasks ◽  
Multisensory Interactions

The renewed interest that has emerged around the topic of crossmodal correspondences in recent years has demonstrated that crossmodal matchings and mappings exist between the majority of sensory dimensions, and across all combinations of sensory modalities. This renewed interest also offers a rapidly-growing list of ways in which correspondences affect — or interact with — metaphorical understanding, feelings of ‘knowing’, behavioral tasks, learning, mental imagery, and perceptual experiences. Here we highlight why, more generally, crossmodal correspondences matter to theories of multisensory interactions.

scholarly journals The Effects of Mandarin Chinese Lexical Tones in Sound–Shape and Sound–Size Correspondences

2021 ◽  
pp. 1-15
Author(s):  
Yen-Han Chang ◽  
Mingxue Zhao ◽  
Yi-Chuan Chen ◽  
Pi-Chun Huang
Keyword(s):  
Mandarin Chinese ◽  
Shape Matching ◽  
Crossmodal Correspondences ◽  
Lexical Tones ◽  
Chinese Speakers ◽  
Sensory Modalities ◽  
Native Mandarin Chinese Speakers

Abstract Crossmodal correspondences refer to when specific domains of features in different sensory modalities are mapped. We investigated how vowels and lexical tones drive sound–shape (rounded or angular) and sound–size (large or small) mappings among native Mandarin Chinese speakers. We used three vowels (/i/, /u/, and /a/), and each vowel was articulated in four lexical tones. In the sound–shape matching, the tendency to match the rounded shape was decreased in the following order: /u/, /i/, and /a/. Tone 2 was more likely to be matched to the rounded pattern, whereas Tone 4 was more likely to be matched to the angular pattern. In the sound–size matching, /a/ was matched to the larger object more than /u/ and /i/, and Tone 2 and Tone 4 correspond to the large–small contrast. The results demonstrated that both vowels and tones play prominent roles in crossmodal correspondences, and sound–shape and sound–size mappings are heterogeneous phenomena.

scholarly journals QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

2020 ◽  
Vol 79 (11-12) ◽  
pp. 7987-8009 ◽  
Author(s):  
Gebremariam Mesfin ◽  
Nadia Hussain ◽  
Elahe Kani-Zabihi ◽  
Alexandra Covaci ◽  
Estêvão B. Saleme ◽  
...  
Keyword(s):  
Heart Rate ◽  
Research Effort ◽  
Eye Gaze ◽  
Video Clip ◽  
Crossmodal Correspondences ◽  
Screen Questionnaire ◽  
Video Clips ◽  
Digital World ◽  
Sensory Modalities ◽  
Tracking Device

AbstractA great deal of research effort has been put in exploring crossmodal correspondences in the field of cognitive science which refer to the systematic associations frequently made between different sensory modalities (e.g. high pitch is matched with angular shapes). However, the possibilities cross-modality opens in the digital world have been relatively unexplored. Therefore, we consider that studying the plasticity and the effects of crossmodal correspondences in a mulsemedia setup can bring novel insights about improving the human-computer dialogue and experience. Mulsemedia refers to the combination of three or more senses to create immersive experiences. In our experiments, users were shown six video clips associated with certain visual features based on color, brightness, and shape. We examined if the pairing with crossmodal matching sound and the corresponding auto-generated haptic effect, and smell would lead to an enhanced user QoE. For this, we used an eye-tracking device as well as a heart rate monitor wristband to capture users’ eye gaze and heart rate whilst they were experiencing mulsemedia. After each video clip, we asked the users to complete an on-screen questionnaire with a set of questions related to smell, sound and haptic effects targeting their enjoyment and perception of the experiment. Accordingly, the eye gaze and heart rate results showed significant influence of the cross-modally mapped multisensorial effects on the users’ QoE. Our results highlight that when the olfactory content is crossmodally congruent with the visual content, the visual attention of the users seems shifted towards the correspondent visual feature. Crosmodally matched media is also shown to result in an enhanced QoE compared to a video only condition.

Evaluative similarity hypothesis of crossmodal correspondences: A developmental view

2012 ◽  
Vol 25 (0) ◽  
pp. 127 ◽  
Author(s):  
Dragan Jankovic
Keyword(s):  
Multisensory Integration ◽  
Age Groups ◽  
Auditory Stimuli ◽  
Crossmodal Correspondences ◽  
Affective Valence ◽  
Similarity Hypothesis ◽  
Visual Patterns ◽  
Sensory Modalities ◽  
Valence And Arousal ◽  
Crossmodal Associations

Crossmodal correspondences have been widely demonstrated, although mechanisms that stand behind the phenomenon have not been fully established yet. According to the Evaluative similarity hypothesis crossmodal correspondences are influenced by evaluative (affective) similarity of stimuli from different sensory modalities (Jankovic, 2010, Journal of Vision 10(7), 859). From this view, detection of similar evaluative information in stimulation from different sensory modalities facilitates crossmodal correspondences and multisensory integration. The aim of this study was to explore the evaluative similarity hypothesis of crossmodal correspondences in children. In Experiment 1 two groups of participants (nine- and thirteen-year-olds) were asked to make explicit matches between presented auditory stimuli (1 s long sound clips) and abstract visual patterns. In Experiment 2 the same participants judged abstract visual patterns and auditory stimuli on the set of evaluative attributes measuring affective valence and arousal. The results showed that crossmodal correspondences are mostly influenced by evaluative similarity of visual and auditory stimuli in both age groups. The most frequently matched were visual and auditory stimuli congruent in both valence and arousal, followed by stimuli congruent in valence, and finally stimuli congruent in arousal. Evaluatively incongruent stimuli demonstrated low crossmodal associations especially in older group.

scholarly journals Smelling Sensations: Olfactory Crossmodal Correspondences †

2021 ◽  
Author(s):  
Ryan J. Ward ◽  
Sophie M. Wuerger ◽  
Alan Marshall
Keyword(s):  
Quality Of Experience ◽  
Crossmodal Correspondences ◽  
Daily Lives ◽  
Crossmodal Interactions ◽  
Sensory Modalities ◽  
Olfactory Stimuli ◽  
Emotional Dimensions ◽  
Musical Genres ◽  
Sensory Expectations

Olfaction is ingrained into the fabric of our daily lives and constitutes an integral part of our perceptual reality. Within this reality, there are crossmodal interactions and sensory expectations; understanding how olfaction interacts with other sensory modalities is crucial for augmenting interactive experiences with more advanced multisensorial capabilities. This knowledge will eventually lead to better designs, more engaging experiences, and enhancing the perceived quality of experience. Toward this end, the authors investigated a range of crossmodal correspondences between ten olfactory stimuli and different modalities (angularity of shapes, smoothness of texture, pleasantness, pitch, colors, musical genres, and emotional dimensions) using a sample of 68 observers. Consistent crossmodal correspondences were obtained in all cases, including our novel modality (the smoothness of texture). These associations are most likely mediated by both the knowledge of an odor’s identity and the underlying hedonic ratings: the knowledge of an odor’s identity plays a role when judging the emotional and musical dimensions but not for the angularity of shapes, smoothness of texture, perceived pleasantness, or pitch. Overall, hedonics was the most dominant mediator of crossmodal correspondences.

How the Blind Hear Colour

Perception ◽  
2019 ◽  
Vol 48 (3) ◽  
pp. 237-241 ◽  
Author(s):  
Armin Saysani
Keyword(s):  
Sensory Experience ◽  
Tone Mapping ◽  
Crossmodal Correspondences ◽  
Congenital Blindness ◽  
Sensory Modalities ◽  
Pure Tones ◽  
Lower Tone

Crossmodal correspondences between seemingly independent sensory modalities are often observed in normal participants. For instance, colours commonly map consistently onto pure tones. In this study, we investigated colour-tone mapping in both normal trichromats and in people with congenital blindness. Participants were asked to match tones of differing pitch to named colours. In both cases, the tones selected varied consistently with the colour. The blind responses were similar to those of the trichromats, except in the case of red and green; the blind did not differentiate these colours, whereas the trichromats associated red with a higher tone and green with a lower tone. Otherwise, the results are consistent with a well-established association between pitch and lightness, with lighter colours associated with higher tones, and darker colours with lower tones. Because the blind never had any sensory experience of colour, their matching of colour to pitch is most likely based on semantic understanding.

scholarly journals Crossmodal Correspondences in the Sounds of Chinese Instruments

Perception ◽  
2019 ◽  
Vol 49 (1) ◽  
pp. 81-97
Author(s):  
Yuxuan Qi ◽  
Fuxing Huang ◽  
Zeyan Li ◽  
Xiaoang Wan
Keyword(s):  
Color Patch ◽  
Crossmodal Correspondences ◽  
Chinese Music ◽  
Sensory Modalities ◽  
Percussion Instruments ◽  
Shed Light

People tend to associate stimuli from different sensory modalities, a phenomenon known as crossmodal correspondences. We conducted two experiments to investigate how Chinese participants associated musical notes produced by four types of Chinese instruments (bowed strings, plucked strings, winds, and percussion instruments) with different colors, taste terms, and fabric textures. Specifically, the participants were asked to select a sound to match each color patch or taste term in Experiment 1 and to match the experience of touching each fabric in Experiment 2. The results demonstrated some associations between pitch and color, taste term, or the smoothness of fabrics. Moreover, certain types of Chinese instruments were preferentially chosen to match some of the colors, taste terms, and the texture of certain fabrics. These findings therefore provided insights about the perception of Chinese music and shed light on how to apply the multisensory features of sounds to enhance the composition, performance, and appreciation of music.

Sign in / Sign up

Export Citation Format

Share Document