Crossmodal Correspondence Between Auditory Timbre and Visual Shape

2021 ◽  
pp. 1-21
Author(s):  
Daniel Gurman ◽  
Colin R. McCormick ◽  
Raymond M. Klein
Keyword(s):  
Large Body ◽  
Irrelevant Stimulus ◽  
Perceived Fit ◽  
Crossmodal Correspondences ◽  
Semantic Task ◽  
Crossmodal Correspondence ◽  
Association Data ◽  
Stimulus Features ◽  
High Level ◽  
Visual Shape

Abstract Crossmodal correspondences are defined as associations between crossmodal stimuli based on seemingly irrelevant stimulus features (i.e., bright shapes being associated with high-pitched sounds). There is a large body of research describing auditory crossmodal correspondences involving pitch and volume, but not so much involving auditory timbre, the character or quality of a sound. Adeli and colleagues (2014, Front. Hum. Neurosci. 8, 352) found evidence of correspondences between timbre and visual shape. The present study aimed to replicate Adeli et al.’s findings, as well as identify novel timbre–shape correspondences. Participants were tested using two computerized tasks: an association task, which involved matching shapes to presented sounds based on best perceived fit, and a semantic task, which involved rating shapes and sounds on a number of scales. The analysis of association matches reveals nonrandom selection, with certain stimulus pairs being selected at a much higher frequency. The harsh/jagged and smooth/soft correspondences observed by Adeli et al. were found to be associated with a high level of consistency. Additionally, high matching frequency of sounds with unstudied timbre characteristics suggests the existence of novel correspondences. Finally, the ability of the semantic task to supplement existing crossmodal correspondence assessments was demonstrated. Convergent analysis of the semantic and association data demonstrates that the two datasets are significantly correlated (−0.36) meaning stimulus pairs associated with a high level of consensus were more likely to hold similar perceived meaning. The results of this study are discussed in both theoretical and applied contexts.

scholarly journals Perceptual decisions are biased toward relevant prior choices

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Helen Feigin ◽  
Shira Baror ◽  
Moshe Bar ◽  
Adam Zaidel
Keyword(s):  
Sensory Information ◽  
Irrelevant Stimulus ◽  
Stimulus Feature ◽  
Serial Dependence ◽  
Low Level ◽  
Subsequent Test ◽  
Key Factor ◽  
High Level ◽  
Motor Actions ◽  
The Brain

AbstractPerceptual decisions are biased by recent perceptual history—a phenomenon termed 'serial dependence.' Here, we investigated what aspects of perceptual decisions lead to serial dependence, and disambiguated the influences of low-level sensory information, prior choices and motor actions. Participants discriminated whether a brief visual stimulus lay to left/right of the screen center. Following a series of biased ‘prior’ location discriminations, subsequent ‘test’ location discriminations were biased toward the prior choices, even when these were reported via different motor actions (using different keys), and when the prior and test stimuli differed in color. By contrast, prior discriminations about an irrelevant stimulus feature (color) did not substantially influence subsequent location discriminations, even though these were reported via the same motor actions. Additionally, when color (not location) was discriminated, a bias in prior stimulus locations no longer influenced subsequent location discriminations. Although low-level stimuli and motor actions did not trigger serial-dependence on their own, similarity of these features across discriminations boosted the effect. These findings suggest that relevance across perceptual decisions is a key factor for serial dependence. Accordingly, serial dependence likely reflects a high-level mechanism by which the brain predicts and interprets new incoming sensory information in accordance with relevant prior choices.

Cognitive Styles Differentiate Crossmodal Correspondences Between Pitch Glide and Visual Apparent Motion

2017 ◽  
Vol 30 (3-5) ◽  
pp. 363-385 ◽  
Author(s):  
Lu Guo ◽  
Ming Bao ◽  
Luyang Guan ◽  
Lihan Chen
Keyword(s):  
Apparent Motion ◽  
Moving Objects ◽  
High Efficiency ◽  
Cognitive Styles ◽  
Crossmodal Correspondences ◽  
Crossmodal Interaction ◽  
Visual Apparent Motion ◽  
Field Independent ◽  
Crossmodal Correspondence ◽  
Figure Test

Crossmodal correspondences are the automatic associations that most people have between different basic sensory stimulus attributes, dimensions, or features. For instance, people often show a systematic tendency to associate moving objects with changing pitches. Cognitive styles are defined as an individual’s consistent approach to think, perceive, and remember information, and they reflect qualitative rather than quantitative differences between individuals in their thinking processes. Here we asked whether cognitive styles played a role in modulating the crossmodal interaction. We used the visual Ternus display in our study, since it elicits two distinct apparent motion percepts: element motion (with a shorter interval between the two Ternus frames) and group motion (with a longer interval between the two frames). We examined the audiovisual correspondences between the visual Ternus movement directions (upward or downward) and the changes of pitches of concurrent glides (ascending frequency or descending frequency). Moreover, we measured the cognitive styles (with the Embedded Figure Test) for each participant. The results showed that congruent correspondence between pitch-ascending (decreasing) glides and moving upward (downward) visual directions led to a more dominant percept of ‘element motion’, and such an effect was typically observed in the field-independent group. Importantly, field-independent participants demonstrated a high efficiency for identifying the properties of audiovisual events and applying the crossmodal correspondence in crossmodal interaction. The results suggest cognitive styles could differentiate crossmodal correspondences in crossmodal interaction.

scholarly journals Rational metareasoning and the plasticity of cognitive control

2017 ◽  
Author(s):  
Falk Lieder ◽  
Amitai Shenhav ◽  
Sebastian Musslick ◽  
Tom Griffiths
Keyword(s):  
Cognitive Control ◽  
Self Control ◽  
Neural Pathways ◽  
Interference Control ◽  
Learning Mechanisms ◽  
Control Failure ◽  
Stimulus Features ◽  
High Level ◽  
Computational Resources ◽  
And Behavior

The human brain has the impressive capacity to adapt how it processes information to high-level goals. While it is known that these cognitive control skills are malleable and can be improved through training, the underlying plasticity mechanisms are not well understood. Here, we develop and evaluate a model of how people learn when to exert cognitive control, which controlled process to use, and how much effort to exert. We derive this model from a general theory according to which the function of cognitive control is to select and configure neural pathways so as to make optimal use of finite time and limited computational resources. The central idea of our Learned Value of Control model is that people use reinforcement learning to predict the value of candidate control signals of different types and intensities based on stimulus features. This model correctly predicts the learning and transfer effects underlying the adaptive control-demanding behavior observed in an experiment on visual attention and four experiments on interference control in Stroop and Flanker paradigms. Moreover, our model explained these findings significantly better than an associative learning model and a Win-Stay Lose-Shift model. Our findings elucidate how learning and experience might shape people’s ability and propensity to adaptively control their minds and behavior. We conclude by predicting under which circumstances these learning mechanisms might lead to self-control failure.

Audiovisual crossmodal correspondences in Autism Spectrum Disorders (ASDs)

2012 ◽  
Vol 25 (0) ◽  
pp. 44
Author(s):  
Valeria Occelli ◽  
Gianluca Esposito ◽  
Paola Venuti ◽  
Peter Walker ◽  
Massimiliano Zampini
Keyword(s):  
Autism Spectrum ◽  
Crossmodal Correspondences ◽  
Association Pattern ◽  
Visual Patterns ◽  
Visual Inputs ◽  
Sensory Modalities ◽  
Crossmodal Correspondence ◽  
Matching Performance ◽  
Spectrum Disorders ◽  
Auditory Features

The label ‘crossmodal correspondences’ has been used to define the nonarbitrary associations that appear to exist between different basic physical stimulus attributes in different sensory modalities. For instance, it has been consistently shown in the neurotypical population that higher pitched sounds are more frequently matched with visual patterns which are brighter, smaller, and sharper than those associated to lower pitched sounds. Some evidence suggests that patients with ASDs tend not to show this crossmodal preferential association pattern (e.g., curvilinear shapes and labial/lingual consonants vs. rectilinear shapes and plosive consonants). In the present study, we compared the performance of children with ASDs (6–15 years) and matched neurotypical controls in a non-verbal crossmodal correspondence task. The participants were asked to indicate which of two bouncing visual patterns was making a centrally located sound. In intermixed trials, the visual patterns varied in either size, surface brightness, or shape, whereas the sound varied in pitch. The results showed that, whereas the neurotypical controls reliably matched the higher pitched sound to a smaller and brighter visual pattern, the performance of participants with ASDs was at chance level. In the condition where the visual patterns differed in shape, no inter-group difference was observed. Children’s matching performance cannot be attributed to intensity matching or difficulties in understanding the instructions, which were controlled. These data suggest that the tendency to associate congruent visual and auditory features vary as a function of the presence of ASDs, possibly pointing to poorer capabilities to integrate auditory and visual inputs in this population.

scholarly journals Audiovisual Crossmodal Correspondence between Bubbles’ Size and Pouring Sounds’ Pitch in Carbonated Beverages

Foods ◽  
2020 ◽  
Vol 9 (8) ◽  
pp. 966 ◽  
Author(s):  
Jérémy Roque ◽  
Jérémie Lafraire ◽  
Malika Auvray
Keyword(s):  
Bubble Size ◽  
Implicit Association Test ◽  
Two Dimensions ◽  
Crossmodal Correspondences ◽  
Modal Interactions ◽  
Attentional Processes ◽  
Carbonated Beverages ◽  
Crossmodal Correspondence ◽  
Correspondence Effect ◽  
Perceptions And Expectations

Visual and auditory carbonation have been separately documented as being two sensory markers of perceived freshness in beverages. The aim of the present study is to investigate the cross-modal interactions between these two dimensions of carbonation. Three experiments focused on crossmodal correspondences between bubble size and pouring sound pitch, which have never been investigated with ecological stimuli. Experiment 1, using an implicit association test (IAT), showed a crossmodal correspondence between bubble size and pouring sound pitch. Experiment 2 confirmed this pitch-size correspondence effect by means of a Go/No-Go Association Task (GNAT). Experiment 3 investigated the mutual dependence between pitch, size, and spatial elevation as well as the influence of attentional factors. No dependence was found, however pitch-size correspondences were obtained only in the condition requiring attentional processes, suggesting that these effects might be driven by top-down influences. These results highlight the robustness of the pitch-size crossmodal correspondence across stimulus contexts varying in complexity. Thus, this correspondence might be fruitfully used to modulate consumers’ perceptions and expectations about carbonated beverages.

scholarly journals Round Faces Are Associated with Sweet Foods: The Role of Crossmodal Correspondence in Social Perception

Foods ◽  
2019 ◽  
Vol 8 (3) ◽  
pp. 103 ◽  
Author(s):  
Kosuke Motoki ◽  
Toshiki Saito ◽  
Rui Nouchi ◽  
Ryuta Kawashima ◽  
Motoaki Sugiura
Keyword(s):  
Social Perception ◽  
Sweet Taste ◽  
Taste Preferences ◽  
Crossmodal Correspondences ◽  
Facial Cues ◽  
Know How ◽  
Crossmodal Correspondence ◽  
Sweet Foods ◽  
Product Recommendations

In retail settings, social perception of other peoples’ preferences is fundamental to successful interpersonal interactions (e.g., product recommendations, gift-giving). This type of perception must be made with little information, very often based solely on facial cues. Although people are capable of accurately predicting others’ preferences from facial cues, we do not yet know how such inferences are made by crossmodal correspondence (arbitrary sensory associations) between facial cues and inferred attributes. The crossmodal correspondence literature implies the existence of sensory associations between shapes and tastes, and people consistently match roundness and angularity to sweet and sour foods, respectively. Given that peoples’ faces have dimensions characterized by roundness and angularity, it may be plausible that people infer others’ preferences by relying on the correspondence between facial roundness and taste. Based on a crossmodal correspondence framework, this study aimed to reveal the role of shape–taste correspondences in social perception. We investigated whether Japanese participants infer others’ taste (sweet/sour) preferences based on facial shapes (roundness/angularity). The results showed that participants reliably inferred that round-faced (vs. angular-faced) individuals preferred sweet foods (Study 1). Round-faced individuals and sweet foods were well matched, and the matching mediated the inference of other person’s preferences (Study 2). An association between facial roundness and inference of sweet taste preferences was observed in more natural faces, and perceived obesity mediated this association (Study 3). These findings advance the applicability of crossmodal correspondences in social perception, and imply the pervasiveness of prejudicial bias in the marketplace.

scholarly journals From Psychoacoustics to Brain Waves: A Longitudinal Approach to Novel Word Learning

2021 ◽  
Vol 33 (1) ◽  
pp. 8-27
Author(s):  
Mylène Barbaroux ◽  
Arnaud Norena ◽  
Maud Rasamimanana ◽  
Eric Castet ◽  
Mireille Besson
Keyword(s):  
Word Learning ◽  
Speech Processing ◽  
Semantic Task ◽  
Far Transfer ◽  
Harmonic Complex ◽  
Complex Sounds ◽  
Perceptual Skills ◽  
Longitudinal Approach ◽  
High Level ◽  
Novel Word Learning

Musical expertise has been shown to positively influence high-level speech abilities such as novel word learning. This study addresses the question whether low-level enhanced perceptual skills causally drives successful novel word learning. We used a longitudinal approach with psychoacoustic procedures to train 2 groups of nonmusicians either on pitch discrimination or on intensity discrimination, using harmonic complex sounds. After short (approximately 3 hr) psychoacoustic training, discrimination thresholds were lower on the specific feature (pitch or intensity) that was trained. Moreover, compared to the intensity group, participants trained on pitch were faster to categorize words varying in pitch. Finally, although the N400 components in both the word learning phase and in the semantic task were larger in the pitch group than in the intensity group, no between-group differences were found at the behavioral level in the semantic task. Thus, these results provide mixed evidence that enhanced perception of relevant features through a few hours of acoustic training with harmonic sounds causally impacts the categorization of speech sounds as well as novel word learning. These results are discussed within the framework of near and far transfer effects from music training to speech processing.

Cross-Modal Correspondence Between Speech Sound and Visual Shape Influencing Perceptual Representation of Shape: the Role of Articulation and Pitch

2020 ◽  
Vol 33 (6) ◽  
pp. 569-598
Author(s):  
Yuna Kwak ◽  
Hosung Nam ◽  
Hyun-Woong Kim ◽  
Chai-Youn Kim
Keyword(s):  
Subjective Experience ◽  
Shape Representation ◽  
Speech Sound ◽  
Shape Correspondence ◽  
Leading Role ◽  
Two Factors ◽  
Pitch Information ◽  
Stimulus Features ◽  
Visual Shape

Abstract Cross-modal correspondence is the tendency to systematically map stimulus features across sensory modalities. The current study explored cross-modal correspondence between speech sound and shape (Experiment 1), and whether such association can influence shape representation (Experiment 2). For the purpose of closely examining the role of the two factors — articulation and pitch — combined in speech acoustics, we generated two sets of 25 vowel stimuli — pitch-varying and pitch-constant sets. Both sets were generated by manipulating articulation — frontness and height of the tongue body’s positions — but differed in terms of whether pitch varied among the sounds within the same set. In Experiment 1, participants made a forced choice between a round and a spiky shape to indicate the shape better associated with each sound. Results showed that shape choice was modulated according to both articulation and pitch, and we therefore concluded that both factors play significant roles in sound–shape correspondence. In Experiment 2, participants reported their subjective experience of shape accompanied by vowel sounds by adjusting an ambiguous shape in the response display. We found that sound–shape correspondence exerts an effect on shape representation by modulating audiovisual interaction, but only in the case of pitch-varying sounds. Therefore, pitch information within vowel acoustics plays the leading role in sound–shape correspondence influencing shape representation. Taken together, our results suggest the importance of teasing apart the roles of articulation and pitch for understanding sound–shape correspondence.

A Novice's Biography of George Gaylord Simpson

2013 ◽  
Vol 32 (1) ◽  
pp. 1-8
Author(s):  
Léo Laporte
Keyword(s):  
Popular Culture ◽  
Large Body ◽  
Field Work ◽  
Sources Of Information ◽  
Personal Interview ◽  
Modern Evolutionary Synthesis ◽  
George Gaylord Simpson ◽  
High Level ◽  
Modern Evolutionary Theory

George G. Simpson (1902-1984) had an enormously successful career for some fifty years during the middle of the twentieth century. Owing to his great intellect, especially his deep analytic skill and broad synthetic insight as well as his single-minded persistence, he produced a large body of published work that became an integral part of modern evolutionary theory. His high level of scientific achievement can be gauged by the number and quality of his publications, his institutional affiliations, his honors and awards, and the recognition he received in mainstream popular culture.1 Because Simpson was arguably the leading paleontologist of the last century and a major contributor to the ‘modern evolutionary synthesis’ I informed him of my biographical interest and asked for a personal interview. I sought further permission to interview his family, colleagues, and former students. For all interviews, I prepared a dozen leading questions, but also allowed interviewees to decide what was important. I encouraged tangential remarks and hence surprising insights were revealed. If possible, I corroborated what the interviewees told me; I did not take everything at face value. I always kept the emphasis on the content and character of Simpson's scientific accomplishments, avoiding ‘psycho-biography.’ I visited archives for unpublished documents—relevant personal letters, photographs, notes, newspaper clippings—and checked school and university records. I traveled to places where he grew up, attended school, was employed, and did field work. I divided the work into stand-alone articles, beginning with the easier and more obvious ones. I published these serially so they could later be revised, reassembled, and crafted into the final larger, unified biography. Doing research and writing in this way, I kept the longer-term project moving forward, making necessary course corrections as I went along. I thus established my credibility, advanced the research, and expanded the sources of information. Piece-meal publication satisfied deans and made possible a graceful escape if the project stalled, or I lost interest. However, seeing my work in print further motivated me to complete the task. Reviews were very positive, but sales more disappointing. Un succès estime!? (Laporte 2000a).

Sign in / Sign up

Export Citation Format

Share Document