How much is a cow like a meow? A novel database of human judgements of audiovisual semantic relatedness

Semantic information about objects, events, and scenes influences how humans perceive, interact with, and navigate the world. Most evidence in support of semantic influence on cognition has been garnered from research conducted with an isolated modality (e.g., vision, audition). However, the influence of semantic information has not yet been extensively studied in multisensory environments potentially because of the difficulty in quantification of semantic relatedness. Past studies have primary relied on either a simplified binary classification of semantic relatedness based on category or on algorithmic values based on text corpora rather than human perceptual experience and judgement. With the aim to accelerate research into multisensory semantics, we created a constrained audiovisual stimulus set and derived similarity ratings between items within three categories (animals, instruments, household items). A set of 140 participants provided similarity judgments between sounds and images. Participants either heard a sound (e.g., a meow) and judged which of two pictures of objects (e.g., a picture of a dog and a duck) it was more similar to, or saw a picture (e.g., a picture of a duck) and selected which of two sounds it was more similar to (e.g., a bark or a meow). Judgements were then used to calculate similarity values of any given cross-modal pair. The derived and reported similarity judgements reflect a range of semantic similarities across three categories and items, and highlight similarities and differences among similarity judgments between modalities. We make the derived similarity values available in a database format to the research community to be used as a measure of semantic relatedness in cognitive psychology experiments, enabling more robust studies of semantics in audiovisual environments.

How Saudi migrants’ metapragmatic judgments of Arabic L1 nonverbal greetings change after intense and prolonged exposure to English

This study investigates the impact of spending more than three years in an English environment on Saudi migrants’ metapragmatic judgments of Arabic L1 nonverbal greetings and their personality traits. Participants are 437 adults comprising three groups: Saudi L2 speakers of English in the UK, Saudis in Saudi Arabia, and British L1 speakers of English in the UK. They observed and rated an audiovisual stimulus illustrating Saudi L1 nonverbal greeting behaviours of handshake and cheek-to-cheek kiss. Statistical analyses revealed that appropriateness ratings by Saudi migrants in the UK diverged from those by Saudis in Saudi Arabia and approximated those of English L1 users in the UK. Moreover, appropriateness ratings by Saudi migrants were differently associated with personality profiles, which differed for three traits between the two Saudi groups. These findings suggest change in L1 metapragmatic judgements as well as personality as a result of prolonged and intense exposure to an L2. The results are interpreted in the light of Cook’s (2012) concept of multicompetence.

Open multimodal iEEG-fMRI dataset from naturalistic stimulation with a short audiovisual film

Intracranial human recordings are a valuable and rare resource that the whole neuroscience community can benefit from. Making such data available to the neuroscience community not only helps tackle the reproducibility issues in science, it also helps make more use of this valuable data. The latter is especially true for data collected using naturalistic tasks. Here, we describe a dataset collected from a large group of human subjects while they watched a short audiovisual film. The dataset is characterized by several unique features. First, it combines a large amount of intracranial data from 51 intracranial electroencephalography (iEEG) participants, who all did the same task. Second, the intracranial data are accompanied by fMRI recordings acquired for the same task in 30 functional magnetic resonance imaging (fMRI) participants. Third, the data were acquired using a rich audiovisual stimulus, for which we provide detailed speech and video annotations. This multimodal dataset can be used to address questions about neural mechanisms of multimodal perception and language comprehension as well as the nature of the neural signal acquired during the same task across brain recording modalities.

Devaluation of a conditioned reinforcer requires its reexposure

A change in motivational state does not guarantee a change in operant behaviour. Only after an organism has had contact with an outcome while in a relevant motivational state does behaviour change, a phenomenon called incentive learning. While ample evidence indicates that this is true for primary reinforcers, it has not been established for conditioned reinforcers. We performed an experiment with rats where lever-presses were reinforced by presentations of an audiovisual stimulus that had previously preceded food delivery; in the critical experimental groups, the audiovisual stimulus was then paired a single time with a strong electric shock. Some animals were reexposed to the audiovisual stimulus. Lever-presses yielding no outcomes were recorded in a subsequent test. Animals that had been reexposed to the audiovisual stimulus after the aversive training responded less than did those that had not received reexposure. Indeed, those animals that were not reexposed did not differ from a control group that received no aversive conditioning of the audiovisual stimulus. Moreover, these results were not mediated by a change in the food’s reinforcement value, but instead reflect a change in behaviour with respect to the conditioned reinforcer itself. These are the first data to indicate that the affective value of conditioned stimuli, like that of unconditioned ones, is established when the organism comes into contact with them.

Brain networks are decoupled from external stimuli during internal cognition

Our cognition can be directed to external stimuli or to internal information. While there are many different forms of internal cognition (mind-wandering, recall, imagery etc.), their essential feature is independence from the immediate sensory input. This is thought to be reflected in the decoupling of brain networks from the external stimuli, but a quantitative investigation of this remains outstanding. Here we present a conceptual and analysis framework that links stimulus responses to connectivity between brain networks. This allows us to quantify the coupling of brain networks to the external stimuli. We tested this framework by presenting subjects with an audiovisual stimulus and instructing them to either attend to the stimulus (external task) or engage in mental imagery, recall or arithmetic (internal tasks) while measuring the evoked brain activity using functional MRI. We found that stimulus responses were generally attenuated for the internal tasks, though they increased in a subset of tasks and brain networks. However, using our new measure of coupling, we showed that brain networks became increasingly decoupled from the stimulus, even in the subset of tasks and brain networks in which stimulus responses increased. These results quantitatively demonstrate that during internal cognition brain networks are decoupled from external stimuli, opening the door for a fundamental and quantitative understanding of internal cognition.

Correlational network visual analysis of adolescents’ film entertainment responses

This study aims to shed light on adolescents’ characterizations of their preferred film entertainment. It seeks to describe the psychological responses of youngsters from two European countries (Italy and Spain) to dramas from the European region about current social/human issues. The study also seeks to determine if the adolescents’ responses differ according to the film and the cultural context (country) to which they belong. For doing so, the research applies an innovative visual research technique in Media Communication: a correlational network analysis. A total of 234 Italian and Spanish adolescents watched three European dramas. Later, they completed four questionnaires measuring hedonic (enjoyment), eudaimonic (appreciation) responses and predictors of them (narrative engagement and perceived realism). The results present the visual intercorrelations of the studied variables, which give deeper insights into youngsters’ gratifications in film consumption. These results suggest that films appear to be more influential than country in adolescents’ responses. They also show there are specific models of responses for each film in each situation. The application of the visual quantitative tool broadens our knowledge on adolescents’ entertainment through dramatic film and on the role of the cultural context and the audiovisual stimulus in the entertainment responses. It also challenges empirical studies using a single stimulus.

Augmenting aesthetic chills using a wearable prosthesis improves their downstream effects on reward and social cognition

Previous studies on aesthetic chills (i.e., psychogenic shivers) demonstrate their positive effects on stress, pleasure, and social cognition. We tested whether we could artificially enhance this emotion and its downstream effects by intervening on its somatic markers using wearable technology. We built a device generating cold and vibrotactile sensations down the spine of subjects in temporal conjunction with a chill-eliciting audiovisual stimulus, enhancing the somatosensation of cold underlying aesthetic chills. Results suggest that participants wearing the device experienced significantly more chills, and chills of greater intensity. Further, these subjects reported sharing the feelings expressed in the stimulus to a greater degree, and felt more pleasure during the experience. These preliminary results demonstrate that emotion prosthetics and somatosensory interfaces offer new possibilities of modulating human emotions from the bottom-up (body to mind). Future challenges will include testing the device on a larger sample and diversifying the type of stimuli to account for negatively valenced chills and intercultural differences. Interoceptive technologies offer a new paradigm for affective neuroscience, allowing controlled intervention on conscious feelings and their downstream effects on higher-order cognition.

Effects of Ensemble Size and Repertoire Difficulty on Ratings of Concert Band Performances

The purpose of this study was to examine effects of ensemble size and repertoire difficulty on listeners’ perceptions of concert band performances. Undergraduate music majors ( N = 210) viewed an audiovisual stimulus consisting of various images of large and small concert bands paired with identical audio performances of either an easy or difficult composition. Participants rated each ensemble’s tone quality/intonation, musicianship/expression, and rhythm/articulation using a 10-point Likert-type scale. Results indicated no main effects for ensemble size or order. There was a significant main effect for repertoire difficulty, with difficult repertoire being evaluated more positively than easier repertoire. We also found a significant Ensemble Size × Repertoire Difficulty × Order interaction, indicating that results were moderated based on order. Within the four orders, the largest mean difference in scores occurred in Order 3 (small/difficult, large/easy, small/easy, large/difficult), with the smallest mean differences occurring in Order 2 (large/difficult, small/easy, large/easy, small/difficult). The small/easy and large/easy videos and the small/difficult and large/difficult videos resulted in positive changes in ratings only when seen first and last, respectively. We recommend blind evaluation and the use of required “test pieces” in concert band festivals as ways to possibly mitigate effects of repertoire difficulty and ensemble size.