Multimodal Assemblies for Prefacing a Dispreferred Response: A Cross-Linguistic Analysis

In this paper we examine how participants’ multimodal conduct maps onto one of the basic organizational principles of social interaction: preference organization – and how it does so in a similar manner across five different languages (Czech, French, Hebrew, Mandarin, and Romanian). Based on interactional data from these languages, we identify a recurrent multimodal practice that respondents deploy in turn-initial position in dispreferred responses to various first actions, such as information requests, assessments, proposals, and informing. The practice involves the verbal delivery of a turn-initial expression corresponding to English ‘I don’t know’ and its variants (‘dunno’) coupled with gaze aversion from the prior speaker. We show that through this ‘multimodal assembly’ respondents preface a dispreferred response within various sequence types, and we demonstrate the cross-linguistic robustness of this practice: Through the focal multimodal assembly, respondents retrospectively mark the prior action as problematic and prospectively alert co-participants to incipient resistance to the constraints set out or to the stance conveyed by that action. By evidencing how grammar and body interface in related ways across a diverse set of languages, the findings open a window onto cross-linguistic, cross-modal, and cross-cultural consistencies in human interactional conduct.

Associations between Acoustic Features of Maternal Speech and Infants' Emotion Regulation following a Social Stressor

Caregiver voices may provide cues to mobilize or calm infants. This study examined whether maternal prosody predicted changes in infants' biobehavioral state during the Still Face, a stressor in which the mother withdraws and reinstates social engagement. Ninety-four dyads participated in the study (infant age 4-8 months). Infants' heart rate and respiratory sinus arrhythmia (measuring cardiac vagal tone) were derived from an electrocardiogram (ECG). Infants' behavioral distress was measured by negative vocalizations, facial expressions, and gaze aversion. Mothers' vocalizations were measured with spectral analysis and spectro-temporal modulation using a two-dimensional fast Fourier transformation of the audio spectrogram. High values on the maternal prosody composite were associated with decreases in infants' heart rate (β=-.26, 95% CI: [-.46, -.05]) and behavioral distress (β=- .20, 95% CI: [-.38, -.02]), and increases in cardiac vagal tone in infants whose vagal tone was low during the stressor (1 SD below mean β=.39, 95% CI: [.06, .73]). High infant heart rate predicted increases in the maternal prosody composite (β=.18, 95% CI: [.03, .33]). These results suggest specific vocal acoustic features of speech that are relevant for regulating infants' biobehavioral state and demonstrate mother-infant bi-directional dynamics.

Processing Mechanisms Of Eye-Head Cues And Eye-Finger-Pointing Cues In The Dot-Perspective Task

<p><b>Calculating others’ visual perspective automatically is a pivotal ability in human’s social communications. In the dot-perspective task, the ability is shown as a consistency effect: adults respond more slowly to judge the number of discs that they can see when a computer-generated avatar sees fewer discs. The implicit mentalising account attributes the effect to relatively automatic tracking of others’ visual perspectives. However, the submentalising account attributes the effect to domain-general attentional orienting. Accordingly, three studies were conducted to elucidate the ongoing implicit mentalising vs. submentalising debate.</b></p> <p>Study 1 (comprising Experiments 1 and 2) replicated the consistency effect either when real-human-face or spatial layout of discs was considered. Study 2 (comprising of Experiments 3 and 4) dissociated two accounts by manipulating real human’s facial cues. In Experiment 3, using a new visual access manipulation (i.e., a black rectangle placed on an agent’s eyes for rendering an invisible condition), a consistency effect was induced for eyes-opened but not eyes-covered faces with head direction, suggesting implicit mentalising. Experiment 4 firstly compared implicit mentalising (via consistency effect in the dot-perspective task) with attentional orienting (via a cue-validity effect in Posner task) when manipulating eye-head cues (head-front-gaze-averted versus head-turned-gaze-maintained). Neither effect was modulated by eye-head-related directional cue, but the cue-validity effect’s elicitation seemed to be related to the directional cue’s dynamic property. Overall, implicit mentalising as revealed in consistency effect cannot be purely reduced to attentional-orienting-related submentalising processes.</p> <p>Study 3 (comprising of Experiments 5 to 7) further clarified the debate by considering the agent’s different body cues. Experiment 5 extended the findings of Experiment 4 by generating a new eye-head-cue comparison (head-front-gaze-averted vs. head-turned-gaze-averted). Directional cue modulated cue-validity effect but not consistency effect, favouring Study 2’s conclusion. Experiment 6 adopted a new body-cue-manipulation (gaze-averted vs. finger-pointing). Both cue-validity and consistency effects were elicited for finger-pointing but not gaze-averted agents, supporting submentalising. Experiment 7 combined finger-pointing with visual access’s manipulation (eyes-opened vs. eyes-covered) on the dot-perspective task. Visual access did not modulate the consistency effect when finger-pointing was simultaneously displayed, supporting submentalising. Altogether, gaze aversion cues appear to play a dominant role in moderating implicit mentalising on the dot-perspective task, but the process may be interfered by the easily-discriminable finger-pointing cues via an attentional orienting mechanism.</p>

PROCESSING MECHANISMS OF EYE-HEAD CUES AND EYE-FINGER-POINTING CUES IN THE DOT-PERSPECTIVE TASK

<p><b>Calculating others’ visual perspective automatically is a pivotal ability in human’s social communications. In the dot-perspective task, the ability is shown as a consistency effect: adults respond more slowly to judge the number of discs that they can see when a computer-generated avatar sees fewer discs. The implicit mentalising account attributes the effect to relatively automatic tracking of others’ visual perspectives. However, the submentalising account attributes the effect to domain-general attentional orienting. Accordingly, three studies were conducted to elucidate the ongoing implicit mentalising vs. submentalising debate.</b></p> <p>Study 1 (comprising Experiments 1 and 2) replicated the consistency effect either when real-human-face or spatial layout of discs was considered. Study 2 (comprising of Experiments 3 and 4) dissociated two accounts by manipulating real human’s facial cues. In Experiment 3, using a new visual access manipulation (i.e., a black rectangle placed on an agent’s eyes for rendering an invisible condition), a consistency effect was induced for eyes-opened but not eyes-covered faces with head direction, suggesting implicit mentalising. Experiment 4 firstly compared implicit mentalising (via consistency effect in the dot-perspective task) with attentional orienting (via a cue-validity effect in Posner task) when manipulating eye-head cues (head-front-gaze-averted versus head-turned-gaze-maintained). Neither effect was modulated by eye-head-related directional cue, but the cue-validity effect’s elicitation seemed to be related to the directional cue’s dynamic property. Overall, implicit mentalising as revealed in consistency effect cannot be purely reduced to attentional-orienting-related submentalising processes.</p> <p>Study 3 (comprising of Experiments 5 to 7) further clarified the debate by considering the agent’s different body cues. Experiment 5 extended the findings of Experiment 4 by generating a new eye-head-cue comparison (head-front-gaze-averted vs. head-turned-gaze-averted). Directional cue modulated cue-validity effect but not consistency effect, favouring Study 2’s conclusion. Experiment 6 adopted a new body-cue-manipulation (gaze-averted vs. finger-pointing). Both cue-validity and consistency effects were elicited for finger-pointing but not gaze-averted agents, supporting submentalising. Experiment 7 combined finger-pointing with visual access’s manipulation (eyes-opened vs. eyes-covered) on the dot-perspective task. Visual access did not modulate the consistency effect when finger-pointing was simultaneously displayed, supporting submentalising. Altogether, gaze aversion cues appear to play a dominant role in moderating implicit mentalising on the dot-perspective task, but the process may be interfered by the easily-discriminable finger-pointing cues via an attentional orienting mechanism.</p>

PROCESSING MECHANISMS OF EYE-HEAD CUES AND EYE-FINGER-POINTING CUES IN THE DOT-PERSPECTIVE TASK

<p><b>Calculating others’ visual perspective automatically is a pivotal ability in human’s social communications. In the dot-perspective task, the ability is shown as a consistency effect: adults respond more slowly to judge the number of discs that they can see when a computer-generated avatar sees fewer discs. The implicit mentalising account attributes the effect to relatively automatic tracking of others’ visual perspectives. However, the submentalising account attributes the effect to domain-general attentional orienting. Accordingly, three studies were conducted to elucidate the ongoing implicit mentalising vs. submentalising debate.</b></p> <p>Study 1 (comprising Experiments 1 and 2) replicated the consistency effect either when real-human-face or spatial layout of discs was considered. Study 2 (comprising of Experiments 3 and 4) dissociated two accounts by manipulating real human’s facial cues. In Experiment 3, using a new visual access manipulation (i.e., a black rectangle placed on an agent’s eyes for rendering an invisible condition), a consistency effect was induced for eyes-opened but not eyes-covered faces with head direction, suggesting implicit mentalising. Experiment 4 firstly compared implicit mentalising (via consistency effect in the dot-perspective task) with attentional orienting (via a cue-validity effect in Posner task) when manipulating eye-head cues (head-front-gaze-averted versus head-turned-gaze-maintained). Neither effect was modulated by eye-head-related directional cue, but the cue-validity effect’s elicitation seemed to be related to the directional cue’s dynamic property. Overall, implicit mentalising as revealed in consistency effect cannot be purely reduced to attentional-orienting-related submentalising processes.</p> <p>Study 3 (comprising of Experiments 5 to 7) further clarified the debate by considering the agent’s different body cues. Experiment 5 extended the findings of Experiment 4 by generating a new eye-head-cue comparison (head-front-gaze-averted vs. head-turned-gaze-averted). Directional cue modulated cue-validity effect but not consistency effect, favouring Study 2’s conclusion. Experiment 6 adopted a new body-cue-manipulation (gaze-averted vs. finger-pointing). Both cue-validity and consistency effects were elicited for finger-pointing but not gaze-averted agents, supporting submentalising. Experiment 7 combined finger-pointing with visual access’s manipulation (eyes-opened vs. eyes-covered) on the dot-perspective task. Visual access did not modulate the consistency effect when finger-pointing was simultaneously displayed, supporting submentalising. Altogether, gaze aversion cues appear to play a dominant role in moderating implicit mentalising on the dot-perspective task, but the process may be interfered by the easily-discriminable finger-pointing cues via an attentional orienting mechanism.</p>

A prosocial character of head-gaze aversion in common marmosets

Gaze aversion is a behavior adopted by several mammalian and non-mammalian species in response to eye contact and usually interpreted as reaction to perceived threat. Unlike many other primates, common marmosets (Callithrix jacchus) are thought to have high tolerance for direct gaze, barely exhibiting gaze avoidance towards conspecifics and humans. Here we show that this does not hold for marmosets interacting with a familiar experimenter who suddenly establishes eye contact in a playful interaction (peek-a-boo game). In video footage synchronously recorded from the two agents, we found that our monkeys consistently alternated between eye contact and head-gaze aversion. The striking similarity with the gaze aversion dynamics exhibited by human infants interacting with their caregivers suggests a shared behavioral strategy to disengage temporarily from overwhelming social stimulation, in order to prepare for a new round of rewarding, affiliative face-to-face interaction. The potential of our finding for a marmoset model of autism is discussed.

Gaze aversion in conversational settings: An investigation based on mock job interview

We report the results of an empirical study on gaze aversion during dyadic human-to-human conversation in an interview setting. To address various methodological challenges in as- sessing gaze-to-face contact, we followed an approach where the experiment was conducted twice, each time with a different set of interviewees. In one of them the interviewer’s gaze was tracked with an eye tracker, and in the other the interviewee’s gaze was tracked. The gaze sequences obtained in both experiments were analyzed and modeled as Discrete-Time Markov Chains. The results show that the interviewer made more frequent and longer gaze contacts compared to the interviewee. Also, the interviewer made mostly diagonal gaze aversions, whereas the interviewee made sideways aversions (left or right). We discuss the relevance of this research for Human-Robot Interaction, and discuss some future research problems.

Autistic Adults May Be Erroneously Perceived as Deceptive and Lacking Credibility

We hypothesized that autistic adults may be erroneously judged as deceptive or lacking credibility due to demonstrating unexpected and atypical behaviors. Thirty autistic and 29 neurotypical individuals participated in video-recorded interviews, and we measured their demonstration of gaze aversion, repetitive body movements, literal interpretation of figurative language, poor reciprocity, and flat affect. Participants (N = 1410) viewed one of these videos and rated their perception of the individual’s truthfulness or credibility. The hypothesis was partially supported, with autistic individuals perceived as more deceptive and less credible than neurotypical individuals when telling the truth. However, this relationship was not influenced by the presence of any of the target behaviors, but instead, by the individual’s overall presentation.

Eye contact perception in high-functioning adults with autism spectrum disorder

The present study investigated how the eye contact perception of ingroup and outgroup faces by Japanese adults with high-functioning autism spectrum disorder differed from that of age-, sex-, and IQ-matched typically developing individuals. The autism spectrum disorder and typically developing individuals were equally likely to perceive subtly averted gazes as self-directed gazes. In both groups, the frequency with which self-directed gazes were perceived decreased as gaze aversion increased. In general, individuals with autism spectrum disorder were equally capable of perceiving a self-directed gaze as typically developing individuals. However, typically developing individuals, but not individuals with autism spectrum disorder, were more likely to perceive self-directed gazes from ingroup faces than from outgroup faces. Stimuli ratings revealed that individuals with autism spectrum disorder, but not those with typically developing, gave higher warmth ratings to ingroup faces with averted gazes and outgroup faces with direct gazes compared to other types of face stimuli, suggesting atypical affective experiences in response to ingroup and outgroup faces in autism spectrum disorder. These results suggest that individuals with autism spectrum disorder did not show an ingroup bias for the perception of a self-directed gaze, and raise the possibility that an atypical emotional experience contributes to the diminished ingroup bias for the perception of a self-directed gaze. Lay abstract The detection of a self-directed gaze is often the starting point for social interactions and a person who feels as if they are being watched can prepare to respond to others’ actions irrespective of the real gaze direction because the other person may likely be motivated to approach. Although many studies demonstrated that individuals with autism spectrum disorder have difficulty discriminating gaze direction, it remains unclear how the perception of self-directed gaze by individuals with autism spectrum disorder differs from that of age-, sex-, and IQ-matched typically developing individuals. Participants observed faces with various gaze directions and answered whether the person in the photograph was looking at them or not. Individuals with and without autism spectrum disorder were just as likely to perceive subtle averted gazes as self-directed gazes. The frequency of perceiving a self-directed gaze decreased as gaze aversion increased in both groups and, in general, individuals with autism spectrum disorder showed a comparable ability to perceive a self-directed gaze as that of typically developing individuals. Interestingly, considering face membership of photographs (ingroup or outgroup faces), typically developing individuals, but not individuals with autism spectrum disorder, were more likely to perceive self-directed gazes from ingroup faces than from outgroup faces. However, individuals with autism spectrum disorder had different affective experiences in response to ingroup and outgroup faces. These results suggest that individuals with autism spectrum disorder did not show an ingroup bias for the perception of a self-directed gaze, and raise a possibility that an atypical emotional experience contributes to the diminished ingroup bias.