Real-Time Coordination in Human-Robot Interaction Using Face and Voice

When humans interact and collaborate with each other, they coordinate their turn-taking behaviors using verbal and nonverbal signals, expressed in the face and voice. If robots of the future are supposed to engage in social interaction with humans, it is essential that they can generate and understand these behaviors. In this article, I give an overview of several studies that show how humans in interaction with a humanlike robot make use of the same coordination signals typically found in studies on human-human interaction, and that it is possible to automatically detect and combine these cues to facilitate real-time coordination. The studies also show that humans react naturally to such signals when used by a robot, without being given any special instructions. They follow the gaze of the robot to disambiguate referring expressions, they conform when the robot selects the next speaker using gaze, and they respond naturally to subtle cues, such as gaze aversion, breathing, facial gestures and hesitation sounds.

Download Full-text

Brain activity during reciprocal social interaction investigated using conversational robots as control condition

Philosophical Transactions of the Royal Society B Biological Sciences ◽

10.1098/rstb.2018.0033 ◽

2019 ◽

Vol 374 (1771) ◽

pp. 20180033 ◽

Cited By ~ 11

Author(s):

Birgit Rauchbauer ◽

Bruno Nazarian ◽

Morgane Bourhis ◽

Magalie Ochs ◽

Laurent Prévot ◽

...

Keyword(s):

Prefrontal Cortex ◽

Social Interaction ◽

Medial Prefrontal Cortex ◽

Brain Activity ◽

Blood Oxygen Level Dependent ◽

Human Robot Interaction ◽

Human Interaction ◽

Peripheral Blood Flow ◽

Robot Interaction ◽

Temporoparietal Junction

We present a novel functional magnetic resonance imaging paradigm for second-person neuroscience. The paradigm compares a human social interaction (human–human interaction, HHI) to an interaction with a conversational robot (human–robot interaction, HRI). The social interaction consists of 1 min blocks of live bidirectional discussion between the scanned participant and the human or robot agent. A final sample of 21 participants is included in the corpus comprising physiological (blood oxygen level-dependent, respiration and peripheral blood flow) and behavioural (recorded speech from all interlocutors, eye tracking from the scanned participant, face recording of the human and robot agents) data. Here, we present the first analysis of this corpus, contrasting neural activity between HHI and HRI. We hypothesized that independently of differences in behaviour between interactions with the human and robot agent, neural markers of mentalizing (temporoparietal junction (TPJ) and medial prefrontal cortex) and social motivation (hypothalamus and amygdala) would only be active in HHI. Results confirmed significantly increased response associated with HHI in the TPJ, hypothalamus and amygdala, but not in the medial prefrontal cortex. Future analysis of this corpus will include fine-grained characterization of verbal and non-verbal behaviours recorded during the interaction to investigate their neural correlates. This article is part of the theme issue ‘From social brains to social robots: applying neurocognitive insights to human–robot interaction'.

Download Full-text

Gaze aversion in conversational settings: An investigation based on mock job interview

Journal of Eye Movement Research ◽

10.16910/jemr.14.1.1 ◽

2021 ◽

Vol 14 (1) ◽

Author(s):

Cengiz Acarturk ◽

Bipin Indurkya ◽

Piotr Nawrocki ◽

Bartlomiej Sniezynski ◽

Mateusz Jarosz ◽

...

Keyword(s):

Human Robot Interaction ◽

The Other ◽

Future Research ◽

Eye Tracker ◽

Robot Interaction ◽

Job Interview ◽

Gaze Aversion ◽

The Gaze ◽

Research Problems ◽

Interview Setting

We report the results of an empirical study on gaze aversion during dyadic human-to-human conversation in an interview setting. To address various methodological challenges in as- sessing gaze-to-face contact, we followed an approach where the experiment was conducted twice, each time with a different set of interviewees. In one of them the interviewer’s gaze was tracked with an eye tracker, and in the other the interviewee’s gaze was tracked. The gaze sequences obtained in both experiments were analyzed and modeled as Discrete-Time Markov Chains. The results show that the interviewer made more frequent and longer gaze contacts compared to the interviewee. Also, the interviewer made mostly diagonal gaze aversions, whereas the interviewee made sideways aversions (left or right). We discuss the relevance of this research for Human-Robot Interaction, and discuss some future research problems.

Download Full-text

The role and relationship of mindreading and social attunement in HRI – position statements of interdisciplinary researchers. Workshop HRI'20

10.31234/osf.io/3p6cd ◽

2020 ◽

Author(s):

Agnieszka Wykowska ◽

Jairo Pérez-Osorio ◽

Stefan Kopp

Keyword(s):

Mental States ◽

Human Robot Interaction ◽

Human Interaction ◽

Artificial Agents ◽

The Novel ◽

Robot Interaction ◽

Novel Coronavirus ◽

Relationship Of ◽

The Relationship

This booklet is a collection of the position statements accepted for the HRI’20 conference workshop “Social Cognition for HRI: Exploring the relationship between mindreading and social attunement in human-robot interaction” (Wykowska, Perez-Osorio & Kopp, 2020). Unfortunately, due to the rapid unfolding of the novel coronavirus at the beginning of the present year, the conference and consequently our workshop, were canceled. On the light of these events, we decided to put together the positions statements accepted for the workshop. The contributions collected in these pages highlight the role of attribution of mental states to artificial agents in human-robot interaction, and precisely the quality and presence of social attunement mechanisms that are known to make human interaction smooth, efficient, and robust. These papers also accentuate the importance of the multidisciplinary approach to advance the understanding of the factors and the consequences of social interactions with artificial agents.

Download Full-text