The Empathic Communication Analytical Framework (ECAF): A multimodal perspective on emotional communication in interpreter-mediated consultations

Empathic communication (EC) in healthcare occurs when patients express empathic opportunities, such as emotions, to which doctors respond empathically. This interactional process during which participants try to achieve specific communicative goals (e.g., seeking and displaying empathy) serves as a context in which doctors and patients perform verbal and nonverbal actions and collaboratively co-construct meaning. This applies to interpreter-mediated consultations (IMCs) too, where interpreters perform additional actions of a similar kind. However, there is a dearth of research on the ways in which participants perform these actions in the context of EC, and how these actions in turn help (re)shape the context of EC in IMCs (Theys et al., 2020). To date, any tools for studying EC investigate participants’ actions in isolation, without studying them in the context of EC or in relation to the participants’ awareness of their own and others’ ongoing interactions. In this article, we present the Empathic Communication Analytical Framework (ECAF). The tool draws on valid, complementary analytical tools that allow for a fine-grained, three-level multimodal analysis of interactions. The first level of analysis allows for instances of EC in spoken language IMCs to be identified and for participants’ verbal actions in the context of EC to be studied. The second level allows analysts to investigate participants’ verbal and nonverbal actions in the previously identified context of EC. The third level of analysis links the participants’ concurrent verbal and nonverbal (inter)actions to their levels of attention and awareness and shows how participants’ actions are shaped and in turn help to reshape the context of EC in IMCs. In this article, we present the various levels of the ECAF framework, discuss its application to real-life data, and adopt a critical stance towards its affordances and limitations by looking into one excerpt of EC in IMCs. It is shown that the three distinct yet interconnected levels of analysis in the ECAF framework allow participants’ concurrent multimodal interactions in the context of EC to be studied.

Sustaining Synchronous Interaction Effectiveness in Distance Writing Courses: A Mixed Method Study in a KSA University

The sudden transition to online learning during the COVID-19 pandemic has been challenging for many learners and teachers due to the fact that most universities suddenly shifted to online learning without providing adequate time for preparing and training teachers and learners in using interactive educational technologies. Such challenges are even more pronounced for language instructors in cultivating and sustaining interactions among learners, especially in writing courses that demand active engagement and interactions. Therefore, this study focused on what and how a writing instructor did through technology in creating an interactive writing environment for KSA learners joining five online writing courses and how learners perceived interactions and identifies the major factors affecting their perceptions. The data were collected from multiple sources: WhatsApp chats, Google Docs chats and comments, screencast recorded discussions, students’ texts, and their responses to an electronic (e-) survey as well as follow-up interviews. The study revealed that in connecting Google Docs to the Blackboard Collaborate Ultra, the instructor engaged learners in multidirectional and multimodal interactions and text writing and revising. The WhatsApp group was also used for individual learner-learner and learner-teacher interaction illustrating support and consultation-seeking behaviors of learners beyond the online classroom time. The learners’ perceptions of technology-mediated interactions (overall, learner-learner and learner-teacher) in the online writing courses were at high levels, though such perceptions varied according to several factors, including socio-demographic characteristics. The study concludes by offering useful pedagogical and research implications.

Clarification and repair in emergency remote EFL classes

During the sudden shift in education onto digital platforms due to the Covid-19 emergency, teachers became streamers and experimented with new tools to involve their students in video-mediated, multi-floor, multiparticipant, and multimodal interactions. In turn, students experienced new ways to participate in lessons and interact with instructors. This study focuses on clarification and repair in videoconferencing as a strategy to address trouble in video-mediated communication and to re-establish mutual understanding. Through participant observation of online classes, the researcher collected data on classroom interactions, which are analyzed through conversation analysis. The findings show how the digital affordances of video-mediated conversation help teachers and students manage intersubjectivity and compensate for the lack of non-verbal cues typical in face-to-face interaction, such as facial expressions or tone of voice. Consequently, this article argues that the wisdom gained during the pandemic can help teachers and lecturers better deal with clarification and repair in digital conversations. Ultimately, it can increase their digital interactional competence, thus giving way to more interaction and learning in EFL classes, both online and in-person. Key words: EMERGENCY REMOTE EDUCATION, CONVERSATION ANALYSIS, CLARIFICATION, REPAIR, EFL Durante la migración hacia las plataformas digitales en la educación debido a la emergencia sanitaria del Covid-19, el profesorado se ha convertido en transmisor digital y ha experimentado con nuevas herramientas para implicar a su alumnado en conversaciones mediadas por vídeo multiparticipativas y multimodales. A su vez, el alumnado ha experimentado nuevas formas de participación en las clases y de interacción con el profesorado. Este estudio se centra en la aclaración y en la reparación en las videoconferencias como una estrategia para afrontar los problemas en la comunicación mediada por vídeo y restablecer el entendimiento mutuo entre docentes y estudiantes. Mediante la observación participante de las sesiones en línea, la investigadora recogió datos sobre las interacciones en clase que son analizados a través del análisis conversacional. Los resultados muestran cómo las posibilidades digitales de la conversación mediada por vídeo ayudan al profesorado y al alumnado a manejar la intersubjetividad y a compensar la falta de señales no verbales propias de la interacción cara a cara, como son las expresiones faciales o el tono de voz. En consecuencia, en este artículo se sostiene que el conocimiento adquirido durante la pandemia puede ayudar al profesorado a afrontar mejor la aclaración y la reparación en las conversaciones digitales. En última instancia, este conocimiento puede aumentar la competencia interactiva digital del profesorado dando lugar a una mayor interacción y a un mayor aprendizaje en las clases de inglés como lengua extranjera, tanto en línea como presenciales. Palabras clave: EDUCACIÓN REMOTA DE EMERGENCIA, ANÁLISIS DE LA CONVERSACIÓN, ACLARACIÓN, REPARACIÓN, EFL Durante l'improvvisa migrazione della didattica sulle piattaforme digitali dovuto all'emergenza Covid-19, i docenti sono diventati streamer e hanno sperimentato nuovi strumenti per interagire e coinvolgere i propri studenti in conversazioni mediate dal il video. A loro volta, gli studenti hanno sperimentato nuovi modi per partecipare alla lezione e interagire con i professori e fra loro. Questo studio si concentra sul chiarimento e la riparazione nella videoconferenza come strategie per affrontare i problemi nella comunicazione mediata dal video e ristabilire la comprensione reciproca. Attraverso l'osservazione partecipante delle lezioni online, sono stati raccolti dati sulle interazioni in classe e sono poi stati analizzati attraverso l'analisi della conversazione. I risultati mostrano come alcune caratteristiche della conversazione mediata dal video aiutino insegnanti e studenti a gestire l'intersoggettività e a compensare la mancanza di segnali non verbali tipici dell'interazione in presenza, come le espressioni facciali o il tono di voce. Si sostiene che la consapevolezza guadagnata durante la pandemia può aiutare i docenti a capire come affrontare il chiarimento e la riparazione nelle conversazioni digitali. Inoltre, può aumentare la loro competenza interattiva digitale, permettendo a una maggiore interazione e apprendimento nelle classi EFL, sia online che in presenza. Parole chiave: ISTRUZIONE A DISTANZA IN EMERGENZA, ANALISI DELLA CONVERSAZIONE, CHIARIMENTO, RIPARAZIONE, EFL

Preschool children’s mathematical arguments in play-based activities

The present study examines the structure and mathematical content of children’s mathematical arguments as part of communication in play-based activities. It shows how Nordin and Boistrup’s (The Journal of Mathematical Behavior 51:15–27, 2018) framework for identifying and reconstructing mathematical arguments, which includes Toulmin’s model of argumentation, the notion of anchoring (Lithner, Educational Studies in Mathematics 67:255–276, 2008) and a multimodal approach, can be used to identify and explore preschool children’s mathematical arguments. Two different types of argument that occurred during play-based activities were identified: partial arguments and full arguments. The findings reveal the extensive use of multimodal interactions in all parts of the children’s mathematical arguments. Moreover, the findings point to the crucial role of adults as dialogue collaborators in the argumentation that emerges in the play-based activities.

Deep Multimodal Emotion Recognition on Human Speech: A Review

This work reviews the state of the art in multimodal speech emotion recognition methodologies, focusing on audio, text and visual information. We provide a new, descriptive categorization of methods, based on the way they handle the inter-modality and intra-modality dynamics in the temporal dimension: (i) non-temporal architectures (NTA), which do not significantly model the temporal dimension in both unimodal and multimodal interaction; (ii) pseudo-temporal architectures (PTA), which also assume an oversimplification of the temporal dimension, although in one of the unimodal or multimodal interactions; and (iii) temporal architectures (TA), which try to capture both unimodal and cross-modal temporal dependencies. In addition, we review the basic feature representation methods for each modality, and we present aggregated evaluation results on the reported methodologies. Finally, we conclude this work with an in-depth analysis of the future challenges related to validation procedures, representation learning and method robustness.

MuMIA: Multimodal Interactions to Better Understand Art Contexts

Cultural heritage is a challenging domain of application for novel interactive technologies, where varying aspects in the way that cultural assets are delivered play a major role in enhancing the visitor experience, either onsite or online. Technology-supported natural human–computer interaction that is based on multimodalities is a key factor in enabling wider and enriched access to cultural heritage assets. In this paper, we present the design and evaluation of an interactive system that aims to support visitors towards a better understanding of art contexts through the use of a multimodal interface, based on visual and audio interactions. The results of the evaluation study shed light on the dimensions of evoking natural interactions within cultural heritage environments, using micro-narratives for self-exploration and understanding of cultural content, and the intersection between human–computer interaction and artificial intelligence within cultural heritage. We expect our findings to provide useful insights for practitioners and researchers of the broad human–computer interaction and cultural heritage communities on designing and evaluating multimodal interfaces to better support visitor experiences.

EREBOTS: Privacy-Compliant Agent-Based Platform for Multi-Scenario Personalized Health-Assistant Chatbots

Context. Asynchronous messaging is increasingly used to support human–machine interactions, generally implemented through chatbots. Such virtual entities assist the users in activities of different kinds (e.g., work, leisure, and health-related) and are becoming ingrained into humans’ habits due to factors including (i) the availability of mobile devices such as smartphones and tablets, (ii) the increasingly engaging nature of chatbot interactions, (iii) the release of dedicated APIs from messaging platforms, and (iv) increasingly complex AI-based mechanisms to power the bots’ behaviors. Nevertheless, most of the modern chatbots rely on state machines (implementing conversational rules) and one-fits-all approaches, neglecting personalization, data-stream privacy management, multi-topic management/interconnection, and multimodal interactions. Objective. This work addresses the challenges above through an agent-based framework for chatbot development named EREBOTS. Methods. The foundations of the framework are based on the implementation of (i) multi-front-end connectors and interfaces (i.e., Telegram, dedicated App, and web interface), (ii) enabling the configuration of multi-scenario behaviors (i.e., preventive physical conditioning, smoking cessation, and support for breast-cancer survivors), (iii) online learning, (iv) personalized conversations and recommendations (i.e., mood boost, anti-craving persuasion, and balance-preserving physical exercises), and (v) responsive multi-device monitoring interface (i.e., doctor and admin). Results. EREBOTS has been tested in the context of physical balance preservation in social confinement times (due to the ongoing pandemic). Thirteen individuals characterized by diverse age, gender, and country distribution have actively participated in the experimentation, reporting advancements in the physical balance and overall satisfaction of the interaction and exercises’ variety they have been proposed.

THE ROLE OF VIDEO CONFERENCING IN FOREIGN LANGUAGE LEARNING AMONG YOUNG STUDENTS

During the coronavirus pandemic research on video conferencing effects on young language learners is more than ever relevant. During the sharp transition, educators need to fully understand the benefits and possible pitfalls of video conferencing, but also overcome resistance to change through knowledge. As author Philip Pullman rightly points out, “You cannot change what you are, only what you do.” This article summarised key research and provided evidence for and against the use of video conferencing in foreign language education. Thus, it gives food for thought and reflection, as well as directions for possible interventions. However, the research was limited in terms of time and available literature, and the topic certainly requires more attention. In future, it is recommended to further research the use of multimodal interactions (sound, audio, text) in video conferencing, as well as coping strategies for dealing with the lack of interaction in the video conferencing-enabled classroom.