From X to Y: Anatomy of a Constructional Pattern

Compositionality is undoubtedly one of the hardest problems in linguistics. In decoding theories, the speaker occupies a leading role, having to carefully choose the form that better encodes the meaning to be communicated. In contrast, in inferential theories, the burden is shifted from speaker to hearer: linguistic information typically underspecifies meaning and the hearer must make a number of inferences to bridge the gap between what is said and what is meant. In this article, I argue that constructional meaning can aid the process of sentence meaning formation by providing a scaffold that can help the hearer with the construal operations. Constructions, by providing an additional layer of meaning, constrain the range of possible meanings activated by words thereby reducing the combinatorial explosion when several words are joined together. This process is examined here by analysing the meanings associated with the grammatical construction [from X to Y], which is connected to a polysemy network of related senses, using examples extracted from a multimodal corpus. A preliminary analysis of the gesturing behaviour associated with the different senses proposed is also included, which can be seen to contribute to the characterisation of the different senses of the polysemy network.

Engaging students in multimodal literacy practices in a university ESP context: towards understanding identity and ideology in government debates

This paper focuses on issues of multimodal literacy practices in ESP higher education settings. In particular, the research explores how students become engaged in various literacy activities aimed at enhancing their critical-thinking skills and interpretation of images. For this purpose, two datasets consisting of video clips were extracted from a larger multimodal corpus and developed for teaching applications: one involved a UK live parliament debate and the other a US House of Representatives debate. The main objective is to identify the key verbal strategies reflecting persuasive, argumentative rhetoric and the non-verbal features accompanying these verbal utterances such as prosodic stress, body/head movements, gaze, gesture. Thus, the focus of the analysis is on how different semiotic modes of communication construct meaning, especially in terms of how they reinforce the construction of identity and ideological stance. The results were systematically categorized and applied on a practical level to a teaching unit on ‘identity and ideology’.

ChiCo: A Multimodal Corpus for the Study of Child Conversation

The study of how children develop their conversational skills is an important scientific frontier at the crossroad of social, cognitive, and linguistic development with important applications in health, education, and child-oriented AI. While recent advances in machine learning techniques allow us to develop formal theories of conversational development in real-life contexts, progress has been slowed down by the lack of corpora that both approximate naturalistic interaction and provide clear access to children’s non-verbal behavior in face-to-face conversations. This work is an effort to fill this gap. We introduce ChiCo (for Child Conversation), a corpus we built using an online video chat system. Using a weakly structured task (a word-guessing game), we recorded 20 conversations involving either children in middle childhood (i.e., 6 to 12 years old) interacting with their caregivers (condition of interest) or the same caregivers interacting with other adults (a control condition), resulting in 40 individual recordings. Our annotation of these videos has shown that the frequency of children’s use of gaze, gesture, and facial expressions mirrors that of adults. Future modeling research can capitalize on this rich behavioral data to study how both verbal and non-verbal cues contribute to the development of conversational coordination

Transmediatisation of the Covid-19 crisis in Brazil: The emergence of (bio-/geo-)political repertoires of (re-)interpretation

This paper focuses on the centrality of media practices to discuss in a transdisciplinary way how the Covid-19 crisis has been framed and communicated in Brazil across different media spaces, whereas the country became the second in the world with most deaths due to the spread of the Covid-19 infection. This discussion mobilises Foucauldian genealogical and critical discursive perspectives (Foucault, [1970]1981, p. 73) to create intelligibility about how domains of media power-knowledge, such as professional journalism and social media, generate textual trajectories, discursive-semiotic and epistemic disputes through (re-)framings. Based on oligoptic decisions (Latour, 2005, p. 182; Souza Júnior, 2020, pp. 59–64), the article explores a multimodal corpus of transmedia texts. 11 posts have been selected for discussion, since they pave the way for tracing a set of interconnected and (in-)visible elements about the corona crisis. In turn, this paper seeks to: (i) give visibility and qualitatively discuss some of the perspectives that have circulated across media spaces along with their related repertoires of biopolitical and geopolitical (re-)interpretation and (ii) expose the dynamics of power and resistance that emerge through pandemic frames, and how the latter communicate a social event like the Covid-19 crisis.

Distant time, distant gesture: speech and gesture correlate to express temporal distance

This study investigates whether there is a relation between the semantics of linguistic expressions that indicate temporal distance and the spatial properties of their co-speech gestures. To this date, research on time gestures has focused on features such as gesture axis, direction, and shape. Here we focus on a gesture property that has been overlooked so far: the distance of the gesture in relation to the body. To achieve this, we investigate two types of temporal linguistic expressions are addressed: proximal (e.g., near future, near past) and distal (e.g., distant past, distant future). Data was obtained through the NewsScape library, a multimodal corpus of television news. A total of 121 co-speech gestures were collected and divided into the two categories. The gestures were later annotated in terms of gesture space and classified in three categories: (i) center, (ii) periphery, and (iii) extreme periphery. Our results suggest that gesture and language are coherent in the expression of temporal distance: when speakers locate an event far from them, they tend to gesture further from their body; similarly, when locating an event close to them, they gesture closer to their body. These results thus reveal how co-speech gestures also reflect a space-time mapping in the dimension of distance.

A multimodal corpus of simulated consultations between a patient and multiple healthcare professionals

Language resources for studying doctor–patient interaction are rare, primarily due to the ethical issues related to recording real medical consultations. Rarer still are resources that involve more than one healthcare professional in consultation with a patient, despite many chronic conditions requiring multiple areas of expertise for effective treatment. In this paper, we present the design, construction and output of the Patient Consultation Corpus, a multimodal corpus of simulated consultations between a patient portrayed by an actor, and at least two healthcare professionals with different areas of expertise. As well as the transcribed text from each consultation, the corpus also contains audio and video where for each consultation: the audio consists of individual tracks for each participant, allowing for clear identification of speakers; the video consists of two framings for each participant—upper-body and face—allowing for close analysis of behaviours and gestures. Having presented the design and construction of the corpus, we then go on to briefly describe how the multi-modal nature of the corpus allows it to be analysed from several different perspectives.