scholarly journals From Annotated Multimodal Corpora to Simulated Human-Like Behaviors

2008 ◽  
pp. 1-17 ◽  
Author(s):  
Matthias Rehm ◽  
Elisabeth André
Keyword(s):  
Multimodal Corpora

The FASil speech and multimodal corpora

2005 ◽  
Author(s):  
Hans Dolfing ◽  
David Reitter ◽  
Luís Almeida ◽  
Nuno Beires ◽  
Michael Cody ◽  
...  
Keyword(s):  
Multimodal Corpora

scholarly journals The future of multimodal corpora

2011 ◽  
Vol 11 (2) ◽  
pp. 391-415 ◽  
Author(s):  
Dawn Knight
Keyword(s):  
Corpus Linguistics ◽  
State Of The Art ◽  
Future Developments ◽  
The Past ◽  
Current State ◽  
Linguistic Research ◽  
Multimodal Corpora ◽  
The Future ◽  
Multimodal Corpus ◽  
Critical Overview

This paper takes stock of the current state-of-the-art in multimodal corpus linguistics, and proposes some projections of future developments in this field. It provides a critical overview of key multimodal corpora that have been constructed over the past decade and presents a wish-list of future technological and methodological advancements that may help to increase the availability, utility and functionality of such corpora for linguistic research.

scholarly journals Applications of multimodal concordances

2017 ◽  
Vol 21 (41) ◽  
pp. 11
Author(s):  
Anthony Baldry ◽  
Paul J. Thibault
Keyword(s):  
Foreign Language ◽  
Corpus Linguistics ◽  
Meaning Making ◽  
Multimodal Literacy ◽  
Multimodal Texts ◽  
Multimodal Discourse ◽  
Language Tests ◽  
Multimodal Corpora ◽  
Multimodal Corpus ◽  
Textual Organization

Multimodal corpus linguistics has so far been a theoretical rather than an applicative discipline. This paper sketches out proposals that attempt to bridge between these two perspectives. It does so with particular reference to the development of the conceptual and software tools required to create and concordance multimodal corpora from the applicative standpoint and as such is designed to underpin the study of texts at universities in foreign-language teaching and testing cycles. One branch of this work relates to multimedia language tests which, as illustrated in Section 2, use concordancing techniques to analyze multimodal texts in relation to students’ understanding of oral and written forms of discourse in English. Another branch is the exploration of multimodal tests concerned with the explicit assessment of students’ knowledge of the principles and/or models of textual organization of multimodal texts. The two types of test are not mutually exclusive. A third branch of research thus relates to the development of hybrid tests which, for example, combine a capacity to analyze multimodal texts with an assessment of students’ language skills, such as fluency in speaking and writing in English or which ascertain the multimodal literacy competencies of students and the differing orientations to meaning-making styles that individuals manifest. The paper considers these different applicative perspectives by describing the different categories of concordance achievable with the MCA online concordancer (Section 2) and by defining their relevance to multimodal discourse analysis (Section 3). It also illustrates the use of meaning-oriented multimodal concordances in the creation and implementation of multimodal tests (Sections 4). It concludes by suggesting that the re-interpretation of the nature and functions of concordances is long overdue and that the exploration of new types of concordance is salutary for linguistics and semiotics in general.

scholarly journals Analytical review of automatic systems for depression detection by speech

2021 ◽  
Vol 20 (3) ◽  
pp. 497-529
Author(s):  
Alena Velichko ◽  
Alexey Karpov
Keyword(s):  
Human Life ◽  
Binary Classification ◽  
Inverse Correlation ◽  
Psychomotor Retardation ◽  
Mental Illnesses ◽  
Emotional States ◽  
Multimodal Systems ◽  
Multimodal Corpora ◽  
Depression Detection ◽  
Analytical Review

In recent years the interest in automatic depression detection has grown within medical and scientific-technical communities. Depression is one of the most widespread mental illnesses that affects human life. In this review we present and analyze the latest researches devoted to depression detection. Basic notions related to the definition of depression were specified, the review includes both unimodal and multimodal corpora containing records of informants diagnosed with depression and control groups of non-depressed people. Theoretical and practical researches which present automated systems for depression detection were reviewed. The last ones include unimodal as well as multimodal systems. A part of reviewed systems addresses the challenge of regressive classification predicting the degree of depression severity (non-depressed, mild, moderate and severe), and another part solves a problem of binary classification predicting the presence of depression (if a person is depressed or not). An original classification of methods for computing of informative features for three communicative modalities (audio, video, text information) is presented. New methods for depression detection in every modality and all modalities in total are defined. The most popular methods for depression detection in reviewed studies are neural networks. The survey has shown that the main features of depression are psychomotor retardation that affects all communicative modalities and strong correlation with affective values of valency, activation and domination, also there has been observed an inverse correlation between depression and aggression. Discovered correlations confirm interrelation of affective disorders and human emotional states. The trend observed in many reviewed papers is that combining modalities improves the results of depression detection systems.

Utilizing Language Technology in the Documentation of Endangered Uralic Languages

2016 ◽  
Vol 4 ◽  
Author(s):  
Ciprian Gerstenberger ◽  
Niko Partanen ◽  
Michael Rießler ◽  
Joshua Wilbur
Keyword(s):  
Multimedia Data ◽  
Future Research ◽  
Language Documentation ◽  
Language Technology ◽  
Speech Communities ◽  
Multimodal Corpora ◽  
Finite State ◽  
Language Data ◽  
Data Collections ◽  
Morphosyntactic Analysis

The paper describes work-in-progress by the Pite Saami, Kola Saami and Izhva Komi language documentation projects, all of which record new spoken language data, digitize available recordings and annotate these multimedia data in order to provide comprehensive language corpora as databases for future research on and for endangered – and under-described – Uralic speech communities. Applying language technology in language documentation helps us to create more systematically annotated corpora, rather than eclectic data collections. Specifically, we describe a script providing interactivity between different morphosyntactic analysis modules implemented as Finite State Transducers and ELAN, a Graphical User Interface tool for annotating and presenting multimodal corpora. Ultimately, the spoken corpora created in our projects will be useful for scientifically significant quantitative investigations on these languages in the future.

Multimodal Event Construals: The Role of Co-Speech Gestures in English vs. Czech Interactions

2020 ◽  
Vol 68 (4) ◽  
pp. 351-377
Author(s):  
Jakub Jehlička ◽  
Eva Lehečková
Keyword(s):  
Multivariate Analysis ◽  
Cognitive Model ◽  
Linguistic Study ◽  
Fine Grained ◽  
Multiple Parameters ◽  
Multimodal Corpora

AbstractAspectuality of events has been shown to be construed through various means in typologically diverse languages, ranging from mainly grammatical devices to conventionalized lexical means. The rise of multimodal studies in linguistics allows incorporating yet another semiotic layer into the description. In this context, we present a cross-linguistic study of multimodal event construals in Czech and English spontaneous conversations, based on multimodal corpora. We follow Croft’s (2012) cognitive model of aspectual types, in order to take into account multiple parameters (out of which the features of (un)boundedness and directedness are the most prominent) determining a particular aspectual contour of a verb in a given context. We investigate which feature combinations are associated with (un)boundedness of corresponding co-speech gestures. The multivariate analysis revealed that in English, gesture boundedness is predicted by the predicate’s general aspectual type, whereas in Czech, the more fine-grained features of directedness and incrementality are stronger predictors.

Variation and Multimodal Corpora

Pragmatics ◽  
2020 ◽  
pp. 153-159
Author(s):  
Joan Cutting ◽  
Kenneth Fordyce
Keyword(s):  
Multimodal Corpora

Inclusive ethnographies

2017 ◽  
Vol 3 (3) ◽  
pp. 306-326 ◽  
Author(s):  
Tamás Péter Szabó ◽  
Robert A. Troyer
Keyword(s):  
Video Recording ◽  
Verbal Interaction ◽  
Space And Place ◽  
Sense Making ◽  
Linguistic Landscape ◽  
Place Making ◽  
Landscape Studies ◽  
Multimodal Corpora ◽  
Photo Camera ◽  
Analyze Data

Abstract In ethnographically oriented linguistic landscape studies, social spaces are studied in co-operation with research participants, many times through mobile encounters such as walking. Talking, walking, photographing and video recording as well as writing the fieldwork diary are activities that result in the accumulation of heterogeneous, multimodal corpora. We analyze data from a Hungarian school ethnography project to reconstruct fieldwork encounters and analyze embodiment, the handling of devices (e.g. the photo camera) and verbal interaction in exploratory, participant-led walking tours. Our analysis shows that situated practices of embodied conduct and verbal interaction blur the boundaries between observation and observers, and thus LL research is not only about space- and place-making and sense-making routines, but the fieldwork encounters are also transformative and contribute to space- and place-making themselves. Our findings provide insight for ethnographic researchers and enrich the already robust qualitative and quantitative strategies employed in the field.

Sign in / Sign up

Export Citation Format

Share Document