scholarly journals What can cognitive linguistics tell us about language-image relations? A multidimensional approach to intersemiotic convergence in multimodal texts

2021 ◽  
Vol 0 (0) ◽  
Author(s):  
Christopher Hart ◽  
Javier Marmol Queralto
Keyword(s):  
Cognitive Linguistics ◽  
Online News ◽  
Event Structure ◽  
Multidimensional Approach ◽  
Symbol Manipulation ◽  
Multimodal Texts ◽  
Specific Focus ◽  
Political Protests ◽  
Multimodal Text ◽  
Text Images

Abstract In contrast to symbol-manipulation approaches, Cognitive Linguistics offers a modal rather than an amodal account of meaning in language. From this perspective, the meanings attached to linguistic expressions, in the form of conceptualisations, have various properties in common with visual forms of representation. This makes Cognitive Linguistics a potentially useful framework for identifying and analysing language-image relations in multimodal texts. In this paper, we investigate language-image relations with a specific focus on intersemiotic convergence. Analogous with research on gesture, we extend the notion of co-text images and argue that images and language usages which are proximal to one another in a multimodal text can be expected to exhibit the same or consistent construals of the target scene. We outline some of the dimensions of conceptualisation along which intersemiotic convergence may be enacted in texts, including event-structure, viewpoint, distribution of attention and metaphor. We take as illustrative data photographs and their captions in online news texts covering a range of topics including immigration, political protests, and inter-state conflict. Our analysis suggests the utility of Cognitive Linguistics in allowing new potential sites of intersemiotic convergence to be identified and in proffering an account of language-image relations that is based in language cognition.

Interactions, intersections and improvisations: Studying the multimodal texts and classroom talk of six- to seven-year-olds

2009 ◽  
Vol 9 (2) ◽  
pp. 188-210 ◽  
Author(s):  
Kate Pahl
Keyword(s):  
Classroom Talk ◽  
Analytic Framework ◽  
Multimodal Texts ◽  
Multimodal Text ◽  
The Relationship ◽  
Ways Of Being

This article examines the relationship between children's talk in the classroom and their multimodal texts. The article uses an analytic framework derived from Bourdieu's concept of habitus to examine how 6—7-year-old children's regular ways of being and doing can be found in their multimodal texts together with their talk (Bourdieu, 1977, 1990). The concept of pedagogic habitus is used to make sense of the teacher's regular ways of being and doing within the classroom (Grenfell, 1996). Improvisations upon these ways of being and doing were considered with reference to data collected over two years. In this article, the term `multimodal text' refers to panorama boxes created from shoe boxes to represent an environment such as the ocean or a jungle. The article concludes that it is important to pay attention to the interrelationship between the talk and the boxes to make sense of children's multimodal texts. The concept of improvisations upon the habitus provides an important context for this understanding.

scholarly journals A Social Semiotic Approach to the Attitudinal Meanings in Multimodal Texts

2019 ◽  
Vol 9 (9) ◽  
pp. 1160
Author(s):  
Yong Hu ◽  
Qing Qiu
Keyword(s):  
Picture Books ◽  
Multimodal Texts ◽  
Visual Narratives ◽  
Visual Semiotics ◽  
Semiotic Approach ◽  
Semiotic Resources ◽  
Multimodal Text ◽  
The Creation

As a special type of multimodal text, picture books for children are highly valued in the creation of meaning by the integrative use of verbal and visual semiotic resources. Informed by Painter and Martin’s framework of visual narratives, this paper primarily deals with the interpersonal meanings encoded and expressed by the two semiotics (image and verbiage) within the Chinese picture books. It aims to analyse the visual and verbal choices available for writers to establish engagement between various participants. In the hope of investigating the collaboration and interplay of verbal and visual semiotics to construe interpersonal meanings, it examines the attitudinal meanings inscribed or invoked in picture books, exploring the ways in which visual and verbal resources are co-instantiated to encode attitudinal convergence and also divergence.

Disabled Moves

2016 ◽  
Author(s):  
Marianne Kielian-Gilbert
Keyword(s):  
Internal Representation ◽  
Multidimensional Approach ◽  
Symbol Manipulation ◽  
Musical Identity ◽  
Critical Potential ◽  
Music Experience

If music and music experience are more than internal representation and symbol manipulation, how might one flesh out and understand their multidimensionality? This essay describes “disabled moves” toward that question by imagining and disturbing what it might mean to think and be through the variant body. First, it considers the implications of the unexamined and assumed centricity of the abled body and the kinds of material-bodily-mental interventions that might unfix or jostle (musical) identity with respect to that centricity. Second, it considers the potential and limitations of medical and aural metaphors in Ethel Smyth’s Concerto for Horn, Violin and Orchestra (1926–1927). Third, it stages musical “theatres of madness,” juxtaposing descriptive listening accounts of Marta Ptaszyńska’s “Thorn Trees” from her Concerto for Marimba and Orchestra, to suggest a multidimensional approach to analysis that works with side-by-side encounters and the critical potential and desire for change.

Multimodal Legitimation Strategies on TV Cooking Shows

2016 ◽  
Vol 5 (2) ◽  
Author(s):  
Keri Matwick
Keyword(s):  
Role Model ◽  
Emotional Responses ◽  
Social Practices ◽  
Communication Studies ◽  
Multimodal Texts ◽  
Television Shows ◽  
Semiotic Approach ◽  
Cooking Shows ◽  
Multimodal Text ◽  
Analytical Approaches

AbstractTelevision cooking shows have grown in popularity within the last two decades. As a media text, they reflect the surrounding culture and social practices and elicit various emotional responses in people. As a multimodal text, television shows utilize multiple modes to create meaning. Based on the view of cooking shows as a multimodal texts, this paper draws on Kress and Van Leeuwen’s social semiotic approach and examines how multimodal elements (linguistic, visual, sound, spatial, gestural) convey the authority of the tv host. In doing so, five different tactics from Van Leeuwen’s legitimation theory – personal, expert, role model, tradition, and conformity – of authority are identified and revealed. This paper provides an analysis of cooking shows that has resulted in a better understanding of the ways in which authority is constructed multimodally, and subsequently contributes to developing applications of multimodal analytical approaches in linguistic, cultural, and communication studies.

scholarly journals A Study on Effects of Multimodal Science Text Design on Meaning-Making of Science Content: Science Teachers’ Perspectives

2021 ◽  
Vol 8 (1) ◽  
pp. 1-20
Author(s):  
Zekai Ayık ◽  
Bayram Coştu
Keyword(s):  
Science Teachers ◽  
Meaning Making ◽  
Variation Theory ◽  
Science Content ◽  
Science Text ◽  
Structured Interviews ◽  
Multimodal Texts ◽  
Science Texts ◽  
Text Design ◽  
Multimodal Text

Numerous studies demonstrated that the meaning-making of scientific knowledge is affected by the design of multimodal science texts. Various modes are co-operated together in certain inter-semiotic mechanisms to produce meaning in multimodal texts. Based on this perspective, this research seeks to investigate the effect of mode level in science texts and compositional arrangement on the meaning-making of science concepts and processes. In this context, four science texts with the same content (transformation of energy) at different mode densities and two science texts with the same content (covalent bonding) one of which is arranged in accordance with variation theory of learning are designed. By using the case study method, this research explored six experienced science teachers’ views about the effects of mode level and multimodal text composition on meaning-making. The data were collected with semi-structured interviews. The thematic analysis was employed for data analysis. The findings demonstrated that mode density may affect meaning-making and so learning since different modes have affordance to represent different meaning and meaning relationship types. Besides, multimodal text composition may foreground the critical aspects of content, and help to design a coherent multimodal science text.

scholarly journals The role of multimodal metaphors in the creation of the fake news category: a proposal for analysis / A atuação das metáforas multimodais na criação da categoria fake news: uma proposta de análise

2020 ◽  
Vol 28 (2) ◽  
pp. 777
Author(s):  
Natália Elvira Sperandio
Keyword(s):  
Cognitive Linguistics ◽  
Meaning Construction ◽  
Fake News ◽  
Conceptual Metaphor Theory ◽  
Multimodal Texts ◽  
Specific Objective ◽  
Editorial Cartoons ◽  
Metaphor Theory ◽  
Analytical Category

Abstract: Almost forty years ago, the proposal of the Conceptual Metaphor Theory represented a milestone for Cognitive Linguistics studies. From this point, many pieces of research were developed around the analytical category of metaphor. However, the majority of these papers are still in the monomodal field, intending to build a concept resulting from the source and target domains that come specifically from the verbal structures. Taking this into consideration, this paper intends to focus on the metaphorical occurrence in diverse semiotic modes that constitute multimodal texts. In order to fulfill this aim, we outlined a corpus of five editorial cartoons about “fake news”. Our specific objective is to, making use of the concept of multimodal metaphors proposed by Forceville (1996, 2009), presented how the different semiotic modes, in this case the verbal and the visual ones, are interwoven in the building of these metaphors. In this way, in our study, besides validating the thesis proposed by Forceville (2009) about the occurrence of metaphors not only in the verbal mode, it was also possible to verify the importance of multimodal metaphors for the meaning construction process in the analyzed genre.Keywords: metaphors; multimodal metaphors; fake news.Resumo: Há quase quarenta anos atrás um marco nos estudos da Linguística Cognitiva, em especial em sua semântica, ocorreu: a proposta da Teoria da Metáfora Conceitual. A partir desse trabalho, muitas pesquisas foram desenvolvidas em torno da categoria analítica da metáfora. Porém, grande parte desses trabalhos ainda encontra-se no campo dos denominados textos monomodais, visando apenas a construção conceitual resultante de domínios fonte e alvo oriundos especificamente do modo verbal. Diante disso, o presente artigo propõe-se a promover um trabalho dedicado à ocorrência metafórica nos diferentes modos semióticos que constituem os textos multimodais. Para cumprirmos tal objetivo, delineamos como corpus cinco charges que versam sobre o conceito fake news. Nosso objetivo específico consiste em apresentar, através do conceito de metáforas multimodais, proposto por Forceville (1996, 2009), a forma pela qual diferentes modos semióticos, nesse caso em especial o verbal e o imagético, imbricaram-se na construção dessas metáforas. Assim, em nosso estudo, além de vislumbrarmos a validação da proposição de Forceville (2009) sobre a não ocorrência do processo metafórico apenas no modo verbal, foi possível a verificação da importância das metáforas multimodais para a construção dos sentidos do gênero em análise.Palavras-chave: metáforas; metáfora multimodal; fake news.

scholarly journals Film Novelization as Multimodal Translation

2019 ◽  
Vol 57 ◽  
pp. 125-136
Author(s):  
Beata Mazurek-Przybylska
Keyword(s):  
Translation Studies ◽  
Book Market ◽  
Translation Process ◽  
Literary Adaptation ◽  
Multimodal Texts ◽  
Image Schemata ◽  
Multimodal Text ◽  
Effective Analysis

Novelization, i.e. a literary adaptation of a film, despite its widespread presence on the book market, was treated as a merely commercial phenomenon, and until the late 1990s, it did not inspire any academics research. The main objective of this paper is to show that the phenomenon of novelization can offer new opportunities for linguistics and to reconsider the place of novelization in adaptation and translation studies. It is claimed that the process of film-to-book transformation can be called a translation process. The term multimodal translation is adopted since transforming a multimodal text film into a monomodal one book involves a change of modalities and their density. What follows is an attempt to propose tools that can be used for the effective analysis of multimodal translation, which involve the classical Aristotelian view of the three-part plot of verbal texts and Elżbieta Tabakowska’s theory of cognitive translation. In order to illustrate the film–book translation process, an Interstellar film segment and its book counterpart are analyzed and the conclusion has been drawn that both the film and the book units use the same orientational image schemata. These findings prove that the extension of Tabakowska’s theory to multimodal texts is an adequate framework for the comparison of a film and its novelization.

Digital stories: Bringing multimodal texts to the Spanish writing classroom

ReCALL ◽  
2016 ◽  
Vol 28 (3) ◽  
pp. 326-342 ◽  
Author(s):  
Ana Oskoz ◽  
Idoia Elola
Keyword(s):  
Foreign Language ◽  
Writing Classroom ◽  
Multimodal Texts ◽  
Writing Class ◽  
Digital Story ◽  
Language Writing ◽  
Multimodal Text ◽  
Digital Stories ◽  
Mode Size ◽  
Oriented Activity

AbstractDespite the availability and growing use of digital story software for authoring and instructional purposes, little is known about learners’ perceptions on its integration in the foreign language writing class. Following both a social semiotics approach and activity theory, this study focuses on six advanced Spanish learners’ perceptions about the production of a digital story in which they integrated a variety of modes (written, oral, images, sounds) and manipulated the semiotic resources within each mode (size, color, lines in the image mode), to convey meaning. Analyzing participants’ reflections, questionnaires, and online journals, results highlight learners’ (a) interpretation of the tools and artifacts and their effect on their understanding of a final product, (b) connections between short-term goal-oriented actions and the longer-term object-oriented activity of developing a multimodal text, and (c) linguistic reorientations when creating a digital story.

scholarly journals On Forensic Linguists and Non-Experts Interpretation of Multimodal Texts

2020 ◽  
pp. 102-114
Author(s):  
Aleksandra Gorbacheva ◽  
Tatiana Nesterova ◽  
Mikhail Osadchiy
Keyword(s):  
Experimental Data ◽  
Experimental Research ◽  
Text Comprehension ◽  
Priming Effect ◽  
Forensic Analysis ◽  
Forensic Assessment ◽  
Coordination Mechanisms ◽  
Expert Assessment ◽  
Multimodal Texts ◽  
Multimodal Text

The article presents intermediate results of an experimental research into psycholinguistic characteristics of understanding multimodal texts with extremist content. Forensic linguists specializing in extremism, well-informed and poorly informed in extremism discourse non-experts had to assess and comment upon multimodal texts. The peculiarities of interpretations have been established. The comprehension of sense in a multimodal text is constructed from established meanings of individual text components and detected semantic connections between them. Semantic and grammatical coordination of meanings of text components was the main mechanism of interpretation. The authors have performed the results of quantitative as well as qualitative analysis of experimental data using linguistic semantics methods. The indicators confirming the effectiveness of the expert assessment are given. The data presented confirm that forensic assessment of a multimodal text is influenced by presence or absence of both professional and general discourse knowledge. Both experts and non-experts have been proved to interpret multimodal texts using the semantic and grammatical coordination mechanisms. The correlation between these mechanisms and distorted text comprehension has been established. The factors of non-forced expert mistakes have been determined as excessive habit of interpretation, predominance of expert experience in assessment of materials of a particular discourse and priming effect. The results of the study can be used to develop science-based methods and recommendations on methodology of forensic analysis of extremist texts for experts.

scholarly journals The Visual Aspect of Translation Training in Multimodal Texts

2016 ◽  
Vol 61 (2) ◽  
pp. 299-319
Author(s):  
George Damaskinidis
Keyword(s):  
Translation Studies ◽  
Translation Efficiency ◽  
Multimodal Approach ◽  
Multimodal Texts ◽  
Multimodal Text ◽  
The Subject ◽  
New Challenges ◽  
Further Development ◽  
Subject Specialist

This paper explores the wider issue of translation training in multimodal contexts. The multimodal text represents a complex semiotic canvas on which the various systems of signification (verbal, images, colour, layout, etc.) interact in complex ways to produce a coherent meaning. Such interactions affect translation students’ understanding of multimodal texts and as such their training must also be visually-oriented in order to improve their translation efficiency when dealing with these texts. The paper is primarily (though not exclusively) concerned with the print multimodal text, and examines how the various aspects of the visual semiotic elements affect the teaching of its translation into another language. One such aspect is the new challenges that have been imposed by the visual on the field of translation studies. A second aspect is the visual implications for translation trainers and students. A third aspect is the wider multimodal context in which they have been found and involves the necessary multimodal approach to translation training, the development of a relevant awareness of multimodal texts and a number of other issues such as students’ creativity and the role of the subject specialist in the translation classroom. Finally, suggestions are made for further development of relevant teaching areas that are driven by the visual aspect of the multimodal text.

Sign in / Sign up

Export Citation Format

Share Document