Challenges in designing digital interfaces for the study of multimodal phenomena

2010 ◽  
Vol 18 (1) ◽  
pp. 2-21 ◽  
Author(s):  
Kay O’Halloran ◽  
Sabine Tan ◽  
Bradley Smith ◽  
Alexey Podlasov
Keyword(s):  
Digital Technology ◽  
Theoretical Foundation ◽  
Meaning Making ◽  
Social Semiotics ◽  
Theoretical Frameworks ◽  
Digital Interface ◽  
Multimodal Data ◽  
Semiotic Approach ◽  
Multimodal Text ◽  
Computer Based

The paper discusses the challenges faced by researchers in developing effective digital interfaces for analyzing the meaning-making processes of multimodal phenomena. The authors propose a social semiotic approach as the underlying theoretical foundation, because interactive digital technology is the embodiment of multimodal social semiotic communication. The paper outlines the complex issues with which researchers are confronted in designing digital interface frameworks for modeling, analyzing, and retrieving meaning from multimodal data, giving due consideration to the multiplicity of theoretical frameworks and theories which have been developed for the study of multimodal text within social semiotics, and their impact on the development of a computer-based tool for the exploration, annotation, and analysis of multimodal data.

East-Asian Philosophical Concepts as Analytical Framework for Interpreting Non-Western Images in Children's Picturebooks

2020 ◽  
pp. 393-420
Author(s):  
Ngoc Tai Huynh ◽  
Angela Thomas ◽  
Vinh Thi To
Keyword(s):  
Young Children ◽  
Meaning Making ◽  
East Asian ◽  
Analytical Framework ◽  
Theoretical Frameworks ◽  
Learning To Read ◽  
Intercultural Awareness ◽  
Semiotic Approach ◽  
Art Forms ◽  
Cultural Constraints

In contemporary Western cultures, picturebooks are a mainstream means for young children to first attend to print and start learning to read. The use of children's picturebooks has been reported as supporting intercultural awareness in children. Multiliteracies researchers suggest that other theoretical frameworks should be applied in addition to the semiotic approach of interpreting picturebooks, especially picturebooks from non-Western cultures. This chapter theorizes how Eastern philosophical concepts influence the meaning-making potential of illustrations in Eastern picturebooks. To do this, the authors first discuss the cultural constraints when applying a contemporary semiotic framework in analyzing non-Western images. The authors introduce a framework developed based on philosophical concepts that have influenced East-Asian art forms, particularly that of painting, to understand the Eastern artistic traditions. The chapter demonstrates how to apply this framework for interpretation of non-Western images to working with multicultural picturebooks.

Multimodal Analysis in Qualitative Research: Extending Grounded Theory Through the Lens of Social Semiotics

2020 ◽  
pp. 107780042097874
Author(s):  
Vivien Sommer
Keyword(s):  
Grounded Theory ◽  
Digital Technology ◽  
Research Approach ◽  
Social Semiotics ◽  
Audiovisual Material ◽  
Text And Image ◽  
Multimodal Data ◽  
Multimodal Analysis ◽  
Analytical Tools ◽  
Analyze Data

Digital technology has made it easier for researchers to conduct and produce multimodal data. In terms of a social semiotic understanding, multimodal means that data are produced from different sign resources, such as field protocols combined with visual recordings or document analysis consisting of audiovisual material. The increase in multimodal data brings the challenge of developing analytical tools not only to collect data but also to examine them. In this article, I introduce a research approach for how to integrate multimodal data within the framework of grounded theory by extending the coding process with a social semiotic understanding of data as a combination of different sign modes. This approach makes it possible not only to analyze data based on different modes separately but also to analyze their combination, for example, the interweaving of text and image.

Social Semiotics for Social Media Visuals

2022 ◽  
pp. 548-570
Author(s):  
Hossam Mohamed Elhamy
Keyword(s):  
Social Media ◽  
Social Practice ◽  
Meaning Making ◽  
Deep Understanding ◽  
Social Semiotics ◽  
Media Environment ◽  
Visual Elements ◽  
Social Actions ◽  
The Social ◽  
Semiotic Approach

The social semiotics approach examines the meaning-making process in order to demonstrate how meaning is constructed in social actions and contexts. The rising interest of researchers in social media and its widespread use in society have both highlighted new challenges for data analysis. Social semiotics can provide a deep understanding of the visual grammar of the social media meaning-making process by assuming that this process is considered a social practice. The main objective of this chapter is to guide researchers and enable them to use the social semiotic approach as a research tool for the analysis of visuals in the social media environment. The chapter introduces the key elements, principles, assumptions, and rules of using the social semiotics approach in the analysis, understanding, and interpretations of social media visuals and how to explore the role played by visual elements in the meaning-making process in a social media within a specific social context.

Representations of the Black Death: Multimodality and Assessment in the History Classroom

2017 ◽  
Vol 6 (2) ◽  
Author(s):  
Eva Insulander
Keyword(s):  
Field Study ◽  
Middle Ages ◽  
Digital Technology ◽  
Meaning Making ◽  
Subject Area ◽  
Assessment Tools ◽  
Black Death ◽  
Social Semiotics ◽  
Central Data ◽  
The Middle Ages

AbstractIn this article I explore how pupils are permitted to use different modes and media to express their knowing of a particular subject area, ‘The Middle Ages’. I combine social semiotics with a design-oriented perspective on learning. In both perspectives, multimodality is central. Data comes from a field study of representation, resources and meaning making in two different history classrooms with contrasting learning designs and use of digital technology. In the paper, I show how the different learning- and assessment designs, and text creation practices, affect pupils’ learning. The paper also discuss the need for development of new assessment tools.

scholarly journals A “Strong” Approach to Sustainability Literacy: Embodied Ecology and Media

Philosophies ◽  
2021 ◽  
Vol 6 (1) ◽  
pp. 14
Author(s):  
Cary Campbell ◽  
Nataša Lacković ◽  
Alin Olteanu
Keyword(s):  
Media Literacy ◽  
Meaning Making ◽  
Environmental Literacy ◽  
Literacy Studies ◽  
Critical Media Literacy ◽  
Semiotic Approach ◽  
Sustainability Literacy ◽  
Critical Media ◽  
Definition Of ◽  
Integrated Sustainability

This article outlines a “strong” theoretical approach to sustainability literacy, building on an earlier definition of strong and weak environmental literacy (Stables and Bishop 2001). The argument builds upon a specific semiotic approach to educational philosophy (sometimes called edusemiotics), to which these authors have been contributing. Here, we highlight how a view of learning that centers on embodied and multimodal communication invites bridging biosemiotics with critical media literacy, in pursuit of a strong, integrated sustainability literacy. The need for such a construal of literacy can be observed in recent scholarship on embodied cognition, education, media and bio/eco-semiotics. By (1) construing the environment as semiosic (Umwelt), and (2) replacing the notion of text with model, we develop a theory of literacy that understands learning as embodied/environmental in/across any mediality. As such, digital and multimedia learning are deemed to rest on environmental and embodied affordances. The notions of semiotic resources and affordances are also defined from these perspectives. We propose that a biosemiotics-informed approach to literacy, connecting both eco- and critical-media literacy, accompanies a much broader scope of meaning-making than has been the case in literacy studies so far.

The Meaning Making of the Built Environment in the Fascist City: A Semiotic Approach

2018 ◽  
Vol 6 (2) ◽  
pp. 379-411
Author(s):  
Antonio Nanni ◽  
Federico Bellentani
Keyword(s):  
Built Environment ◽  
Meaning Making ◽  
Semiotic Approach

scholarly journals Revisioning History: A Deconstructionist Reading of a Learner’s Multimodal Text, Revenge

2016 ◽  
Vol 12 (8) ◽  
pp. 64
Author(s):  
Zillasafarina Ja'afar ◽  
Noraini Md. Yusof ◽  
Noraini Ibrahim
Keyword(s):  
Digital Communication ◽  
Meaning Making ◽  
Critical Reading ◽  
Text And Image ◽  
Theoretical Concepts ◽  
Political Issues ◽  
Historical Texts ◽  
Recent Interest ◽  
Multimodal Text ◽  
Verbal Data

<p>Recent interest in multimodality recognizes the integration of text and image in meaning-making as representing reality. It has also been argued that with the use of digital communication, the meanings of visual and verbal data can be easily manipulated rendering them unreliable. As such, a close and critical reading of the text is required to discover what is hidden, absent, or inconsistent with it. In a deconstruction of a multimodal digital composition of a poem that involves revisioning of history, this paper privileges the absences of cultural and historical texts to signify socio-political issues. An eclectic use of theoretical concepts on meaning-making, especially those proposed by Kress and van Leeuwen, Foucault and Baudrillard, constructs the discussion of the analysis. The digital poem entitled ‘Revenge’ is deconstructed to further discover such absence in the text. The findings reveal that language and images are used by the learner as a source of power to negotiate the boundaries of identity. It has also been discovered that the message in rhetoric and visuals complement each other to support the process of meaning-making.</p>

scholarly journals A Social Semiotic Approach to the Attitudinal Meanings in Multimodal Texts

2019 ◽  
Vol 9 (9) ◽  
pp. 1160
Author(s):  
Yong Hu ◽  
Qing Qiu
Keyword(s):  
Picture Books ◽  
Multimodal Texts ◽  
Visual Narratives ◽  
Visual Semiotics ◽  
Semiotic Approach ◽  
Semiotic Resources ◽  
Multimodal Text ◽  
The Creation

As a special type of multimodal text, picture books for children are highly valued in the creation of meaning by the integrative use of verbal and visual semiotic resources. Informed by Painter and Martin’s framework of visual narratives, this paper primarily deals with the interpersonal meanings encoded and expressed by the two semiotics (image and verbiage) within the Chinese picture books. It aims to analyse the visual and verbal choices available for writers to establish engagement between various participants. In the hope of investigating the collaboration and interplay of verbal and visual semiotics to construe interpersonal meanings, it examines the attitudinal meanings inscribed or invoked in picture books, exploring the ways in which visual and verbal resources are co-instantiated to encode attitudinal convergence and also divergence.

scholarly journals Multimodal metaphors: from language as a condition to text to the notion of texture as a meaning-making semiotic resource

2020 ◽  
Vol 28 (2) ◽  
pp. 893
Author(s):  
Clarice Gualberto ◽  
Záira Santos ◽  
Ana Clara Meira
Keyword(s):  
Social Media ◽  
Meaning Making ◽  
Social Semiotics ◽  
Concepts And Categories ◽  
Ways Of Thinking ◽  
Do So

Abstract: In this paper, we attempt to provide some ways of thinking about text, relating it to the concept of metaphors, multimodality and texture. Our aim is to develop new insights in meaning making and communication more generally, by bringing examples of memes; a relatively new genre, often seen on social media posts. To do so, we discuss the notion of text (HALLIDAY; HASAN, 2002; BEAUGRANDE, 1997; KRESS, 2010) and then, we discuss texture as a semiotic resource for the production of texts and its metaphors (DJONOV; VAN LEEUWEN, 2011). To exemplify some concepts and categories, we explore memes, seeking to understand their constitution as well as their qualities and potential meanings of visual textures deployed in the text to make meaning material through multimodal metaphors.Keywords: text; texture; social semiotics; multimodal metaphors.Resumo: Neste artigo, pretendemos propor algumas maneiras para se pensar a noção de texto, relacionando-a aos conceitos de metáfora, multimodalidade e textura. Nosso objetivo é desenvolver novos insights a respeito da produção de sentido e da comunicação de forma geral. Como exemplo, trazemos  memes com o bordão “É verdade esse bilete”. Como esse gênero é relativamente novo e frequentemente visto em posts nas redes sociais, discutimos a noção de texto (HALLIDAY; HASAN, 2002; BEAUGRANDE, 1997; KRESS, 2010) e de textura como um recurso semiótico para a produção de textos e suas metáforas (DJONOV; VAN LEEUWEN, 2011). Para exemplificar algumas noções e categorias, exploramos os memes, buscando compreender como eles se constituem, assim como suas qualidades e seus possíveis sentidos de texturas visuais utilizadas para produção de metáforas multimodais.Palavras-chave: texto; textura; semiótica social; metáforas multimodais.

Entering a New World

2020 ◽  
pp. 125-147 ◽  
Author(s):  
Nasima Mohamed Hoosen Carrim
Keyword(s):  
Professional Identity ◽  
South African ◽  
Digital Technology ◽  
New World ◽  
Identity Work ◽  
South African Context ◽  
Theoretical Frameworks ◽  
Use Of Technology ◽  
African Context ◽  
Indian Male

This chapter focuses on the identity work engaged in by two older Indian male entrepreneurs and their use of technology in their business enterprises. The chapter briefly outlines the concepts of older entrepreneurs and entrepreneurial innovation and gives an overview of the South African context that pertains to Indians. McClelland's theory of motivation, Bandura's theory of self-efficacy and identity theory serve as the theoretical frameworks. The author conducted interviews with two older Indian male entrepreneurs to ascertain their use of digital technology in their respective businesses. The result of the investigation indicates that older Indian males' use of digital technology is minimal and that they engage in entrepreneurial professional identity work to a minimal extent. Some recommendations are made regarding solutions to the problems identified.

Sign in / Sign up

Export Citation Format

Share Document