multimodal interaction
Recently Published Documents


TOTAL DOCUMENTS

680
(FIVE YEARS 155)

H-INDEX

21
(FIVE YEARS 5)

Author(s):  
Наталья Борисовна Акоева ◽  
Лилун Чэнь

«Мировая церемония предков хакка» - одна из самых влиятельных церемоний поклонения предкам в Китае. В 2011 году это мероприятие было включено в Национальный список нематериального культурного наследия. Благодаря слиянию и сосуществованию нескольких моделей взаимодействия в «церемонии поклонения предкам» и точному пониманию намерений мультимодального взаимодействия синхронизировано представление многоканального восприятия (визуального, звукового и сенсорного). На основе Node.js и Three.js технологий авторами была реализована цифровая система для «церемонии поклонения предкам». Цифровая модель представляет действия в двух залах: Хуайцзу, где можно выразить свои чувства памяти и помолиться за свои семьи, и Юйпин, где можно произвести интерактивные операции, такие как подношение свечей, подношение благовоний и т.п. Тесты показывают, что в системе достигается баланс между производительностью и удобством для пользователя, что дает реальное решение для оцифровки нематериального культурного наследия ритуального типа. Цифровой контент церемонии не ограничен во времени и пространстве, не требует использования сторонних плагинов и имеет быструю доступность, что может способствовать быстрому распространению и развитию «Церемонии предков мира хакка». The Hakka World Ancestor Ceremony is one of the most influential ancestor worship ceremonies in China. In 2011, this event was included in the National List of Intangible Cultural Heritage. Due to the fusion and coexistence of several interaction models in the "ancestor worship ceremony" and an accurate understanding of the intentions of multimodal interaction, the representation of multi-channel perception (visual, sound and sensory) is synchronized. Based on Node.js and Three.js technologies, the authors implemented a digital system for the "ancestor worship ceremony". The digital model represents actions in two halls: Huaizu, where you can express your feelings of memory and pray for your families, and Yuping, where you can perform interactive operations, such as offering candles, offering incense, etc. Tests show that the system achieves a balance between performance and user-friendliness, which provides a real solution for digitizing intangible cultural heritage of the "ritual" type. The digital content of the ceremony is not limited in time and space, does not require the use of third-party plug-ins and has rapid availability, which can contribute to the rapid spread and development of the "Ceremony of the Ancestors of the Hakka World".


2022 ◽  
Vol 12 ◽  
Author(s):  
Denise Wakke ◽  
Vivien Heller

This study examines interactions in which students help each other with their learning during classroom instruction, forming groups in the process. From a conversation analytic perspective, helping is assumed to be a sequentially organized activity jointly accomplished by the participants. As an activity that proceeds alongside other ongoing classroom activities, helping can be conceived as part of a multiactivity that poses students with multi-faceted interactional and moral challenges. While previous research on helping in educational contexts has primarily focused on the influence of helping on learning outcomes and social dynamics in helping interactions, the present study investigates how students cope with the intricacies of moral commitments inherent in helping as a concurrent activity. The aim of this paper is two-fold. First, we aim to elaborate on how students’ dual involvements – i.e., their involvement in classroom activities while simultaneously providing help – manifest in the ways in which groups are constituted, maintained, and dissolved. The analyses reveal that both the compatibility of helping with the activity already in progress as well as the students’ problem definition are consequential for the sequential and bodily-spatial unfolding of the help interaction, inducing different arrangements that constitute a continuum, at each end of which there is a dominant orientation toward the shared space of helping or toward the individual/collective space. Furthermore, from a methodological perspective, our study aims to demonstrate the extent to which multimodal interaction analysis is applicable when examining naturally occurring groups, in this case, in interactive processes of helping. The study is based on a data corpus that comprises video recordings of mathematics and German lessons from two fifth-grade classrooms.


2021 ◽  
Vol 5 (12) ◽  
pp. 81
Author(s):  
Ismo Rakkolainen ◽  
Ahmed Farooq ◽  
Jari Kangas ◽  
Jaakko Hakulinen ◽  
Jussi Rantala ◽  
...  

When designing extended reality (XR) applications, it is important to consider multimodal interaction techniques, which employ several human senses simultaneously. Multimodal interaction can transform how people communicate remotely, practice for tasks, entertain themselves, process information visualizations, and make decisions based on the provided information. This scoping review summarized recent advances in multimodal interaction technologies for head-mounted display-based (HMD) XR systems. Our purpose was to provide a succinct, yet clear, insightful, and structured overview of emerging, underused multimodal technologies beyond standard video and audio for XR interaction, and to find research gaps. The review aimed to help XR practitioners to apply multimodal interaction techniques and interaction researchers to direct future efforts towards relevant issues on multimodal XR. We conclude with our perspective on promising research avenues for multimodal interaction technologies.


2021 ◽  
Author(s):  
◽  
Keely Kidner

<p>A multimodal, critical approach to Discourse allows us to understand contemporary environmental debates in a nuanced way. Fossil fuel mining has become especially controversial due to the environmental, health, and social impacts, as well as the substantial economic dependence on such development. Wider discussions surrounding mining projects tend to diverge into two major oppositions: that of the industry itself, and that of local activists protesting development on their lands. Research in these areas has leaned towards a focus on the use of environmental language by polluting industries, termed ‘greenwash’, missing to some degree the ways in which these and other Discourses are articulated multimodally through interaction. This thesis brings a critical, multimodal analysis to controversial mining debates which go ‘beyond greenwash’, in order to track how Discourses are appropriated, resisted, and re-entextualised.  In this thesis I adopt overlapping critical, multimodal, and ethnographic theoretical lenses to view interaction surrounding two controversial mining case studies: the Athabasca tar sands in Alberta, Canada and a lignite coal mine expansion in Southland, Aotearoa/New Zealand. Drawing upon an understanding of human communication as inherently multimodal, I include video-recorded interviews with both activists and industry representatives, as well as relevant artefacts (such as pamphlets, photographs, signs, etc) in my dataset. Using mediated action as the unit of analysis, I employ Multimodal Interaction Analysis to examine interview interaction, coupled with methods from Social Semiotics to interrogate designed artefacts. These analytical frameworks, viewed through combined theoretical lenses, provide a unique perspective on the way Discourses are appropriated and creatively resisted in debates about resource development.  In both case studies, Discourses of the environment and the economy are appropriated by activists and industry actors, forming the basic ‘Environment v. Economy’ Discourse. This dichotomy is expanded through the appropriation of additional Discourses, such as regional identity in both Southland and Alberta. Activist groups subsequently resist and re-appropriate these regional Discourses by multimodally re-contextualising them. In Alberta, additional Discourses of Indigenous and LGBTQ+ identities are appropriated by industry actors in attempts to legitimise mining expansion. While some of these appropriations draw upon ideas of intersecting oppressions, mining industries fail to adequately address the ways in which resource extraction contributes to those oppressions. However, these actions are both recognised and resisted by local anti-tar sands activists, who use public events and designed artefacts to display their opposition and reappropriate Discourses.  Although both case studies are concerned with similar types of fossil fuel extraction, there are notable differences. Whereas Discourses of the environment, the economy, and regional identity are both appropriated and resisted in Southland and Alberta, it appears that in Alberta, industry actors draw upon a wider variety of Discourses. This may be due, in part, to the fact that the Athabasca tar sands controversy has had a longer development period and more international resistance, meaning there is more pressure to legitimise new and ongoing projects. This has led to a type of ‘discursive arms race’, where wider Discourses are appropriated as other, more directly-linked, Discourses are exhausted (e.g. the environment). In response, activists in both Southland and Alberta make use of creativity and humour in mundane performances to enact satirical representations of powerful industry actors. While these performances occur regularly in many of my interviews, activists in Alberta also tend to stage elaborate and humorous theatrics in order to criticise government and industry approaches to fossil fuel development.  The unique framework employed in this thesis has resulted in a number of research implications. These include the combination of Critical Discourse Analysis with a multimodal, ethnographic approach, and the coupling of Multimodal Interaction Analysis with Social Semiotics to expand my analytical reach. Additionally, I have made use of a variety of modes in this thesis’ presentation, in order to exploit each mode’s affordances and better represent the complexity of my dataset. Finally, from a critical perspective, this research offers an agenda of empowerment for environmental justice activist groups struggling to protect their lands.</p>


2021 ◽  
Author(s):  
◽  
Keely Kidner

<p>A multimodal, critical approach to Discourse allows us to understand contemporary environmental debates in a nuanced way. Fossil fuel mining has become especially controversial due to the environmental, health, and social impacts, as well as the substantial economic dependence on such development. Wider discussions surrounding mining projects tend to diverge into two major oppositions: that of the industry itself, and that of local activists protesting development on their lands. Research in these areas has leaned towards a focus on the use of environmental language by polluting industries, termed ‘greenwash’, missing to some degree the ways in which these and other Discourses are articulated multimodally through interaction. This thesis brings a critical, multimodal analysis to controversial mining debates which go ‘beyond greenwash’, in order to track how Discourses are appropriated, resisted, and re-entextualised.  In this thesis I adopt overlapping critical, multimodal, and ethnographic theoretical lenses to view interaction surrounding two controversial mining case studies: the Athabasca tar sands in Alberta, Canada and a lignite coal mine expansion in Southland, Aotearoa/New Zealand. Drawing upon an understanding of human communication as inherently multimodal, I include video-recorded interviews with both activists and industry representatives, as well as relevant artefacts (such as pamphlets, photographs, signs, etc) in my dataset. Using mediated action as the unit of analysis, I employ Multimodal Interaction Analysis to examine interview interaction, coupled with methods from Social Semiotics to interrogate designed artefacts. These analytical frameworks, viewed through combined theoretical lenses, provide a unique perspective on the way Discourses are appropriated and creatively resisted in debates about resource development.  In both case studies, Discourses of the environment and the economy are appropriated by activists and industry actors, forming the basic ‘Environment v. Economy’ Discourse. This dichotomy is expanded through the appropriation of additional Discourses, such as regional identity in both Southland and Alberta. Activist groups subsequently resist and re-appropriate these regional Discourses by multimodally re-contextualising them. In Alberta, additional Discourses of Indigenous and LGBTQ+ identities are appropriated by industry actors in attempts to legitimise mining expansion. While some of these appropriations draw upon ideas of intersecting oppressions, mining industries fail to adequately address the ways in which resource extraction contributes to those oppressions. However, these actions are both recognised and resisted by local anti-tar sands activists, who use public events and designed artefacts to display their opposition and reappropriate Discourses.  Although both case studies are concerned with similar types of fossil fuel extraction, there are notable differences. Whereas Discourses of the environment, the economy, and regional identity are both appropriated and resisted in Southland and Alberta, it appears that in Alberta, industry actors draw upon a wider variety of Discourses. This may be due, in part, to the fact that the Athabasca tar sands controversy has had a longer development period and more international resistance, meaning there is more pressure to legitimise new and ongoing projects. This has led to a type of ‘discursive arms race’, where wider Discourses are appropriated as other, more directly-linked, Discourses are exhausted (e.g. the environment). In response, activists in both Southland and Alberta make use of creativity and humour in mundane performances to enact satirical representations of powerful industry actors. While these performances occur regularly in many of my interviews, activists in Alberta also tend to stage elaborate and humorous theatrics in order to criticise government and industry approaches to fossil fuel development.  The unique framework employed in this thesis has resulted in a number of research implications. These include the combination of Critical Discourse Analysis with a multimodal, ethnographic approach, and the coupling of Multimodal Interaction Analysis with Social Semiotics to expand my analytical reach. Additionally, I have made use of a variety of modes in this thesis’ presentation, in order to exploit each mode’s affordances and better represent the complexity of my dataset. Finally, from a critical perspective, this research offers an agenda of empowerment for environmental justice activist groups struggling to protect their lands.</p>


2021 ◽  
Vol 0 (0) ◽  
Author(s):  
Katarina Silvestri ◽  
Mary McVee ◽  
Christopher Jarmark ◽  
Lynn Shanahan ◽  
Kenneth English

Abstract This exploratory case study uses multimodal positioning analysis to determine and describe how a purposefully crafted emergent artifact comes to influence and/or manipulate social dynamics, structure, and positionings of one design team comprised of five third-graders in an afterschool elementary engineering and literacy club. In addition to social semiotic theories of multimodality (e.g., Kress, G. (2010). Multimodality: a social semiotic approach to contemporary communication. New York, NY: Routledge) and multimodal interactional analysis (Norris, S. (2004). Analyzing multimodal interaction: a methodological framework. New York, NY: Routledge, Norris, S. (2019). Systematically working with multimodal data: research methods in multimodal discourse analysis. Hoboken, NJ: Wiley-Blackwell), Positioning Theory (Harré, R. and Van Langenhove, L. (1991). Varieties of positioning. J. Theor. Soc. Behav. 21: 393–407) is used to examine group interactions with the artifact, with observational data collected from audio, video, researcher field notes, analytic memos, photographs, student artifacts (e.g., drawn designs, built designs), and transcriptions of audio and video data. Analysis of interactions of the artifact as it unfolds demonstrates multiple types of role-based positioning with students (e.g., builder, helper, idea-sharer). Foregrounding analysis of the artifact, rather than the student participants, exposed students’ alignment or opposition with their groupmates during the project. This study contributes to multimodal and artifactual scholarship through a close examination of positions emergent across time through multimodal communicative actions and illustrates how perspectives on multimodality may be analytically combined with Positioning Theory.


Author(s):  
Chengyu Su ◽  
Chao Yang ◽  
Yonghui Chen ◽  
Fupan Wang ◽  
Fang Wang ◽  
...  

2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Chuyi Liu

There are many problems in the current process product design, such as poor interaction and low overall efficiency, so a process product design method based on multimode interaction design is proposed. Based on the relationship model, environment model, and object-oriented model of process product design, the process of process product development and design is analyzed. On this basis, the finite state machine is used to analyze the state of the designer to complete the task sequence, as well as the related design task planning, constructing craft products multimodal interaction mechanism, with the improved fuzzy analytic hierarchy process to evaluate the quality of the products, according to the evaluation results of technology of product design optimization. The experimental results show that both experts and ordinary users give a high evaluation of the product designed in this paper, and the design cycle of the product is significantly shortened, indicating that this method can fully save the design time and improve the overall efficiency.


2021 ◽  
Vol 11 (11) ◽  
pp. 723
Author(s):  
Jose Belda-Medina

The number of publications on live online teaching and distance learning has significantly increased over the past two years since the outbreak and worldwide spread of the COVID-19 pandemic, but more research is needed on effective methodologies and their impact on the learning process. This research aimed to analyze student interaction and multimodal communication through Task-Based Language Teaching (TBLT) in a Synchronous Computer-Mediated Communication (SCMC) environment. For this purpose, 90 teacher candidates enrolled in the subject Applied Linguistics at a university were randomly assigned in different teams to create collaboratively digital infographics based on different language teaching methods. Then, all the teams explained their projects online and the classmates completed two multimedia activities based on each method. Finally, the participants discussed the self-perceived benefits (relevance, enjoyment, interest) and limitations (connectivity, distraction) of SCMC in language learning. Quantitative and qualitative data were gathered through pre- and post-tests, class observation and online discussion. The statistical data and research findings revealed a positive attitude towards the integration of TBLT in an SCMC environment and a high level of satisfaction with multimodal communication (written, verbal, visual) and student interaction. However, the language teacher candidates complained about the low quality of the digital materials, the use of technology just for substitution, and the lack of peer-to-peer interaction in their live online classes during the pandemic.


Sign in / Sign up

Export Citation Format

Share Document