Constraint-Based Sound-Motion Objects in Music Performance

The aim of this paper is to present principles of constraint-based sound-motion objects in music performance. Sound-motion objects are multimodal fragments of combined sound and sound-producing body motion, usually in the duration range of just a few seconds, and conceived, produced, and perceived as intrinsically coherent units. Sound-motion objects have a privileged role as building blocks in music because of their duration, coherence, and salient features and emerge from combined instrumental, biomechanical, and motor control constraints at work in performance. Exploring these constraints and the crucial role of the sound-motion objects can enhance our understanding of generative processes in music and have practical applications in performance, improvisation, and composition.

Understanding Musical Instants

We may typically experience music as continuous streams of sound and associated body motion, yet we may also perceive music as sequences of more discontinuous events, or as strings of chunks with multimodal sensations of sound and body motion, chunks that can be called ‘sound-motion objects’. The focus in this chapter is on how such sound-motion objects emerge at intermittent points in time called ‘musical instants’, and how musical instants are necessary in order to perceive salient features in music such as of timbre, pitch, texture, contour, and overall stylistic and affective features. The emergence of musical instants is also understood as based on the combined constraints of musical instruments, sound-producing body motion, and music perception, also suggesting that understanding musical instants may have practical applications in making music.

Signal envelope and speech intelligibility differentially impact auditory motion perception

Our acoustic environment contains a plethora of complex sounds that are often in motion. To gauge approaching danger and communicate effectively, listeners need to localize and identify sounds, which includes determining sound motion. This study addresses which acoustic cues impact listeners’ ability to determine sound motion. Signal envelope (ENV) cues are implicated in both sound motion tracking and stimulus intelligibility, suggesting that these processes could be competing for sound processing resources. We created auditory chimaera from speech and noise stimuli and varied the number of frequency bands, effectively manipulating speech intelligibility. Normal-hearing adults were presented with stationary or moving chimaeras and reported perceived sound motion and content. Results show that sensitivity to sound motion is not affected by speech intelligibility, but shows a clear difference for original noise and speech stimuli. Further, acoustic chimaera with speech-like ENVs which had intelligible content induced a strong bias in listeners to report sounds as stationary. Increasing stimulus intelligibility systematically increased that bias and removing intelligible content reduced it, suggesting that sound content may be prioritized over sound motion. These findings suggest that sound motion processing in the auditory system can be biased by acoustic parameters related to speech intelligibility.

THE TRANSITIVITY SYSTEM OF SOCIAL SEMIOTICS IN THE PROCESS OF MEANING MAKING IN KINETIC TYPOGRAPHY

Kinetic typography, which is used firstly in film generics by metaphors created, has become widespread in social, economic and cultural fields such as internet, advertisements, TV with increasing of the need of communication under technological advance and the influence of globalization. The most important factors in the widespread use of this communication tool are that the message that is intended to be conveyed can be presented interactively and the design elements can be manipulated directly in order to create a certain mood in the audience. In kinetic typography, the elements of sound, motion and time present the abstract concepts as concrete and each element is evaluated as a mode (method, tool). Using these elements, the sign system is revealed through the visualization of the language, and all our experiences of the world are represented by creating a visual, spatial and aural environment in kinetic typography. Therefore, it is stated that kinetic typography is not just composed of letter forms, it is also a means of multimodal semiotic expression integrated with abstract concepts such as sound, motion, and color. Since multimodal systems have two or more semiotic tools to fulfil communication, the process of making meaning needs to be analyzed from different angles. The concept of social semiotic, which deals with how people communicate in society, this study has been tried to be carried out by adapting it to the field of visual communication. Expressing how designers portray the physical movement unique to the world in the digital environment is one of the main points of this research. Therefore, by analyzing the applications created with kinetic typography, why and how each design formation was created was expressed by examined in terms of the methods used. The aim of this study is to demonstrate the potential to create meaning with instance examinations in terms of the transitivity system which is a part of social semiotics in kinetic typography which has a multimodal (methodical) structure. Keywords: Typography, kinetic typography, social semiotics, transitivity system

The Use of Multi-modal Texts in An English Classroom of Hard-of- Hearing Learners

This article provides a story of a teacher’s experience in using ICT-integrated multimodal texts to help heard-of-hearing learners develop or improve their literacy skills especially in mastering reading comprehension. Specifically, this study tried to investigate students’ engagement in classroom activities involving multimodal reading materials. In addition, their perceptions toward the use of multimodality were also portrayed. The study was carried out in a special need school. The research is a narrative inquiry study of an English teacher and three hard-of-hearing learners. The material consists of video observation, photographs, field notes, documents, and interviews with the teacher and the students. The findings showed that multimodal texts which were built with diverse modes of semiotic resources such as color, sound, motion, written text, and gesture could ease the teacher to help the students perform better in the area of reading comprehension. Moreover, the use of multimodal texts allowed students to participate more actively in classroom activities. I believe that the result of the study contributes to the body of knowledge regarding the use of texts for reading purposes.