Semantic Potential of Tactile Metaphors (Based on Words Naming the Property of Softness)

This article describes the results of the study on the psychophysiological phenomenon of synaesthesia projected onto language and speech. Synaesthesia is considered on the basis of words with the root myagk- (myagky, myagko, myagkost’, smyagchit’), which in their original meaning name the tactile sensation of softness. The paper aimed to identify the potential of this vocabulary and the sensory impression behind it as a means of conceptualizing knowledge about the world as well as to study the mechanism for the development of metaphorical, primarily synaesthetic, semantics. It is important to assess the sequence of expression of synaesthetic semantics and meanings that go beyond synaesthetic meanings, as well as to study whether vocabulary with the original semantics of softness can figuratively conceptualize knowledge. The research revealed a universal ability of this kind of vocabulary to conceptualize all types of sensory impressions (visual, auditory, olfactory and gustatory) when used metaphorically. In addition to conveying a multimodal perception of the world, these words can express new sensory sensations, unusual for the speaker, for which there are no special names in the language. Along with the synaesthetic semantics itself, vocabulary with the meaning of softness is able to reflect the psychological impressions of comfort, pleasure and enjoyment, as well as the psychological characteristics of a person and the psychological aspect of interpersonal interaction, including speech. The next stage of abstraction of the tactile impression is the development on its basis of quantitative semantics, i.e. the meaning of low intensity of the manifestation of an attribute or an action in their psychological perception. The results obtained can be used to identify the mechanism for conceptualizing sensory impressions and forming synaesthetic metaphors, as well as applied in the practice of lexicographic description of cognitive metaphors.

Does Sound Influence Perceived Duration of Visual Motion?

Multimodal perception is a key factor in obtaining a rich and meaningful representation of the world. However, how each stimulus combines to determine the overall percept remains a matter of research. The present work investigates the effect of sound on the bimodal perception of motion. A visual moving target was presented to the participants, associated with a concurrent sound, in a time reproduction task. Particular attention was paid to the structure of both the auditory and the visual stimuli. Four different laws of motion were tested for the visual motion, one of which is biological. Nine different sound profiles were tested, from an easier constant sound to more variable and complex pitch profiles, always presented synchronously with motion. Participants’ responses show that constant sounds produce the worst duration estimation performance, even worse than the silent condition; more complex sounds, instead, guarantee significantly better performance. The structure of the visual stimulus and that of the auditory stimulus appear to condition the performance independently. Biological motion provides the best performance, while the motion featured by a constant-velocity profile provides the worst performance. Results clearly show that a concurrent sound influences the unified perception of motion; the type and magnitude of the bias depends on the structure of the sound stimulus. Contrary to expectations, the best performance is not generated by the simplest stimuli, but rather by more complex stimuli that are richer in information.

Unimodal and Multimodal Perception for Forest Management: Review and Dataset

Robotics navigation and perception for forest management are challenging due to the existence of many obstacles to detect and avoid and the sharp illumination changes. Advanced perception systems are needed because they can enable the development of robotic and machinery solutions to accomplish a smarter, more precise, and sustainable forestry. This article presents a state-of-the-art review about unimodal and multimodal perception in forests, detailing the current developed work about perception using a single type of sensors (unimodal) and by combining data from different kinds of sensors (multimodal). This work also makes a comparison between existing perception datasets in the literature and presents a new multimodal dataset, composed by images and laser scanning data, as a contribution for this research field. Lastly, a critical analysis of the works collected is conducted by identifying strengths and research trends in this domain.

Language Experience and Subjective Word Familiarity on the Multimodal Perception of Non-Native Speakers’ Vowels

The present study investigated native (L1) and non-native (L2) speakers’ perception of the French vowels /ɔ̃, ɑ̃, ɛ̃, o/. Thirty-four American-English learners of French and 33 native speakers of Parisian French were asked to identify 60 monosyllabic words produced by a native speaker in three modalities of presentation: auditory-only (A-only); audiovisual (AV); and visual-only (V-only). The L2 participants also completed a vocabulary knowledge test of the words presented in the perception experiment that aimed to explore whether subjective word familiarity affected speech perception. Results showed that overall performance was better in the AV and A-only conditions for the two groups with the pattern of confusion differing across modalities. The lack of audiovisual benefit was not due to the vowel contrasts being not visually salient enough, as shown by the native group’s performance in the V-only modality, but to the L2 group’s weaker sensitivity to visual information. Additionally, a significant relationship was found between subjective word familiarity and AV and A-only (but not V-only) perception of non-native contrasts.

An autonomous excavator system for material loading tasks

Excavators are widely used for material handling applications in unstructured environments, including mining and construction. Operating excavators in a real-world environment can be challenging due to extreme conditions—such as rock sliding, ground collapse, or excessive dust—and can result in fatalities and injuries. Here, we present an autonomous excavator system (AES) for material loading tasks. Our system can handle different environments and uses an architecture that combines perception and planning. We fuse multimodal perception sensors, including LiDAR and cameras, along with advanced image enhancement, material and texture classification, and object detection algorithms. We also present hierarchical task and motion planning algorithms that combine learning-based techniques with optimization-based methods and are tightly integrated with the perception modules and the controller modules. We have evaluated AES performance on compact and standard excavators in many complex indoor and outdoor scenarios corresponding to material loading into dump trucks, waste material handling, rock capturing, pile removal, and trenching tasks. We demonstrate that our architecture improves the efficiency and autonomously handles different scenarios. AES has been deployed for real-world operations for long periods and can operate robustly in challenging scenarios. AES achieves 24 hours per intervention, i.e., the system can continuously operate for 24 hours without any human intervention. Moreover, the amount of material handled by AES per hour is closely equivalent to an experienced human operator.

Open multimodal iEEG-fMRI dataset from naturalistic stimulation with a short audiovisual film

Intracranial human recordings are a valuable and rare resource that the whole neuroscience community can benefit from. Making such data available to the neuroscience community not only helps tackle the reproducibility issues in science, it also helps make more use of this valuable data. The latter is especially true for data collected using naturalistic tasks. Here, we describe a dataset collected from a large group of human subjects while they watched a short audiovisual film. The dataset is characterized by several unique features. First, it combines a large amount of intracranial data from 51 intracranial electroencephalography (iEEG) participants, who all did the same task. Second, the intracranial data are accompanied by fMRI recordings acquired for the same task in 30 functional magnetic resonance imaging (fMRI) participants. Third, the data were acquired using a rich audiovisual stimulus, for which we provide detailed speech and video annotations. This multimodal dataset can be used to address questions about neural mechanisms of multimodal perception and language comprehension as well as the nature of the neural signal acquired during the same task across brain recording modalities.

High-Level Perception and Multimodal Perception

What is the correct procedure for determining the contents of perception? Philosophers tackling this question increasingly rely on empirically oriented procedures. This chapter argues that this strategy constitutes an improvement over the armchair methodology of phenomenal contrast arguments, but that there is a respect in which current empirical procedures remain limited: they are unimodal in nature, wrongly treating the senses as isolatable. The chapter thus has two aims: first, to motivate a reorientation of the admissible contents debate into a multimodal framework. The second is to explore whether experimental studies of multimodal perception support a so-called Liberal account of perception’s admissible contents. The chapter concludes that the McGurk effect and the ventriloquist effect are both explicable without the postulation of high-level content, but that at least one multimodal experimental paradigm may necessitate such content: the rubber hand illusion. One upshot is that Conservatives who claim that the Liberal view intolerably broadens the scope of perceptual illusions, particularly from the perspective of perceptual psychology, should pursue other arguments against that view.

Sociosemiotic approach to teaching foreign language listening: linguodidactic implications of adopting views on interrelation of verbal and non-verbal semiotic resources in communication.

The following article raises the issue built around teaching foreign language listening by the restrictions imposed by the theory of oral activity that influences the view on the nature of communication and human social behavior. In particular, the conducted analysis of the theoretical underpinnings of social semiotics and multimodal approach to human communication gives scientific credence to assuming the meaning-making potential of non-verbal language comparable to the one of verbal language. In addition to that, it becomes evident that non-verbal language has a tendency to complicate inferencing the meaning of the verbal activity rather than merely simplify it. As a consequence, the linguodidactic basis of teaching listening necessitates revision in this vein. Grounded on multimodal and sociocognitive approaches, the idea is put forward to interpret foreign language listening as a constituent part of sociocognitive activity, the distinguishing features of which are embodied perception and orientation to embodied meaning-making. On the basis of aforementioned, the article enriches psycholinguistic theory of listening with aspects of aural-visual (multimodal) perception as well as offers a view on the alignment of semiotic resources to produce meaning. As a result, there were formulated a set of specific methodological principles intended to govern the teaching process of intertwined aural-visual perception and inferencing of cross-mode meanings and subsequently ensure high performance in foreign language listening in the context of multimodal communication.