Natural Infant-Directed Speech Facilitates Neural Tracking of Prosody

Infants prefer to be addressed with infant-directed speech (IDS). IDS benefits language acquisition through amplified low-frequency amplitude modulations. It has been reported that this amplification increases electrophysiological tracking of IDS compared to adult-directed speech (ADS). It is still unknown which particular frequency band triggers this effect. Here, we compare tracking at the rates of syllables and prosodic stress, which are both critical to word segmentation and recognition. In mother-infant dyads (n=30), mothers described novel objects to their 9-month-olds while infants' EEG was recorded. For IDS, mothers were instructed to speak to their children as they typically do, while for ADS, mothers described the objects as if speaking with an adult. Phonetic analyses confirmed that pitch features were more prototypically infant-directed in the IDS-condition compared to the ADS-condition. Neural tracking of speech was assessed by speech-brain coherence, which measures the synchronization between speech envelope and EEG. Results revealed significant speech-brain coherence at both syllabic and prosodic stress rates, indicating that infants track speech in IDS and ADS at both rates. We found significantly higher speech-brain coherence for IDS compared to ADS in the prosodic stress rate but not the syllabic rate. This indicates that the IDS benefit arises primarily from enhanced prosodic stress. Thus, neural tracking is sensitive to parents’ speech adaptations during natural interactions, possibly facilitating higher-level inferential processes such as word segmentation from continuous speech.

Exploiting Serialized Fine-Grained Action Recognition Using WiFi Sensing

Gestures serve an important role in enabling natural interactions with computing devices, and they form an important part of everyday nonverbal communication. In increasingly many application scenarios of gesture interaction, such as gesture-based authentication, calligraphy, sketching, and even artistic expression, not only are the underlying gestures complex and consist of multiple strokes but also the correctness of the gestures depends on the order at which the strokes are performed. In this paper, we present WiCG, an innovative and novel WiFi sensing approach for capturing and providing feedback on stroke order. Our approach tracks the user’s hand movement during writing and exploits this information in combination with statistical methods and machine learning techniques to infer what characters have been written and at which stroke order. We consider Chinese calligraphy as our use case as the resulting gestures are highly complex, and their assessment depends on the correct stroke order. We develop a set of analyses and algorithms to overcome many issues of this challenging task. We have conducted extensive experiments and user studies to evaluate our approach. Experimental results show that our approach is highly effective in identifying the written characters and their written stroke order. We show that our approach can adapt to different deployment environments and user patterns.

Does it pay to imitate? No evidence for social gains from lexical imitation

According to an influential hypothesis, people imitate motor movements to foster social interactions. Could imitation of language serve a similar function? We investigated this question in two pre-registered experiments. In Experiment 1, participants were asked to alternate naming pictures and matching pictures to a name provided by a partner. Crucially, and unknown to participants, the partner was in fact a computer program which in one group produced the same names as previously used by the participant, and in the other group consistently produced different names. We found no difference in how the two groups evaluated the partner or the interaction and no difference in their willingness to cooperate with the partner. In Experiment 2, we made the task more similar to natural interactions by adding a stage in which a participant and the partner introduced themselves to each other and included a measure of the participant's autistic traits. Once again, we found no effects of being imitated. We discuss how these null results may inform imitation research.

All-weather, natural silent speech recognition via machine-learning-assisted tattoo-like electronics

The internal availability of silent speech serves as a translator for people with aphasia and keeps human–machine/human interactions working under various disturbances. This paper develops a silent speech strategy to achieve all-weather, natural interactions. The strategy requires few usage specialized skills like sign language but accurately transfers high-capacity information in complicated and changeable daily environments. In the strategy, the tattoo-like electronics imperceptibly attached on facial skin record high-quality bio-data of various silent speech, and the machine-learning algorithm deployed on the cloud recognizes accurately the silent speech and reduces the weight of the wireless acquisition module. A series of experiments show that the silent speech recognition system (SSRS) can enduringly comply with large deformation (~45%) of faces by virtue of the electricity-preferred tattoo-like electrodes and recognize up to 110 words covering daily vocabularies with a high average accuracy of 92.64% simply by use of small-sample machine learning. We successfully apply the SSRS to 1-day routine life, including daily greeting, running, dining, manipulating industrial robots in deafening noise, and expressing in darkness, which shows great promotion in real-world applications.

A SYSTEMATIC COLLECTION OF NATURAL INTERACTIONS FOR IMMERSIVE MODELING FROM BUILDING BLOCKS

The increased availability of affordable virtual reality hardware in the last years boosted research and development of such systems for many fields of application. While extended reality systems are well established for visualization of product data, immersive authoring tools that can create and modify that data are yet to see widespread productive use. Making use of building blocks, we see the possibility that such tools allow quick expression of spatial concepts, even for non-expert users. Optical hand-tracking technology allows the implementation of this immersive modeling using natural user interfaces. Here the users manipulated the virtual objects with their bare hands. In this work, we present a systematic collection of natural interactions suited for immersive building-block-based modeling systems. The interactions are conceptually described and categorized by the task they fulfil.

A CAVE based 3d immersive interactive city with gesture interface

This thesis presents a system that visualizes 3D city data and supports gesture interactions in a fully immersive Cave Automatic Virtual Environment (CAVE). To facilitate more natural interactions in this immersive virtual city, novel techniques are proposed for operations such as object selection, object manipulation, navigation and menu control. These operations form a basis of interactions for most Virtual Reality (VR) applications. The proposed techniques are predominantly controlled using gestures. We also propose the use of pattern recognition methods, specifically a Hidden Markov Model, to support real time dynamic gesture recognition and demonstrate its use for menu control in VR applications. Qualitative and quantitative user studies are conducted to evaluate the proposed techniques. The results of the user studies demonstrate that the interaction techniques for object selection and manipulation are measurably better than traditional techniques. The results also show that the proposed gesture based navigation and menu control techniques are preferred by experienced users. These findings can guide future user interface design in immersive environments.

A CAVE based 3d immersive interactive city with gesture interface

This thesis presents a system that visualizes 3D city data and supports gesture interactions in a fully immersive Cave Automatic Virtual Environment (CAVE). To facilitate more natural interactions in this immersive virtual city, novel techniques are proposed for operations such as object selection, object manipulation, navigation and menu control. These operations form a basis of interactions for most Virtual Reality (VR) applications. The proposed techniques are predominantly controlled using gestures. We also propose the use of pattern recognition methods, specifically a Hidden Markov Model, to support real time dynamic gesture recognition and demonstrate its use for menu control in VR applications. Qualitative and quantitative user studies are conducted to evaluate the proposed techniques. The results of the user studies demonstrate that the interaction techniques for object selection and manipulation are measurably better than traditional techniques. The results also show that the proposed gesture based navigation and menu control techniques are preferred by experienced users. These findings can guide future user interface design in immersive environments.

MuMIA: Multimodal Interactions to Better Understand Art Contexts

Cultural heritage is a challenging domain of application for novel interactive technologies, where varying aspects in the way that cultural assets are delivered play a major role in enhancing the visitor experience, either onsite or online. Technology-supported natural human–computer interaction that is based on multimodalities is a key factor in enabling wider and enriched access to cultural heritage assets. In this paper, we present the design and evaluation of an interactive system that aims to support visitors towards a better understanding of art contexts through the use of a multimodal interface, based on visual and audio interactions. The results of the evaluation study shed light on the dimensions of evoking natural interactions within cultural heritage environments, using micro-narratives for self-exploration and understanding of cultural content, and the intersection between human–computer interaction and artificial intelligence within cultural heritage. We expect our findings to provide useful insights for practitioners and researchers of the broad human–computer interaction and cultural heritage communities on designing and evaluating multimodal interfaces to better support visitor experiences.

Vulnerability of Subaerial and Submarine Landscapes: The Sand Falls in Cabo San Lucas, Mexico

The coastal landscape of the south of the Baja California peninsula provides significant socio-economic benefits based on tourism. An analysis of coastal vulnerability was conducted for Cabo San Lucas, considering wave climate conditions, sediment characterization, beach profiles, and the historical occurrence of coastline changes, hurricanes, and El Niño Southern Oscillation (ENSO) events. The coastal scenery was also classified considering the landscape value of the environment from a touristic point of view, based on human and natural interactions on the landscape. Results show that the vulnerability increases close to the submarine sand falls, near intense urbanization, in resort areas, and at locations with narrow beach and dune widths. The degree of vulnerability along the coast alters abruptly, as urban and recreational sites alternate with natural sites. This coastline has seen exponential development since the 1980s, resulting in highly vulnerable areas with a low, and decreasing, touristic value, as the landscape has been changed into an urban settlement with limited natural attractions. Urban and recreational settlements threaten to cover dunes and reservoirs of natural sediments, increasingly affecting vulnerability in the area as well as the landscape values of many parts of the coast, including the submarine sand falls.