Continuous neural activations of embodied-linguistic representations in modality switching costs

Modality switching costs (MSCs) are one of the classic effects that support the embodied views of conceptual representations. They refer to a delay in response time to verify a sensory property of a certain perceptual modality (e.g., visual: SUN – bright), when the previous sensory property has been of a different modality (e.g., auditory: BLENDER – loud) compared to a property of the same modality (e.g., visual: ROSE – red). Such costs indicate that conceptual representations require the recruitment of modality-specific resources. However, MSCs could also result from the distributional pattern of property words: the reason why loud -> bright takes longer than red -> bright could be because bright and loud do not co-occur in the same linguistic context as frequently as bright and red. In the present study, we examined how well MSCs were predicted by an embodied model (switch / no-switch between perceptual modalities) versus a linguistic model (switch / no-switch between linguistic distributional clusters), in behavioural (RT) and continuous event-related EEG potentials (ERP) paradigms. The behavioural data supported the linguistic model in explaining MSCs and found MSCs to be moderated by the target modality, the ERPs showed that linguistic distributional pattern played a crucial role in the neural activations of MSCs. What used to be found as a result of perceptual switching (e.g., "early N400" effect) could be better explained by the linguistic model. The embodied component was activated later than the linguistic component, accounting for activations associated with semantic representation (typically in N400 area). Later during processing, both components were active for decision making (often manifested as LPC).

Optimizing Military Human-Robot Teaming: An Evaluation of Task Load and Modality Switch Cost

The effect of task switching on performance has been examined in many different fields and contexts. Sudden changes in task load can significantly impair performance, which can have detrimental consequences in dismounted military operations. As technology is advancing, robots are sought to take on the role of a teammate to the human soldier in the field. Robot-to-human communication modality may need to switch when mayhem occurs in military missions. Modality switching has been associated with performance decrements, although these effects are largely unknown in military human-robot teaming situations. The present study examined the cost associated with switching task demand and robot-to-human communication modality type on performance in a simulated cordon-and-search mission. The results showed that switches in task load affected threat detection performance. Auditory reporting increased performance more than visual reporting in low-after-high task load epochs. Performance with auditory reports was also higher in high-after-low demand blocks than low-after-high. The effect of switching needs to be taken into account for high-stakes human-robot interactions.

The modality switching costs of Chinese–English bilinguals in the processing of L1 and L2

Modality switching cost indicates that people’s performance becomes worse when they judge sequential information that is related to different sensory modalities than judging information that is related to the same modality. In this study, we conducted three experiments on proficient and non-proficient bilingual individuals to investigate the modality switching costs in L1 and L2 processing separately. In Experiment 1, materials were L1 and L2 words that were either conceptually related to a visual modality (e.g., light) or related to an auditory modality (e.g., song). The modality switching costs were investigated in a lexical decision task in both L1 and L2. Experiment 2 further explored the modality switching costs while weakening the activation level of the perceptual modality by adding a set of fillers. Experiment 3 used a word-naming task to explore the modality switching effect in language production in L1 and L2. Results of these experiments showed that the modality switching costs appeared in both language comprehension and production in L1 and L2 conditions. The magnitude of the modality switching costs was conditionally modulated by the L2 proficiency level, such as in the L2 condition in Experiment 1 and in both L1 and L2 conditions in Experiment 3. These results suggest that sensorimotor simulation is involved in not only language comprehension but also language production. The sensorimotor simulation that is acquired in L1 can be transferred to L2.