Putting perception into action: Inverse optimal control for continuous psychophysics

Psychophysical methods are a cornerstone of psychology, cognitive science, and neuroscience where they have been used to quantify behavior and its neural correlates for a vast range of mental phenomena. Their power derives from the combination of controlled experiments and rigorous analysis through signal detection theory. Unfortunately, they require many tedious trials and preferably highly trained participants. A recently developed approach, continuous psychophysics, promises to transform the field by abandoning the rigid trial structure involving binary responses and replacing it with continuous behavioral adjustments to dynamic stimuli. However, what has precluded wide adoption of this approach is that current analysis methods recover perceptual thresholds, which are one order of magnitude larger compared to equivalent traditional psychophysical experiments. Here we introduce a computational analysis framework for continuous psychophysics based on Bayesian inverse optimal control. We show via simulations and on previously published data that this not only recovers the perceptual thresholds but additionally estimates subjects' action variability, internal behavioral costs, and subjective beliefs about the experimental stimulus dynamics. Taken together, we provide further evidence for the importance of including acting uncertainties, subjective beliefs, and, crucially, the intrinsic costs of behavior, even in experiments seemingly only investigating perception.

Eye tracking and tourism research: A systematic literature review

In recent years, the number of studies in tourism using the eye tracking technique has increased and started generating valuable information for both academics and the industry. However, there is a gap in the literature concerning systematic reviews focused on recent articles and their findings. Thus, the aim of this study is to close this gap by systematically analysing 70 research papers tackling the subject of eye tracking in tourism and published in highly ranked tourism journals. The study identifies the most popular topics and trends for eye tracking research, as well as the most used types of visual stimuli, such as exhibitions, restaurant menus, promotional pictures or websites. The study also details on measurements specific for the analysis of eye tracking data, including fixations, saccades and heat maps. Results are emphasized along with their theoretical and practical implications. In addition, we highlight the lack of the use of dynamic stimuli in the existing literature and suggest further research directions using the eye tracking technique.

The Generation of Involuntary Mental Imagery in an Ecologically-Valid Task

Laboratory tasks (e.g., the flanker task) reveal that incidental stimuli (e.g., distractors) can reliably trigger involuntary conscious imagery. Can such involuntary effects, involving competing representations, arise during dual-task conditions? Another concern about these laboratory tasks is whether such effects arise in highly ecologically-valid conditions. For example, do these effects arise from tasks involving dynamic stimuli (e.g., simulations of semi-automated driving experiences)? The data from our experiment suggest that the answer to our two questions is yes. Subjects were presented with video footage of the kinds of events that one would observe if one were seated in the driver's seat of a semi-automated vehicle. Before being presented with this video footage, subjects had been trained to respond to street signs according to laboratory techniques that cause stimulus-elicited involuntary imagery. After training, in the Respond condition, subjects responded to the signs; in the Suppress condition, subjects were instructed to not respond to the signs in the video footage. Subjects in the Suppress condition reported involuntary imagery on a substantive proportion of the trials. Such involuntary effects arose even under dual-task conditions (while performing the n-back task or psychomotor vigilance task). The present laboratory task has implications for semi-automated driving, because the safe interaction between driver and vehicle requires that the communicative signals from vehicle to driver be effective at activating the appropriate cognitions and behavioral inclinations. In addition, our data from the dual-task conditions provide constraints for theoretical models of cognitive resources.

Data-driven group comparisons of eye fixations to dynamic stimuli

Recent advances in software and hardware have allowed eye tracking to move away from static images to more ecologically relevant video streams. The analysis of eye tracking data for such dynamic stimuli, however, is not without challenges. The frame-by-frame coding of regions of interest (ROIs) is labour-intensive and computer vision techniques to automatically code such ROIs are not yet mainstream, restricting the use of such stimuli. Combined with the more general problem of defining relevant ROIs for video frames, methods are needed that facilitate data analysis. Here, we present a first evaluation of an easy-to-implement data-driven method with the potential to address these issues. To test the new method, we examined the differences in eye movements of self-reported politically left- or right-wing leaning participants to video clips of left- and right-wing politicians. The results show that our method can accurately predict group membership on the basis of eye movement patterns, isolate video clips that best distinguish people on the political left–right spectrum, and reveal the section of each video clip with the largest group differences. Our methodology thereby aids the understanding of group differences in gaze behaviour, and the identification of critical stimuli for follow-up studies or for use in saccade diagnosis.

Face in collision: emotional looming stimuli modulate interpersonal space across development and gender

Basic visual functions have evolved to allow for rapid detection of dynamic stimuli in our surrounding environment. In particular, looming stimuli are of relevance because they are expected to enter the individual’s interpersonal space representing a potential threat. Different studies showed that emotions can modulate the perception of visual looming stimuli and the borders of interpersonal space, defined as the area around the body that individuals maintain between themselves and others during social interactions. Here, we investigated how emotions modulate the perception and the physiological correlates of interpersonal space and whether such indexes change across age and gender. Children and adults were asked to quickly react to emotional looming stimuli while measuring their skin conductance response (SCR). We found that emotional looming stimuli shrink the borders of interpersonal space of males more than females, and that this pattern does not change with age. In addition, adults reacted faster to angry than happy and neutral faces, which is in line with the notion that threatening stimuli capture attention more quickly than other types of emotional stimuli. However, this was not observed in children, suggesting that experience with negative stimuli, rather than the evolutionary meaning they possess, may influence the boundaries of interpersonal space. Overall, our study suggests that interpersonal space is modulated by emotions, but this appears to be modulated by gender and age: while behavioural responses to emotional looming stimuli refine with age, physiological responses are adult-like as early as 5 years of age.

Identifying phenotypic and physiological subgroups of preschoolers with autism spectrum disorder

Background To understand the emergence of symptoms in autism spectrum disorder (ASD), we need to identify the mechanisms that underpin the development of core social skills. Mounting evidence indicates that young children with later ASD attend less to other people, which could compromise learning opportunities with cascading effects. Passive looking behaviour does not tell us about engagement with visual information, but measures of physiological arousal can provide information on the depth of engagement. In the current study, we use heart rate (HR) and heart rate variability (HRV) to measure engagement with social dynamic stimuli in ASD. Methods Sixty-seven preschoolers with ASD and 65 typical developing preschoolers between 2 and 4 years of age participated in a study where HR was measured during viewing of social and non-social videos. Using latent profile analyses, more homogeneous subgroups of children were created based on phenotype and physiology. Results Preschool-aged children with ASD, regardless of their non-verbal, verbal and social competencies, do not differ in overall HR or HRV compared to TD children. However, the ASD group showed a larger increase in HR (more disengagement) than the TD group to later-presented social stimuli. Phenotypic and physiological profiles showed this was primarily the case for children with below average verbal and non-verbal skills, but not necessarily those with more ASD symptoms. Conclusion Children with ASD, especially a subgroup showing moderate cognitive delays, show an increase in HR to social stimuli over time; this may reflect difficulties re-engaging with social information when attention is waning.

Professional actors demonstrate variability, not stereotypical expressions, when portraying emotional states in photographs

It is long hypothesized that there is a reliable, specific mapping between certain emotional states and the facial movements that express those states. This hypothesis is often tested by asking untrained participants to pose the facial movements they believe they use to express emotions during generic scenarios. Here, we test this hypothesis using, as stimuli, photographs of facial configurations posed by professional actors in response to contextually-rich scenarios. The scenarios portrayed in the photographs were rated by a convenience sample of participants for the extent to which they evoked an instance of 13 emotion categories, and actors’ facial poses were coded for their specific movements. Both unsupervised and supervised machine learning find that in these photographs, the actors portrayed emotional states with variable facial configurations; instances of only three emotion categories (fear, happiness, and surprise) were portrayed with moderate reliability and specificity. The photographs were separately rated by another sample of participants for the extent to which they portrayed an instance of the 13 emotion categories; they were rated when presented alone and when presented with their associated scenarios, revealing that emotion inferences by participants also vary in a context-sensitive manner. Together, these findings suggest that facial movements and perceptions of emotion vary by situation and transcend stereotypes of emotional expressions. Future research may build on these findings by incorporating dynamic stimuli rather than photographs and studying a broader range of cultural contexts.

Investigating facilitatory versus inhibitory effects of dynamic social and non-social cues on attention in a realistic space

This study aimed to investigate the facilitatory versus inhibitory effects of dynamic non-predictive central cues presented in a realistic environment. Realistic human-avatars initiated eye contact and then dynamically looked to the left, right or centre of a table. A moving stick served as a non-social control cue and participants localised (Experiment 1) or discriminated (Experiment 2) a contextually relevant target (teapot/teacup). The cues movement took 500 ms and stimulus onset asynchronies (SOA, 150 ms/300 ms/500 ms/1000 ms) were measured from movement initiation. Similar cuing effects were seen for the social avatar and non-social stick cue across tasks. Results showed facilitatory processes without inhibition, though there was some variation by SOA and task. This is the first time facilitatory versus inhibitory processes have been directly investigated where eye contact is initiated prior to gaze shift. These dynamic stimuli allow a better understanding of how attention might be cued in more realistic environments.

Temporal stability of stimulus representation increases along rodent visual cortical hierarchies

Cortical representations of brief, static stimuli become more invariant to identity-preserving transformations along the ventral stream. Likewise, increased invariance along the visual hierarchy should imply greater temporal persistence of temporally structured dynamic stimuli, possibly complemented by temporal broadening of neuronal receptive fields. However, such stimuli could engage adaptive and predictive processes, whose impact on neural coding dynamics is unknown. By probing the rat analog of the ventral stream with movies, we uncovered a hierarchy of temporal scales, with deeper areas encoding visual information more persistently. Furthermore, the impact of intrinsic dynamics on the stability of stimulus representations grew gradually along the hierarchy. A database of recordings from mouse showed similar trends, additionally revealing dependencies on the behavioral state. Overall, these findings show that visual representations become progressively more stable along rodent visual processing hierarchies, with an important contribution provided by intrinsic processing.

Inferences on enacted understanding: using immersive technologies to assess intuitive physical science knowledge

Purpose This study aims to advance the proposal to use immersive virtual learning environments to stimulate and reveal deep-seated knowledge about science, giving instructors and researchers unique possibilities for assessing and identifying intuitive physical science knowledge. Aside from the ability to present rich and dynamic stimuli, these environments afford bodily enactment of people’s understanding, which draws less from declarative knowledge stores and more from everyday experiences with the physical world. Design/methodology/approach The authors ground their proposal in a critical review of the impact of stimulus and task characteristics of traditional physics inventories. Using a grounded theory approach, the authors present classifications and interpretations of observed bodily enactments of physics understandings in a study where participants enacted their understanding of force and motion of space in an immersive, interactive mixed reality (MR) environment. Findings The authors find that instances of these action categories can be interpreted as relating to underlying knowledge, often identified by other studies. The authors thus replicate a number of prior findings, which provide evidence to establish validation for using MR simulation as a tool for identifying people’s physical intuitions. Research limitations/implications This study targeted only a few specific physical science scenarios. Further, while a number of key insights about student knowledge came from the analysis, many of the observations are mere leads in need of further investigation and interpretation rather than core findings. Originality/value Immersive digital learning environments are primarily used for instruction. The authors propose to use and design them for assessment as well. This paper should prompt more research and development in this direction.