Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing

Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which allows researchers to simulate environments in controlled laboratory conditions with high levels of sense of presence and interactivity, is becoming more popular in emotion research. Moreover, its synergy with implicit measurements and machine-learning techniques has the potential to impact transversely in many research areas, opening new opportunities for the scientific community. This paper presents a systematic review of the emotion recognition research undertaken with physiological and behavioural measures using head-mounted displays as elicitation devices. The results highlight the evolution of the field, give a clear perspective using aggregated analysis, reveal the current open issues and provide guidelines for future research.

Download Full-text

Cybersickness in current-generation virtual reality head-mounted displays: systematic review and outlook

Virtual Reality ◽

10.1007/s10055-021-00513-6 ◽

2021 ◽

Author(s):

Polona Caserman ◽

Augusto Garcia-Agundez ◽

Alvar Gámez Zerban ◽

Stefan Göbel

Keyword(s):

Systematic Review ◽

Virtual Reality ◽

Meta Analysis ◽

Subjective Evaluation ◽

Future Research ◽

Current Generation ◽

Research Areas ◽

Head Mounted Displays ◽

Skin Response ◽

Golden Standard

AbstractCybersickness (CS) is a term used to refer to symptoms, such as nausea, headache, and dizziness that users experience during or after virtual reality immersion. Initially discovered in flight simulators, commercial virtual reality (VR) head-mounted displays (HMD) of the current generation also seem to cause CS, albeit in a different manner and severity. The goal of this work is to summarize recent literature on CS with modern HMDs, to determine the specificities and profile of immersive VR-caused CS, and to provide an outlook for future research areas. A systematic review was performed on the databases IEEE Xplore, PubMed, ACM, and Scopus from 2013 to 2019 and 49 publications were selected. A summarized text states how different VR HMDs impact CS, how the nature of movement in VR HMDs contributes to CS, and how we can use biosensors to detect CS. The results of the meta-analysis show that although current-generation VR HMDs cause significantly less CS ($$p<0.001$$ p < 0.001 ), some symptoms remain as intense. Further results show that the nature of movement and, in particular, sensory mismatch as well as perceived motion have been the leading cause of CS. We suggest an outlook on future research, including the use of galvanic skin response to evaluate CS in combination with the golden standard (Simulator Sickness Questionnaire, SSQ) as well as an update on the subjective evaluation scores of the SSQ.

Download Full-text

Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors

Scientific Reports ◽

10.1038/s41598-018-32063-4 ◽

2018 ◽

Vol 8 (1) ◽

Cited By ~ 55

Author(s):

Javier Marín-Morales ◽

Juan Luis Higuera-Trujillo ◽

Alberto Greco ◽

Jaime Guixeres ◽

Carmen Llinares ◽

...

Keyword(s):

Virtual Reality ◽

Emotion Recognition ◽

Affective Computing ◽

Wearable Sensors

Download Full-text

A Machine Learning Approach to EEG-based Prediction of Human Affective States Using Recursive Feature Elimination Method

MATEC Web of Conferences ◽

10.1051/matecconf/202133504001 ◽

2021 ◽

Vol 335 ◽

pp. 04001

Author(s):

Didar Dadebayev ◽

Goh Wei Wei ◽

Tan Ee Xion

Keyword(s):

Feature Selection ◽

Fractal Dimension ◽

Emotion Recognition ◽

Affective Computing ◽

Feature Selection Method ◽

Recursive Feature Elimination ◽

Support Vector ◽

Emotional States ◽

Affective States ◽

Emotion Classification

Emotion recognition, as a branch of affective computing, has attracted great attention in the last decades as it can enable more natural brain-computer interface systems. Electroencephalography (EEG) has proven to be an effective modality for emotion recognition, with which user affective states can be tracked and recorded, especially for primitive emotional events such as arousal and valence. Although brain signals have been shown to correlate with emotional states, the effectiveness of proposed models is somewhat limited. The challenge is improving accuracy, while appropriate extraction of valuable features might be a key to success. This study proposes a framework based on incorporating fractal dimension features and recursive feature elimination approach to enhance the accuracy of EEG-based emotion recognition. The fractal dimension and spectrum-based features to be extracted and used for more accurate emotional state recognition. Recursive Feature Elimination will be used as a feature selection method, whereas the classification of emotions will be performed by the Support Vector Machine (SVM) algorithm. The proposed framework will be tested with a widely used public database, and results are expected to demonstrate higher accuracy and robustness compared to other studies. The contributions of this study are primarily about the improvement of the EEG-based emotion classification accuracy. There is a potential restriction of how generic the results can be as different EEG dataset might yield different results for the same framework. Therefore, experimenting with different EEG dataset and testing alternative feature selection schemes can be very interesting for future work.

Download Full-text

The Use of Portable EEG Devices in Development of Immersive Virtual Reality Environments for Converting Emotional States into Specific Commands

Proceedings ◽

10.3390/proceedings2020054043 ◽

2020 ◽

Vol 54 (1) ◽

pp. 43

Author(s):

Catarina Sá ◽

Paulo Veloso Gomes ◽

António Marques ◽

António Correia

Keyword(s):

Virtual Reality ◽

Portable Devices ◽

Immersive Virtual Reality ◽

Emotional States ◽

Social Functions ◽

Interactive Environments ◽

Oculus Rift ◽

Non Invasive ◽

Brain Signals

The application of electroencephalography electrodes in Virtual Reality (VR) glasses allows users to relate cognitive, emotional, and social functions with the exposure to certain stimuli. The development of non-invasive portable devices, coupled with VR, allows for the collection of electroencephalographic data. One of the devices that embraced this new trend is Looxid LinkTM, a system that adds electroencephalography to HTC VIVETM, VIVE ProTM, VIVE Pro EyeTM, or Oculus Rift STM glasses to create interactive environments using brain signals. This work analyzes the possibility of using the Looxid LinkTM device to perceive, evaluate and monitor the emotions of users exposed to VR.

Download Full-text

Analyzing Emotional Oscillatory Brain Network for Valence and Arousal-Based Emotion Recognition Using EEG Data

International Journal of Information Technology & Decision Making ◽

10.1142/s0219622019500238 ◽

2019 ◽

Vol 18 (04) ◽

pp. 1359-1378

Author(s):

Jianzhuo Yan ◽

Hongzhi Kuai ◽

Jianhui Chen ◽

Ning Zhong

Keyword(s):

Emotion Recognition ◽

Affective Computing ◽

Affective State ◽

Feature Selection Method ◽

Emotional Valence ◽

Brain Network ◽

Previous Method ◽

Emotional States ◽

Eeg Data ◽

Valence And Arousal

Emotion recognition is a highly noteworthy and challenging work in both cognitive science and affective computing. Currently, neurobiology studies have revealed the partially synchronous oscillating phenomenon within brain, which needs to be analyzed from oscillatory synchronization. This combination of oscillations and synchronism is worthy of further exploration to achieve inspiring learning of the emotion recognition models. In this paper, we propose a novel approach of valence and arousal-based emotion recognition using EEG data. First, we construct the emotional oscillatory brain network (EOBN) inspired by the partially synchronous oscillating phenomenon for emotional valence and arousal. And then, a coefficient of variation and Welch’s [Formula: see text]-test based feature selection method is used to identify the core pattern (cEOBN) within EOBN for different emotional dimensions. Finally, an emotional recognition model (ERM) is built by combining cEOBN-inspired information obtained in the above process and different classifiers. The proposed approach can combine oscillation and synchronization characteristics of multi-channel EEG signals for recognizing different emotional states under the valence and arousal dimensions. The cEOBN-based inspired information can effectively reduce the dimensionality of the data. The experimental results show that the previous method can be used to detect affective state at a reasonable level of accuracy.

Download Full-text

EEG-Based Emotion Recognition: A State-of-the-Art Review of Current Trends and Opportunities

Computational Intelligence and Neuroscience ◽

10.1155/2020/8875426 ◽

2020 ◽

Vol 2020 ◽

pp. 1-19

Author(s):

Nazmi Sofian Suhaimi ◽

James Mountstephens ◽

Jason Teo

Keyword(s):

Emotion Recognition ◽

State Of The Art ◽

Research Community ◽

Human Interaction ◽

Human Cognition ◽

Future Research ◽

Emotional States ◽

Human Beings ◽

Eeg Signals ◽

Recent Developments

Emotions are fundamental for human beings and play an important role in human cognition. Emotion is commonly associated with logical decision making, perception, human interaction, and to a certain extent, human intelligence itself. With the growing interest of the research community towards establishing some meaningful “emotional” interactions between humans and computers, the need for reliable and deployable solutions for the identification of human emotional states is required. Recent developments in using electroencephalography (EEG) for emotion recognition have garnered strong interest from the research community as the latest developments in consumer-grade wearable EEG solutions can provide a cheap, portable, and simple solution for identifying emotions. Since the last comprehensive review was conducted back from the years 2009 to 2016, this paper will update on the current progress of emotion recognition using EEG signals from 2016 to 2019. The focus on this state-of-the-art review focuses on the elements of emotion stimuli type and presentation approach, study size, EEG hardware, machine learning classifiers, and classification approach. From this state-of-the-art review, we suggest several future research opportunities including proposing a different approach in presenting the stimuli in the form of virtual reality (VR). To this end, an additional section devoted specifically to reviewing only VR studies within this research domain is presented as the motivation for this proposed new approach using VR as the stimuli presentation device. This review paper is intended to be useful for the research community working on emotion recognition using EEG signals as well as for those who are venturing into this field of research.

Download Full-text

A comparison of immersive virtual reality with traditional neuropsychological measures in the assessment of executive functions

Acta Neuropsychiatrica ◽

10.1017/neu.2017.14 ◽

2017 ◽

Vol 30 (2) ◽

pp. 79-89 ◽

Cited By ~ 11

Author(s):

Sophie Melissa Clare Davison ◽

Catherine Deeprose ◽

Sylvia Terbeck

Keyword(s):

Older Adults ◽

Virtual Reality ◽

Real World ◽

Future Research ◽

Immersive Virtual Reality ◽

Neuropsychological Measures ◽

Seating Arrangement ◽

Age Related ◽

Initial Work ◽

Neuropsychological Tasks

ObjectiveThis study investigated immersive virtual reality (IVR), as a novel technique to test executive function of healthy younger and older adults. We predicted IVR tasks to have greater predictive power than traditional measures when assessing age-related cognitive functioning due to the real-world validity of the tasks.MethodsParticipants (n=40) completed the Stroop colour–word test and the trail-making test (TMT) as traditional and commonly used assessments of executive functioning. Participants then completed three IVR tasks; a seating arrangement task, an item location task (both set in a virtual chemistry lab), and a virtual parking simulator.ResultsYounger adults completed significantly more parking simulator levels (p<0.001), placed significantly more objects (p<0.001), and located significantly more items than older adults (p<0.01), demonstrating higher levels of performance. Significant correlations were found between performance on traditional neuropsychological measures and IVR measures. For example, Stroop CW performance significantly correlated with the number of parking simulator levels completed (τ=0.43, p<0.01). This suggests that IVR measures assess the same underlying cognitive constructs as traditional tasks. In addition, IVR measures contributed a significant percentage of the explained variance in age.ConclusionIVR measures (i.e. number of parking simulator levels completed and number of objects placed in the seating arrangement task) were found to be stronger contributors than existing traditional neuropsychological tasks in predicting age-related cognitive decline. Future research should investigate the implementation of these real-world-based tasks in clinical groups given this promising initial work.

Download Full-text

Attentional Skills in Soccer: Evaluating the Involvement of Attention in Executing a Goalkeeping Task in Virtual Reality

Applied Sciences ◽

10.3390/app11199341 ◽

2021 ◽

Vol 11 (19) ◽

pp. 9341

Author(s):

Andria Shimi ◽

Vanessa Tsestou ◽

Marios Hadjiaros ◽

Kleanthis Neokleous ◽

Marios Avraamides

Keyword(s):

Virtual Reality ◽

Executive Control ◽

Cognitive Abilities ◽

Cognitive Skills ◽

Critical Role ◽

Attention Network Test ◽

Immersive Virtual Reality ◽

Attention Network ◽

Physical Abilities

Physical abilities are essential to goalkeepers in soccer but the involved cognitive abilities for these players have only recently become the focus of extensive research. In this study, we investigated the role of different aspects of attention in a basic goalkeeping task in soccer. One hundred participants assumed the role of a goalkeeper in immersive virtual reality (VR) and carried out a task that entailed blocking balls shot towards their goal. In addition, they carried out two computerized tasks each assessing different attentional abilities: the Attention Network Test provided scores for three well-established networks of attention, namely the alerting, the orienting, and the executive control. The Whack-a-Mole task evaluated inhibitory control, by measuring performance in a classic Go/No-Go task and tapping on response inhibition. A regression analysis revealed that all three attention network scores contributed to performance in the VR goalkeeping task. Furthermore, performance in the Whack-a-Mole task correlated significantly with performance in the VR goalkeeping task. Overall, findings confirm that cognitive skills relating to attention play a critical role in the efficient execution of soccer-specific tasks. These findings have important implications for the training of cognitive skills in sports.

Download Full-text

Overcoming Key Barriers for Secondary Cloud Forest Management in Mexico

Land ◽

10.3390/land10101078 ◽

2021 ◽

Vol 10 (10) ◽

pp. 1078

Author(s):

Tarin Toledo-Aceves ◽

Manuel R. Guariguata ◽

Sven Günter ◽

Luciana Porter-Bolland ◽

Leticia Merino

Keyword(s):

Ecosystem Services ◽

Stand Structure ◽

Cloud Forest ◽

Critical Role ◽

Sustainable Use ◽

Floristic Composition ◽

Economic Incentives ◽

Future Research ◽

Timber Production ◽

Research Areas

Secondary cloud forests (SCFs), those that regenerate naturally following abandonment of human activities in previously deforested land, are of great value as refuges of high species diversity and for their critical role in hydrological regulation. This opinion paper analyzes the main environmental, socio-economic, and regulatory aspects that currently hamper the sustainable use and conservation of SCFs in Mexico for the provision of timber and ecosystem services. The main constraints identified include contradictory norms and policies and the marginalization of smallholders in timber production activities. Developing economic incentives for forest product harvesting and provision of ecosystem services derived from SCFs, while also addressing legal and normative aspects related to their sustainable use, is paramount. Given the high heterogeneity in floristic composition and stand structure of SCFs among localities, technical and social norms for sustainable use should be sufficiently flexible to allow adaptive management approaches. Future research areas should be focused on monitoring the response of SCFs to silvicultural interventions, documenting existing traditional practices as well as conducting socio-economic analyses of timber production and associated ecosystem services. This is essential for developing sound policies and approaches for the sustainable use and long-term management of SCFs in Mexico.

Download Full-text

Shimi et al_GKVR1

10.31234/osf.io/y5q2c ◽

2021 ◽

Author(s):

Andria Shimi ◽

Vanessa Tsestou ◽

Marios Hatziaros ◽

Kleanthis Neokleous ◽

Marios N Avraamides

Keyword(s):

Virtual Reality ◽

Executive Control ◽

Cognitive Abilities ◽

Cognitive Skills ◽

Critical Role ◽

Attention Network Test ◽

Immersive Virtual Reality ◽

Attention Network ◽

Physical Abilities

Soccer is one of the most popular sports and goalkeepers are central to a team’s winning. Physical abilities are essential to goalkeepers but the involved cognitive abilities for these players are understudied and not well understood. In this study, we investigated the role of different aspects of attention in a goalkeeping task in soccer. Participants assumed the role of a goalkeeper in immersive Virtual Reality and carried out a task that entailed blocking balls shot towards their goal. In addition, they carried out two computerized tasks each assessing different attentional abilities: the Attention Network Test provided scores for three well-established networks of attention, namely the alerting, the orienting, and the executive control. The Whack-a-Mole task evaluated inhibitory control, by measuring performance in a classic Go/No-Go task and tapping on response inhibition. Results revealed that all three attention network scores predicted performance in the VR goalkeeping task. Furthermore, performance in the Whack-a-Mole task correlated significantly with performance in the VR goalkeeping task. Overall, findings confirm that cognitive skills relating to attention play a critical role in the efficient execution of soccer-specific tasks. These findings have important implications for the training of cognitive skills in sports.

Download Full-text