Facial Expression and Electrodermal Activity Analysis for Continuous Pain Intensity Monitoring On The X-ITE Pain Database

Abstract Automatic systems enable continuous monitoring of patients' pain intensity as shown in prior studies. Facial expression and physiological data such as electrodermal activity (EDA) are very informative for pain recognition. The features extracted from EDA indicate the stress and anxiety caused by different levels of pain. In this paper, we investigate using the EDA modality and fusing two modalities (frontal RGB video and EDA) for continuous pain intensity recognition with the X-ITE Pain Database. Further, we compare the performance of automated models before and after reducing the imbalance problem in heat and electrical pain datasets that include phasic (short) and tonic (long) stimuli. We use three distinct real-time methods: A Random Forest (RF) baseline methods [Random Forest classifier (RFc) and Random Forest regression (RFr)], Long-Short Term Memory Network (LSTM), and LSTM using sample weighting method (called LSTM-SW). Experimental results (1) report the first results of continuous pain intensity recognition using EDA data on the X-ITE Pain Database, (2) show that LSTM and LSTM-SW outperform guessing and baseline methods (RFc and RFr), (3) confirm that the electrodermal activity (EDA) with most models is the best, (4) show the fusion of the output of two LSTM models using facial expression and EDA data (called Decision Fusion = DF). The DF improves results further with some datasets (e.g. Heat Phasic Dataset (HTD)).

Download Full-text

Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?

Sensors ◽

10.3390/s20174723 ◽

2020 ◽

Vol 20 (17) ◽

pp. 4723

Author(s):

Patrícia Bota ◽

Chen Wang ◽

Ana Fred ◽

Hugo Silva

Keyword(s):

Emotion Recognition ◽

Classification Accuracy ◽

Feature Fusion ◽

State Of The Art ◽

Electrodermal Activity ◽

Decision Fusion ◽

Classification Performance ◽

Physiological Data ◽

Systematic Analysis ◽

Public Datasets

Emotion recognition based on physiological data classification has been a topic of increasingly growing interest for more than a decade. However, there is a lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of expected accuracy, just to name a few limitations. In this work, we evaluate emotion in terms of low/high arousal and valence classification through Supervised Learning (SL), Decision Fusion (DF) and Feature Fusion (FF) techniques using multimodal physiological data, namely, Electrocardiography (ECG), Electrodermal Activity (EDA), Respiration (RESP), or Blood Volume Pulse (BVP). The main contribution of our work is a systematic study across five public datasets commonly used in the Emotion Recognition (ER) state-of-the-art, namely: (1) Classification performance analysis of ER benchmarking datasets in the arousal/valence space; (2) Summarising the ranges of the classification accuracy reported across the existing literature; (3) Characterising the results for diverse classifiers, sensor modalities and feature set combinations for ER using accuracy and F1-score; (4) Exploration of an extended feature set for each modality; (5) Systematic analysis of multimodal classification in DF and FF approaches. The experimental results showed that FF is the most competitive technique in terms of classification accuracy and computational complexity. We obtain superior or comparable results to those reported in the state-of-the-art for the selected datasets.

Download Full-text

A Multi-Layered LSTM for Predicting Physician Stress During an ED Shift

10.21203/rs.3.rs-318589/v1 ◽

2021 ◽

Author(s):

Vishnunarayan Girishan Prabhu ◽

Kevin Taaffe ◽

Ronald Pirrallo

Keyword(s):

Short Term Memory ◽

Electrodermal Activity ◽

Physiological Data ◽

Physiological Measures ◽

Work Related ◽

The Us ◽

Related Stress ◽

High Level ◽

Physician Stress ◽

Attending Physicians

Abstract Emergency Department (ED) physicians are faced with complex care settings, including a high level of uncertainty and intensity. Burnout among physicians is increasing every year, and ED physicians are one of the groups most prone to burnouts and work-related stress in the US. This research focused on developing a supervised Long Short-Term Memory (LSTM) artificial neural network model to predict a physician’s stress level based on their physiological data. Twelve attending physicians working a 3:00 pm - 11:00 pm shift at Greenville Memorial Hospital (GMH) in Greenville, SC, participated in the study. Stress levels were estimated using physiological measures, including heart rate and electrodermal activity. Over 100 hours of physiological data were collected from 12 eight-hour shifts. Initially, an 80:20 split was used on the 12 individual datasets for training and testing the model. Further, to develop a generalizable model, the data were merged, and a 60:20:20 split was used for training, validating, and testing the model. On the test set, the model achieved values of 0.98, 0.17, and 0.005 as the R-squared, RMSE, and loss for EDA data and 0.99, 0.41, and 0.002 for HR data.

Download Full-text

Prediction of Relative Physical Activity Intensity Using Multimodal Sensing of Physiological Data

Sensors ◽

10.3390/s19204509 ◽

2019 ◽

Vol 19 (20) ◽

pp. 4509 ◽

Cited By ~ 2

Author(s):

Alok Kumar Chowdhury ◽

Dian Tjondronegoro ◽

Vinod Chandran ◽

Jinglan Zhang ◽

Stewart G. Trost

Keyword(s):

Physical Activity ◽

Machine Learning ◽

Perceived Exertion ◽

Feature Fusion ◽

Electrodermal Activity ◽

Decision Fusion ◽

Sensor Data ◽

Physiological Data ◽

Support Vector ◽

Improve Performance

This study examined the feasibility of a non-laboratory approach that uses machine learning on multimodal sensor data to predict relative physical activity (PA) intensity. A total of 22 participants completed up to 7 PA sessions, where each session comprised 5 trials (sitting and standing, comfortable walk, brisk walk, jogging, running). Participants wore a wrist-strapped sensor that recorded heart-rate (HR), electrodermal activity (Eda) and skin temperature (Temp). After each trial, participants provided ratings of perceived exertion (RPE). Three classifiers, including random forest (RF), neural network (NN) and support vector machine (SVM), were applied independently on each feature set to predict relative PA intensity as low (RPE ≤ 11), moderate (RPE 12–14), or high (RPE ≥ 15). Then, both feature fusion and decision fusion of all combinations of sensor modalities were carried out to investigate the best combination. Among the single modality feature sets, HR provided the best performance. The combination of modalities using feature fusion provided a small improvement in performance. Decision fusion did not improve performance over HR features alone. A machine learning approach using features from HR provided acceptable predictions of relative PA intensity. Adding features from other sensing modalities did not significantly improve performance.

Download Full-text

MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition

Frontiers in Human Neuroscience ◽

10.3389/fnhum.2021.621493 ◽

2021 ◽

Vol 15 ◽

Author(s):

Ruixin Li ◽

Yan Liang ◽

Xiaojian Liu ◽

Bingbing Wang ◽

Wenxin Huang ◽

...

Keyword(s):

Facial Expression ◽

Emotion Recognition ◽

Open Source ◽

Short Term Memory ◽

Weight Enumerator ◽

Physiological Data ◽

Support Vector ◽

Decision Level ◽

Model Based ◽

The Subject

Emotion recognition plays an important role in intelligent human–computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram (EEG) and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a decision level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network (CNN) based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine (SVM) and a subject-independent model based on long short-term memory network (LSTM). In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals (DEAP) dataset and the Multimodal Database for Affect Recognition and Implicit Tagging (MAHNOB-HCI) dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated.

Download Full-text

Sensor-Based Prediction of Mental Effort during Learning from Physiological Data: A Longitudinal Case Study

Signals ◽

10.3390/signals2040051 ◽

2021 ◽

Vol 2 (4) ◽

pp. 886-901

Author(s):

Ankita Agarwal ◽

Josephine Graft ◽

Noah Schroeder ◽

William Romine

Keyword(s):

Short Term Memory ◽

Electrodermal Activity ◽

Tracking System ◽

Learning Algorithms ◽

Mental Effort ◽

Machine Learning Algorithms ◽

Self Report ◽

Physiological Data ◽

Longitudinal Case Study

Trackers for activity and physical fitness have become ubiquitous. Although recent work has demonstrated significant relationships between mental effort and physiological data such as skin temperature, heart rate, and electrodermal activity, we have yet to demonstrate their efficacy for the forecasting of mental effort such that a useful mental effort tracker can be developed. Given prior difficulty in extracting relationships between mental effort and physiological responses that are repeatable across individuals, we make the case that fusing self-report measures with physiological data within an internet or smartphone application may provide an effective method for training a useful mental effort tracking system. In this case study, we utilized over 90 h of data from a single participant over the course of a college semester. By fusing the participant’s self-reported mental effort in different activities over the course of the semester with concurrent physiological data collected with the Empatica E4 wearable sensor, we explored questions around how much data were needed to train such a device, and which types of machine-learning algorithms worked best. We concluded that although baseline models such as logistic regression and Markov models provided useful explanatory information on how the student’s physiology changed with mental effort, deep-learning algorithms were able to generate accurate predictions using the first 28 h of data for training. A system that combines long short-term memory and convolutional neural networks is recommended in order to generate smooth predictions while also being able to capture transitions in mental effort when they occur in the individual using the device.

Download Full-text

Automatic vs. Human Recognition of Pain Intensity from Facial Expression on the X-ITE Pain Database

Sensors ◽

10.3390/s21093273 ◽

2021 ◽

Vol 21 (9) ◽

pp. 3273

Author(s):

Ehsan Othman ◽

Philipp Werner ◽

Frerk Saxen ◽

Ayoub Al-Hamadi ◽

Sascha Gruss ◽

...

Keyword(s):

Random Forest ◽

Pain Intensity ◽

Human Performance ◽

Experimental Studies ◽

Human Observer ◽

Weighting Method ◽

Face Images ◽

Proposed Model ◽

Sample Weighting ◽

Better Than

Prior work on automated methods demonstrated that it is possible to recognize pain intensity from frontal faces in videos, while there is an assumption that humans are very adept at this task compared to machines. In this paper, we investigate whether such an assumption is correct by comparing the results achieved by two human observers with the results achieved by a Random Forest classifier (RFc) baseline model (called RFc-BL) and by three proposed automated models. The first proposed model is a Random Forest classifying descriptors of Action Unit (AU) time series; the second is a modified MobileNetV2 CNN classifying face images that combine three points in time; and the third is a custom deep network combining two CNN branches using the same input as for MobileNetV2 plus knowledge of the RFc. We conduct experiments with X-ITE phasic pain database, which comprises videotaped responses to heat and electrical pain stimuli, each of three intensities. Distinguishing these six stimulation types plus no stimulation was the main 7-class classification task for the human observers and automated approaches. Further, we conducted reduced 5-class and 3-class classification experiments, applied Multi-task learning, and a newly suggested sample weighting method. Experimental results show that the pain assessments of the human observers are significantly better than guessing and perform better than the automatic baseline approach (RFc-BL) by about 1%; however, the human performance is quite poor due to the challenge that pain that is ethically allowed to be induced in experimental studies often does not show up in facial reaction. We discovered that downweighting those samples during training improves the performance for all samples. The proposed RFc and two-CNNs models (using the proposed sample weighting) significantly outperformed the human observer by about 6% and 7%, respectively.

Download Full-text

Calibrations of Low-Cost Air Pollution Monitoring Sensors for CO, NO2, O3, and SO2

Sensors ◽

10.3390/s21010256 ◽

2021 ◽

Vol 21 (1) ◽

pp. 256

Author(s):

Pengfei Han ◽

Han Mei ◽

Di Liu ◽

Ning Zeng ◽

Xiao Tang ◽

...

Keyword(s):

Random Forest ◽

Linear Regression ◽

Short Term Memory ◽

Hot Spot ◽

Low Cost ◽

Field Evaluation ◽

Pollution Monitoring ◽

Coefficient Of Determination ◽

Air Pollution Monitoring ◽

National Monitoring

Pollutant gases, such as CO, NO2, O3, and SO2 affect human health, and low-cost sensors are an important complement to regulatory-grade instruments in pollutant monitoring. Previous studies focused on one or several species, while comprehensive assessments of multiple sensors remain limited. We conducted a 12-month field evaluation of four Alphasense sensors in Beijing and used single linear regression (SLR), multiple linear regression (MLR), random forest regressor (RFR), and neural network (long short-term memory (LSTM)) methods to calibrate and validate the measurements with nearby reference measurements from national monitoring stations. For performances, CO > O3 > NO2 > SO2 for the coefficient of determination (R2) and root mean square error (RMSE). The MLR did not increase the R2 after considering the temperature and relative humidity influences compared with the SLR (with R2 remaining at approximately 0.6 for O3 and 0.4 for NO2). However, the RFR and LSTM models significantly increased the O3, NO2, and SO2 performances, with the R2 increasing from 0.3–0.5 to >0.7 for O3 and NO2, and the RMSE decreasing from 20.4 to 13.2 ppb for NO2. For the SLR, there were relatively larger biases, while the LSTMs maintained a close mean relative bias of approximately zero (e.g., <5% for O3 and NO2), indicating that these sensors combined with the LSTMs are suitable for hot spot detection. We highlight that the performance of LSTM is better than that of random forest and linear methods. This study assessed four electrochemical air quality sensors and different calibration models, and the methodology and results can benefit assessments of other low-cost sensors.

Download Full-text

Real-Time AI-Based Informational Decision-Making Support System Utilizing Dynamic Text Sources

Applied Sciences ◽

10.3390/app11136237 ◽

2021 ◽

Vol 11 (13) ◽

pp. 6237

Author(s):

Azharul Islam ◽

KyungHi Chang

Keyword(s):

Machine Learning ◽

Decision Making ◽

Random Forest ◽

Support System ◽

Classification Accuracy ◽

Short Term Memory ◽

Learning Algorithm ◽

Unstructured Data ◽

Stochastic Gradient Descent ◽

Decision Making Support

Unstructured data from the internet constitute large sources of information, which need to be formatted in a user-friendly way. This research develops a model that classifies unstructured data from data mining into labeled data, and builds an informational and decision-making support system (DMSS). We often have assortments of information collected by mining data from various sources, where the key challenge is to extract valuable information. We observe substantial classification accuracy enhancement for our datasets with both machine learning and deep learning algorithms. The highest classification accuracy (99% in training, 96% in testing) was achieved from a Covid corpus which is processed by using a long short-term memory (LSTM). Furthermore, we conducted tests on large datasets relevant to the Disaster corpus, with an LSTM classification accuracy of 98%. In addition, random forest (RF), a machine learning algorithm, provides a reasonable 84% accuracy. This research’s main objective is to increase the application’s robustness by integrating intelligence into the developed DMSS, which provides insight into the user’s intent, despite dealing with a noisy dataset. Our designed model selects the random forest and stochastic gradient descent (SGD) algorithms’ F1 score, where the RF method outperforms by improving accuracy by 2% (to 83% from 81%) compared with a conventional method.

Download Full-text