Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?

Emotion recognition based on physiological data classification has been a topic of increasingly growing interest for more than a decade. However, there is a lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of expected accuracy, just to name a few limitations. In this work, we evaluate emotion in terms of low/high arousal and valence classification through Supervised Learning (SL), Decision Fusion (DF) and Feature Fusion (FF) techniques using multimodal physiological data, namely, Electrocardiography (ECG), Electrodermal Activity (EDA), Respiration (RESP), or Blood Volume Pulse (BVP). The main contribution of our work is a systematic study across five public datasets commonly used in the Emotion Recognition (ER) state-of-the-art, namely: (1) Classification performance analysis of ER benchmarking datasets in the arousal/valence space; (2) Summarising the ranges of the classification accuracy reported across the existing literature; (3) Characterising the results for diverse classifiers, sensor modalities and feature set combinations for ER using accuracy and F1-score; (4) Exploration of an extended feature set for each modality; (5) Systematic analysis of multimodal classification in DF and FF approaches. The experimental results showed that FF is the most competitive technique in terms of classification accuracy and computational complexity. We obtain superior or comparable results to those reported in the state-of-the-art for the selected datasets.

Download Full-text

A Multi-Branch Feature Fusion Strategy Based on an Attention Mechanism for Remote Sensing Image Scene Classification

Remote Sensing ◽

10.3390/rs13101950 ◽

2021 ◽

Vol 13 (10) ◽

pp. 1950

Author(s):

Cuiping Shi ◽

Xin Zhao ◽

Liguo Wang

Keyword(s):

Remote Sensing ◽

Feature Extraction ◽

Classification Accuracy ◽

Feature Fusion ◽

State Of The Art ◽

Rapid Development ◽

Remote Sensing Image ◽

Classification Performance ◽

Attention Mechanism ◽

Scene Classification

In recent years, with the rapid development of computer vision, increasing attention has been paid to remote sensing image scene classification. To improve the classification performance, many studies have increased the depth of convolutional neural networks (CNNs) and expanded the width of the network to extract more deep features, thereby increasing the complexity of the model. To solve this problem, in this paper, we propose a lightweight convolutional neural network based on attention-oriented multi-branch feature fusion (AMB-CNN) for remote sensing image scene classification. Firstly, we propose two convolution combination modules for feature extraction, through which the deep features of images can be fully extracted with multi convolution cooperation. Then, the weights of the feature are calculated, and the extracted deep features are sent to the attention mechanism for further feature extraction. Next, all of the extracted features are fused by multiple branches. Finally, depth separable convolution and asymmetric convolution are implemented to greatly reduce the number of parameters. The experimental results show that, compared with some state-of-the-art methods, the proposed method still has a great advantage in classification accuracy with very few parameters.

Download Full-text

Prediction of Relative Physical Activity Intensity Using Multimodal Sensing of Physiological Data

Sensors ◽

10.3390/s19204509 ◽

2019 ◽

Vol 19 (20) ◽

pp. 4509 ◽

Cited By ~ 2

Author(s):

Alok Kumar Chowdhury ◽

Dian Tjondronegoro ◽

Vinod Chandran ◽

Jinglan Zhang ◽

Stewart G. Trost

Keyword(s):

Physical Activity ◽

Machine Learning ◽

Perceived Exertion ◽

Feature Fusion ◽

Electrodermal Activity ◽

Decision Fusion ◽

Sensor Data ◽

Physiological Data ◽

Support Vector ◽

Improve Performance

This study examined the feasibility of a non-laboratory approach that uses machine learning on multimodal sensor data to predict relative physical activity (PA) intensity. A total of 22 participants completed up to 7 PA sessions, where each session comprised 5 trials (sitting and standing, comfortable walk, brisk walk, jogging, running). Participants wore a wrist-strapped sensor that recorded heart-rate (HR), electrodermal activity (Eda) and skin temperature (Temp). After each trial, participants provided ratings of perceived exertion (RPE). Three classifiers, including random forest (RF), neural network (NN) and support vector machine (SVM), were applied independently on each feature set to predict relative PA intensity as low (RPE ≤ 11), moderate (RPE 12–14), or high (RPE ≥ 15). Then, both feature fusion and decision fusion of all combinations of sensor modalities were carried out to investigate the best combination. Among the single modality feature sets, HR provided the best performance. The combination of modalities using feature fusion provided a small improvement in performance. Decision fusion did not improve performance over HR features alone. A machine learning approach using features from HR provided acceptable predictions of relative PA intensity. Adding features from other sensing modalities did not significantly improve performance.

Download Full-text

Group-Wise Dynamic Dropout Based on Latent Semantic Variations

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i07.6782 ◽

2020 ◽

Vol 34 (07) ◽

pp. 11229-11236

Author(s):

Zhiwei Ke ◽

Zhiwei Wen ◽

Weicheng Xie ◽

Yi Wang ◽

Linlin Shen

Keyword(s):

Neural Networks ◽

Deep Neural Networks ◽

Semantic Information ◽

State Of The Art ◽

Classification Performance ◽

Network Robustness ◽

Feature Detectors ◽

Data Points ◽

Adversarial Examples ◽

Public Datasets

Dropout regularization has been widely used in various deep neural networks to combat overfitting. It works by training a network to be more robust on information-degraded data points for better generalization. Conventional dropout and variants are often applied to individual hidden units in a layer to break up co-adaptations of feature detectors. In this paper, we propose an adaptive dropout to reduce the co-adaptations in a group-wise manner by coarse semantic information to improve feature discriminability. In particular, we showed that adjusting the dropout probability based on local feature densities can not only improve the classification performance significantly but also enhance the network robustness against adversarial examples in some cases. The proposed approach was evaluated in comparison with the baseline and several state-of-the-art adaptive dropouts over four public datasets of Fashion-MNIST, CIFAR-10, CIFAR-100 and SVHN.

Download Full-text

Geometry Sharing Network for 3D Point Cloud Classification and Segmentation

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i07.6938 ◽

2020 ◽

Vol 34 (07) ◽

pp. 12500-12507 ◽

Cited By ~ 2

Author(s):

Mingye Xu ◽

Zhipeng Zhou ◽

Yu Qiao

Keyword(s):

Point Cloud ◽

State Of The Art ◽

Nearest Neighbors ◽

Classification Performance ◽

3D Point Cloud ◽

Geometric Information ◽

Geometric Transformations ◽

Geometry Similarity ◽

Public Datasets ◽

Point Cloud Classification

In spite of the recent progresses on classifying 3D point cloud with deep CNNs, large geometric transformations like rotation and translation remain challenging problem and harm the final classification performance. To address this challenge, we propose Geometry Sharing Network (GS-Net) which effectively learns point descriptors with holistic context to enhance the robustness to geometric transformations. Compared with previous 3D point CNNs which perform convolution on nearby points, GS-Net can aggregate point features in a more global way. Specially, GS-Net consists of Geometry Similarity Connection (GSC) modules which exploit Eigen-Graph to group distant points with similar and relevant geometric information, and aggregate features from nearest neighbors in both Euclidean space and Eigenvalue space. This design allows GS-Net to efficiently capture both local and holistic geometric features such as symmetry, curvature, convexity and connectivity. Theoretically, we show the nearest neighbors of each point in Eigenvalue space are invariant to rotation and translation. We conduct extensive experiments on public datasets, ModelNet40, ShapeNet Part. Experiments demonstrate that GS-Net achieves the state-of-the-art performances on major datasets, 93.3% on ModelNet40, and are more robust to geometric transformations.

Download Full-text

GameEmo-CapsNet: Emotion Recognition from Single-Channel EEG Signals Using the 1D Capsule Networks

Traitement du signal ◽

10.18280/ts.380612 ◽

2021 ◽

Vol 38 (6) ◽

pp. 1689-1698

Author(s):

Suat Toraman ◽

Ömer Osman Dursun

Keyword(s):

Emotion Recognition ◽

Facial Expressions ◽

Classification Accuracy ◽

Single Channel ◽

Visual Stimuli ◽

Classification Performance ◽

Eeg Signals ◽

Machine Learning Methods ◽

First Time ◽

Better Than

Human emotion recognition with machine learning methods through electroencephalographic (EEG) signals has become a highly interesting subject for researchers. Although it is simple to define emotions that can be expressed physically such as speech, facial expressions, and gestures, it is more difficult to define psychological emotions that are expressed internally. The most important stimuli in revealing inner emotions are aural and visual stimuli. In this study, EEG signals using both aural and visual stimuli were examined and emotions were evaluated in both binary and multi-class emotion recognitions models. A general emotion recognition model was proposed for non-subject-based classification. Unlike in previous studies, a subject-based testing was performed for the first time in the literature. Capsule Networks, a new neural network model, has been developed for binary and multi-class emotion recognition. In the proposed method, a novel fusion strategy was introduced for binary-class emotion recognition and the model was tested using the GAMEEMO dataset. Binary-class emotion recognition achieved a classification accuracy which was 10% better than the classification performance achieved in other studies in the literature. Based on these findings, we suggest that the proposed method will bring a different perspective to emotion recognition.

Download Full-text

Facial Expression and Electrodermal Activity Analysis for Continuous Pain Intensity Monitoring On The X-ITE Pain Database

10.21203/rs.3.rs-927204/v1 ◽

2021 ◽

Author(s):

Ehsan Othman ◽

Philipp Werner ◽

Frerk Saxen ◽

Ayoub Al-Hamadi ◽

Sascha Gruss ◽

...

Keyword(s):

Random Forest ◽

Facial Expression ◽

Pain Intensity ◽

Short Term Memory ◽

Electrodermal Activity ◽

Decision Fusion ◽

Physiological Data ◽

Weighting Method ◽

Imbalance Problem ◽

Pain Recognition

Abstract Automatic systems enable continuous monitoring of patients' pain intensity as shown in prior studies. Facial expression and physiological data such as electrodermal activity (EDA) are very informative for pain recognition. The features extracted from EDA indicate the stress and anxiety caused by different levels of pain. In this paper, we investigate using the EDA modality and fusing two modalities (frontal RGB video and EDA) for continuous pain intensity recognition with the X-ITE Pain Database. Further, we compare the performance of automated models before and after reducing the imbalance problem in heat and electrical pain datasets that include phasic (short) and tonic (long) stimuli. We use three distinct real-time methods: A Random Forest (RF) baseline methods [Random Forest classifier (RFc) and Random Forest regression (RFr)], Long-Short Term Memory Network (LSTM), and LSTM using sample weighting method (called LSTM-SW). Experimental results (1) report the first results of continuous pain intensity recognition using EDA data on the X-ITE Pain Database, (2) show that LSTM and LSTM-SW outperform guessing and baseline methods (RFc and RFr), (3) confirm that the electrodermal activity (EDA) with most models is the best, (4) show the fusion of the output of two LSTM models using facial expression and EDA data (called Decision Fusion = DF). The DF improves results further with some datasets (e.g. Heat Phasic Dataset (HTD)).

Download Full-text

Zheng Classification with Missing Feature Values Using Local-Validity Approach

Evidence-based Complementary and Alternative Medicine ◽

10.1155/2013/493626 ◽

2013 ◽

Vol 2013 ◽

pp. 1-6

Author(s):

Yan Wang ◽

Lizhuang Ma

Keyword(s):

Liver Cirrhosis ◽

Clinical Practice ◽

Chinese Medicine ◽

Classification Accuracy ◽

Classification Performance ◽

New Approach ◽

Feature Values ◽

Public Datasets ◽

The Given ◽

Missing Feature

Zheng classification is a very important step in the diagnosis of traditional Chinese medicine (TCM). In clinical practice of TCM, feature values are often missing and incomplete cases. The performance of Zheng classification is strictly related to rates of missing feature values. Based on the pattern of the missing feature values, a new approach named local-validity is proposed to classify zheng classification with missing feature values. Firstly, the maximum submatrix for the given dataset is constructed and local-validity method finds subsets of cases for which all of the feature values are available. To reduce the computational scale and improve the classification accuracy, the method clusters subsets with similar patterns to form local-validity subsets. Finally, the proposed method trains a classifier for each local-validity subset and combines the outputs of individual classifiers to diagnose zheng classification. The proposed method is applied to the real liver cirrhosis dataset and three public datasets. Experimental results show that classification performance of local-validity method is superior to the widely used methods under missing feature values.

Download Full-text

Development of Acoustical Feature Based Classifier Using Decision Fusion Technique for Malay Language Disfluencies Classification

Indonesian Journal of Electrical Engineering and Computer Science ◽

10.11591/ijeecs.v8.i1.pp262-267 ◽

2017 ◽

Vol 8 (1) ◽

pp. 262

Author(s):

Raseeda Hamzah ◽

Nursuriati Jamil ◽

Rosniza Roslan

Keyword(s):

Feature Fusion ◽

Decision Fusion ◽

Classification Performance ◽

Zero Crossing ◽

Single Feature ◽

Automated Speech Recognition ◽

Feature Based ◽

Speech Envelope ◽

Speech Disfluency

<p>Speech disfluency such as filled pause (FP) is a hindrance in Automated Speech Recognition as it degrades the accuracy performance. Previous work of FP detection and classification have fused a number of acoustical features as fusion classification is known to improve classification results. This paper presents new decision fusion of two well-established acoustical features that are zero crossing rates (ZCR) and speech envelope (ENV) with eight popular acoustical features for classification of Malay language filled pause (FP) and elongation (ELO). Five hundred ELO and 500 FP are selected from a spontaneous speeches of a parliamentary session and Naïve Bayes classifier is used for the decision fusion classification. The proposed feature fusion produced better classification performance compared to single feature classification with the highest F-measure of 82% for both classes.</p>

Download Full-text

A Novel Hybrid Deep Learning Model for Human Activity Recognition Based on Transitional Activities

Sensors ◽

10.3390/s21248227 ◽

2021 ◽

Vol 21 (24) ◽

pp. 8227

Author(s):

Saad Irfan ◽

Nadeem Anjum ◽

Nayyer Masood ◽

Ahmad S. Khattak ◽

Naeem Ramzan

Keyword(s):

Deep Learning ◽

Activity Recognition ◽

Human Activity ◽

Classification Accuracy ◽

State Of The Art ◽

Decision Fusion ◽

Human Activity Recognition ◽

Transition Activities ◽

Accuracy And Precision ◽

Deep Learning Model

In recent years, a plethora of algorithms have been devised for efficient human activity recognition. Most of these algorithms consider basic human activities and neglect postural transitions because of their subsidiary occurrence and short duration. However, postural transitions assume a significant part in the enforcement of an activity recognition framework and cannot be neglected. This work proposes a hybrid multi-model activity recognition approach that employs basic and transition activities by utilizing multiple deep learning models simultaneously. For final classification, a dynamic decision fusion module is introduced. The experiments are performed on the publicly available datasets. The proposed approach achieved a classification accuracy of 96.11% and 98.38% for the transition and basic activities, respectively. The outcomes show that the proposed method is superior to the state-of-the-art methods in terms of accuracy and precision.

Download Full-text

Completely Automated CNN Architecture Design Based on Blocks

10.26686/wgtn.13158305.v1 ◽

2020 ◽

Author(s):

Y Sun ◽

Bing Xue ◽

Mengjie Zhang ◽

GG Yen

Keyword(s):

Genetic Algorithm ◽

Classification Accuracy ◽

Domain Knowledge ◽

State Of The Art ◽

Classification Performance ◽

Data Sets ◽

Computational Resource ◽

Problem Domain ◽

Benchmark Data ◽

Interested User

© 2019 IEEE. The performance of convolutional neural networks (CNNs) highly relies on their architectures. In order to design a CNN with promising performance, extensive expertise in both CNNs and the investigated problem domain is required, which is not necessarily available to every interested user. To address this problem, we propose to automatically evolve CNN architectures by using a genetic algorithm (GA) based on ResNet and DenseNet blocks. The proposed algorithm is completely automatic in designing CNN architectures. In particular, neither preprocessing before it starts nor postprocessing in terms of CNNs is needed. Furthermore, the proposed algorithm does not require users with domain knowledge on CNNs, the investigated problem, or even GAs. The proposed algorithm is evaluated on the CIFAR10 and CIFAR100 benchmark data sets against 18 state-of-the-art peer competitors. Experimental results show that the proposed algorithm outperforms the state-of-the-art CNNs hand-crafted and the CNNs designed by automatic peer competitors in terms of the classification performance and achieves a competitive classification accuracy against semiautomatic peer competitors. In addition, the proposed algorithm consumes much less computational resource than most peer competitors in finding the best CNN architectures.

Download Full-text