EMOTION ANALYSIS USING SIGNAL AND IMAGE PROCESSING APPROACH BY IMPLEMENTING DEEP  NEURAL NETWORK

Emotion recognition is important in human communication and to achieve a complete interaction between humans and machines. In medical applications, emotion recognition is used to assist the children with Autism Spectrum Disorder (ASD to improve their socio-emotional communication, helps doctors with diagnosis of diseases such as depression and dementia and also helps the caretakers of older patients to monitor their well-being. This paper discusses the application of feature level fusion of speech and facial expressions of different emotions such as neutral, happy, sad, angry, surprise, fearful and disgust. Also, to explore how best to build the deep learning networks to classify the emotions independently and jointly from these two modalities. VGG-model is utilized to extract features from facial images, and spectral features are extracted from speech signals. Further, feature level fusion technique is adopted to fuse the features extracted from the two modalities. Principal Component Analysis (PCA is implemented to choose the significant features. The proposed method achieved a maximum score of 90% on training set and 82% on validation set. The recognition rate in case of multimodal data improved greatly when compared to unimodal system. The multimodal system gave an improvement of 9% compared to the performance of the system based on speech. Thus, result shows that the proposed Multimodal Emotion Recognition (MER outperform the unimodal emotion recognition system.

Download Full-text

Using bidirectional Binary Particle Swarm Optimization for feature selection in feature-level fusion recognition system

2009 4th IEEE Conference on Industrial Electronics and Applications ◽

10.1109/iciea.2009.5138918 ◽

2009 ◽

Author(s):

Dawei Wang ◽

Wei Ge ◽

Yanjie Wang

Keyword(s):

Feature Selection ◽

Particle Swarm Optimization ◽

Particle Swarm ◽

Recognition System ◽

Binary Particle Swarm Optimization ◽

Swarm Optimization ◽

Feature Level Fusion ◽

Level Fusion

Download Full-text

Speech-based human emotion recognition

10.32920/ryerson.14651964 ◽

2021 ◽

Author(s):

Talieh Seyed Tabtabae

Keyword(s):

Emotion Recognition ◽

Emotional State ◽

Speaker Identification ◽

Recognition System ◽

Research Area ◽

Speech Signals ◽

Spectral Features ◽

Emotional States ◽

Human Communication ◽

Set Up

Automatic Emotion Recognition (AER) is an emerging research area in the Human-Computer Interaction (HCI) field. As Computers are becoming more and more popular every day, the study of interaction between humans (users) and computers is catching more attention. In order to have a more natural and friendly interface between humans and computers, it would be beneficial to give computers the ability to recognize situations the same way a human does. Equipped with an emotion recognition system, computers will be able to recognize their users' emotional state and show the appropriate reaction to that. In today's HCI systems, machines can recognize the speaker and also content of the speech, using speech recognition and speaker identification techniques. If machines are equipped with emotion recognition techniques, they can also know "how it is said" to react more appropriately, and make the interaction more natural. One of the most important human communication channels is the auditory channel which carries speech and vocal intonation. In fact people can perceive each other's emotional state by the way they talk. Therefore in this work the speech signals are analyzed in order to set up an automatic system which recognizes the human emotional state. Six discrete emotional states have been considered and categorized in this research: anger, happiness, fear, surprise, sadness, and disgust. A set of novel spectral features are proposed in this contribution. Two approaches are applied and the results are compared. In the first approach, all the acoustic features are extracted from consequent frames along the speech signals. The statistical values of features are considered to constitute the features vectors. Suport Vector Machine (SVM), which is a relatively new approach in the field of machine learning is used to classify the emotional states. In the second approach, spectral features are extracted from non-overlapping logarithmically-spaced frequency sub-bands. In order to make use of all the extracted information, sequence discriminant SVMs are adopted. The empirical results show that the employed techniques are very promising.

Download Full-text

Audiovisual Emotion Recognition Using Entropy-estimation-based Multimodal Information Fusion

10.32920/ryerson.14668260.v1 ◽

2021 ◽

Author(s):

Zhibing Xie

Keyword(s):

Human Computer Interaction ◽

Emotion Recognition ◽

Information Fusion ◽

Score Level Fusion ◽

Information Theoretic ◽

Multimodal Information Fusion ◽

Multimodal Information ◽

Feature Level Fusion ◽

Level Fusion ◽

Computer Interaction

Understanding human emotional states is indispensable for our daily interaction, and we can enjoy more natural and friendly human computer interaction (HCI) experience by fully utilizing human’s affective states. In the application of emotion recognition, multimodal information fusion is widely used to discover the relationships of multiple information sources and make joint use of a number of channels, such as speech, facial expression, gesture and physiological processes. This thesis proposes a new framework of emotion recognition using information fusion based on the estimation of information entropy. The novel techniques of information theoretic learning are applied to feature level fusion and score level fusion. The most critical issues for feature level fusion are feature transformation and dimensionality reduction. The existing methods depend on the second order statistics, which is only optimal for Gaussian-like distributions. By incorporating information theoretic tools, a new feature level fusion method based on kernel entropy component analysis is proposed. For score level fusion, most previous methods focus on predefined rule based approaches, which are usually heuristic. In this thesis, a connection between information fusion and maximum correntropy criterion is established for effective score level fusion. Feature level fusion and score level fusion methods are then combined to introduce a two-stage fusion platform. The proposed methods are applied to audiovisual emotion recognition, and their effectiveness is evaluated by experiments on two publicly available audiovisual emotion databases. The experimental results demonstrate that the proposed algorithms achieve improved performance in comparison with the existing methods. The work of this thesis offers a promising direction to design more advanced emotion recognition systems based on multimodal information fusion and has great significance to the development of intelligent human computer interaction systems.

Download Full-text

Important Attributes Selection Based on Rough Set for Speech Emotion Recognition

Transdisciplinary Advancements in Cognitive Mechanisms and Human Information Processing ◽

10.4018/978-1-60960-553-7.ch016 ◽

2011 ◽

pp. 262-271

Author(s):

Jian Zhou ◽

Guoyin Wang ◽

Yong Yang

Keyword(s):

Emotion Recognition ◽

Set Theory ◽

Rough Set ◽

Rough Set Theory ◽

Recognition Rate ◽

Feature Selection Method ◽

Recognition System ◽

Attribute Selection ◽

Computer Application ◽

Speech Emotion Recognition

Speech emotion recognition is becoming more and more important in such computer application fields as health care, children education, etc. In order to improve the prediction performance or providing faster and more cost-effective recognition system, an attribute selection is often carried out beforehand to select the important attributes from the input attribute sets. However, it is time-consuming for traditional feature selection method used in speech emotion recognition to determine an optimum or suboptimum feature subset. Rough set theory offers an alternative, formal and methodology that can be employed to reduce the dimensionality of data. The purpose of this study is to investigate the effectiveness of Rough Set Theory in identifying important features in speech emotion recognition system. The experiments on CLDC emotion speech database clearly show this approach can reduce the calculation cost while retaining a suitable high recognition rate.

Download Full-text

Toward Design and Enhancement of Emotion Recognition System Through Speech Signals of Autism Spectrum Disorder Children for Tamil Language Using Multi-Support Vector Machine

Proceedings of International Conference on Computational Intelligence and Data Engineering - Lecture Notes on Data Engineering and Communications Technologies ◽

10.1007/978-981-10-6319-0_13 ◽

2017 ◽

pp. 145-158 ◽

Cited By ~ 1

Author(s):

C. Sunitha Ram ◽

R. Ponnusamy

Keyword(s):

Autism Spectrum Disorder ◽

Support Vector Machine ◽

Emotion Recognition ◽

Recognition System ◽

Autism Spectrum ◽

Spectrum Disorder ◽

Support Vector ◽

Speech Signals

Download Full-text

A Hybrid Approach for Person Identification Using Palmprint and Face Biometrics

International Journal of Pattern Recognition and Artificial Intelligence ◽

10.1142/s0218001415560091 ◽

2015 ◽

Vol 29 (06) ◽

pp. 1556009 ◽

Cited By ~ 7

Author(s):

Mina Farmanbar ◽

Önsen Toygar

Keyword(s):

Hybrid Approach ◽

Principal Component ◽

Local Binary Patterns ◽

Recognition System ◽

Extraction Methods ◽

Score Level Fusion ◽

Linear Discriminant ◽

Hybrid Approaches ◽

Fusion Methods ◽

Level Fusion

This paper proposes hybrid approaches based on both feature level and score level fusion strategies to provide a robust recognition system against the distortions of individual modalities. In order to compare the proposed schemes, a virtual multimodal database is formed from FERET face and PolyU palmprint databases. The proposed hybrid systems concatenate features extracted by local and global feature extraction methods such as Local Binary Patterns, Log Gabor, Principal Component Analysis and Linear Discriminant Analysis. Match score level fusion is performed in order to show the effectiveness and accuracy of the proposed schemes. The experimental results based on these databases reported a significant improvement of the proposed schemes compared with unimodal systems and other multimodal face–palmprint fusion methods.

Download Full-text

A Novel Algorithm for Feature Level Fusion Using SVM Classifier for Multibiometrics-Based Person Identification

Applied Computational Intelligence and Soft Computing ◽

10.1155/2013/515918 ◽

2013 ◽

Vol 2013 ◽

pp. 1-11 ◽

Cited By ~ 16

Author(s):

Ujwalla Gawande ◽

Mukesh Zaveri ◽

Avichal Kapur

Keyword(s):

Learning Algorithm ◽

Recognition Performance ◽

Recognition Rate ◽

Rejection Rate ◽

Support Vector ◽

Svm Classifier ◽

Fusion Algorithm ◽

Biometric Systems ◽

Feature Level Fusion ◽

Level Fusion

Recent times witnessed many advancements in the field of biometric and ultimodal biometric fields. This is typically observed in the area, of security, privacy, and forensics. Even for the best of unimodal biometric systems, it is often not possible to achieve a higher recognition rate. Multimodal biometric systems overcome various limitations of unimodal biometric systems, such as nonuniversality, lower false acceptance, and higher genuine acceptance rates. More reliable recognition performance is achievable as multiple pieces of evidence of the same identity are available. The work presented in this paper is focused on multimodal biometric system using fingerprint and iris. Distinct textual features of the iris and fingerprint are extracted using the Haar wavelet-based technique. A novel feature level fusion algorithm is developed to combine these unimodal features using the Mahalanobis distance technique. A support-vector-machine-based learning algorithm is used to train the system using the feature extracted. The performance of the proposed algorithms is validated and compared with other algorithms using the CASIA iris database and real fingerprint database. From the simulation results, it is evident that our algorithm has higher recognition rate and very less false rejection rate compared to existing approaches.

Download Full-text

Texture Driven Hierarchical Fusion for Multi-Biometric Sys-tem

International Journal of Engineering & Technology ◽

10.14419/ijet.v7i4.24.21766 ◽

2018 ◽

Vol 7 (4.24) ◽

pp. 33 ◽

Cited By ~ 1

Author(s):

Devendra Reddy Rachapalli ◽

Hemantha Kumar Kalluri

Keyword(s):

Feature Selection ◽

Recognition Rate ◽

Texture Feature ◽

Principal Component ◽

Identification Process ◽

Template Security ◽

Storage Complexity ◽

Feature Selection Approach ◽

Fusion Scheme ◽

Level Fusion

This article presents hierarchical fusion models for multi-biometric systems with improved recognition rate. Isolated texture regions are used to encode spatial variations from the composite biometric image which is generated by signal level fusion scheme. In this paper, the prominent issues of the existing multi-biometric system, namely, fusion methodology, storage complexity, reliability and template security are discussed. Here wavelet decomposition driven multi-resolution approach is used to generate the composite images. Texture feature metrics are extracted from multi-level texture regions and principal component analyzes are evaluated to select potentially useful texture values during template creation. Here through consistency and correlation driven hierarchical feature selection both inter-class similarity and intra-class variance problems can be solved. Finally, t-normalized feature level fusion is incorporated as a last stage to create the most reliable template for the identification process. To ensure the security and add robustness to spoof attacks random key driven permutations are used to encrypt the generated multi-biometric templates before storing it in a database. Our experimental results proved that the proposed hierarchical fusion and feature selection approach can embed fine detailed information about the input multi modal biometric images with the least complex identification process.

Download Full-text