scholarly journals Trimodal Attention Module for Multimodal Sentiment Analysis (Student Abstract)

2020 ◽  
Vol 34 (10) ◽  
pp. 13803-13804
Author(s):  
Anirudh Bindiganavale Harish ◽  
Fatiha Sadat
Keyword(s):  
Sentiment Analysis ◽  
State Of The Art ◽  
Contextual Information ◽  
Multimodal Fusion ◽  
Decision Level ◽  
Training Approach ◽  
Decision Level Fusion ◽  
Multimodal Sentiment Analysis ◽  
The Individual ◽  
Level Fusion

In our research, we propose a new multimodal fusion architecture for the task of sentiment analysis. The 3 modalities used in this paper are text, audio and video. Most of the current methods deal with either a feature level or a decision level fusion. In contrast, we propose an attention-based deep neural network and a training approach to facilitate both feature and decision level fusion. Our network effectively leverages information across all three modalities using a 2 stage fusion process. We test our network on the individual utterance based contextual information extracted from the CMU-MOSI Dataset. A comparison is drawn between the state-of-the-art and our network.

scholarly journals Two-Level Multimodal Fusion for Sentiment Analysis in Public Security

2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Jianguo Sun ◽  
Hanqi Yin ◽  
Ye Tian ◽  
Junpeng Wu ◽  
Linshan Shen ◽  
...  
Keyword(s):  
Sentiment Analysis ◽  
Multimodal Fusion ◽  
Public Security ◽  
Multimodal Data ◽  
Decision Level ◽  
Analysis Task ◽  
Fusion Methods ◽  
Decision Level Fusion ◽  
Level Fusion ◽  
Data Level

Large amounts of data are widely stored in cyberspace. Not only can they bring much convenience to people’s lives and work, but they can also assist the work in the information security field, such as microexpression recognition and sentiment analysis in the criminal investigation. Thus, it is of great significance to recognize and analyze the sentiment information, which is usually described by different modalities. Due to the correlation among different modalities data, multimodal can provide more comprehensive and robust information than unimodal in data analysis tasks. The complementary information from different modalities can be obtained by multimodal fusion methods. These approaches can process multimodal data through fusion algorithms and ensure the accuracy of the information used for subsequent classification or prediction tasks. In this study, a two-level multimodal fusion (TlMF) method with both data-level and decision-level fusion is proposed to achieve the sentiment analysis task. In the data-level fusion stage, a tensor fusion network is utilized to obtain the text-audio and text-video embeddings by fusing the text with audio and video features, respectively. During the decision-level fusion stage, the soft fusion method is adopted to fuse the classification or prediction results of the upstream classifiers, so that the final classification or prediction results can be as accurate as possible. The proposed method is tested on the CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets, and the empirical results and ablation studies confirm the effectiveness of TlMF in capturing useful information from all the test modalities.

scholarly journals One for All: Neural Joint Modeling of Entities and Events

2019 ◽  
Vol 33 ◽  
pp. 6851-6858 ◽  
Author(s):  
Trung Minh Nguyen ◽  
Thien Huu Nguyen
Keyword(s):  
Deep Learning ◽  
Recent Work ◽  
State Of The Art ◽  
Contextual Information ◽  
Joint Modeling ◽  
Event Extraction ◽  
Event Trigger ◽  
The Individual ◽  
Novel Model ◽  
Argument Roles

The previous work for event extraction has mainly focused on the predictions for event triggers and argument roles, treating entity mentions as being provided by human annotators. This is unrealistic as entity mentions are usually predicted by some existing toolkits whose errors might be propagated to the event trigger and argument role recognition. Few of the recent work has addressed this problem by jointly predicting entity mentions, event triggers and arguments. However, such work is limited to using discrete engineering features to represent contextual information for the individual tasks and their interactions. In this work, we propose a novel model to jointly perform predictions for entity mentions, event triggers and arguments based on the shared hidden representations from deep learning. The experiments demonstrate the benefits of the proposed method, leading to the state-of-the-art performance for event extraction.

scholarly journals Feature and Decision Level Fusion in Children Multimodal Biometrics

2020 ◽  
Vol 8 (5) ◽  
pp. 2522-2527
Keyword(s):  
Nearest Neighbor ◽  
Principal Component ◽  
Identification Accuracy ◽  
Support Vector ◽  
Biometric System ◽  
K Nearest Neighbor ◽  
Decision Level ◽  
Multimodal Biometric System ◽  
Decision Level Fusion ◽  
Level Fusion

In this paper, we design method for recognition of fingerprint and IRIS using feature level fusion and decision level fusion in Children multimodal biometric system. Initially, Histogram of Gradients (HOG), Gabour and Maximum filter response are extracted from both the domains of fingerprint and IRIS and considered for identification accuracy. The combination of feature vector of all the possible features is recommended by biometrics traits of fusion. For fusion vector the Principal Component Analysis (PCA) is used to select features. The reduced features are fed into fusion classifier of K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Navie Bayes(NB). For children multimodal biometric system the suitable combination of features and fusion classifiers is identified. The experimentation conducted on children’s fingerprint and IRIS database and results reveal that fusion combination outperforms individual. In addition the proposed model advances the unimodal biometrics system.

Sign in / Sign up

Export Citation Format

Share Document