multimodal fusion
Recently Published Documents


TOTAL DOCUMENTS

415
(FIVE YEARS 214)

H-INDEX

27
(FIVE YEARS 7)

Sensors ◽  
2022 ◽  
Vol 22 (2) ◽  
pp. 680
Author(s):  
Sehyeon Kim ◽  
Dae Youp Shin ◽  
Taekyung Kim ◽  
Sangsook Lee ◽  
Jung Keun Hyun ◽  
...  

Motion classification can be performed using biometric signals recorded by electroencephalography (EEG) or electromyography (EMG) with noninvasive surface electrodes for the control of prosthetic arms. However, current single-modal EEG and EMG based motion classification techniques are limited owing to the complexity and noise of EEG signals, and the electrode placement bias, and low-resolution of EMG signals. We herein propose a novel system of two-dimensional (2D) input image feature multimodal fusion based on an EEG/EMG-signal transfer learning (TL) paradigm for detection of hand movements in transforearm amputees. A feature extraction method in the frequency domain of the EEG and EMG signals was adopted to establish a 2D image. The input images were used for training on a model based on the convolutional neural network algorithm and TL, which requires 2D images as input data. For the purpose of data acquisition, five transforearm amputees and nine healthy controls were recruited. Compared with the conventional single-modal EEG signal trained models, the proposed multimodal fusion method significantly improved classification accuracy in both the control and patient groups. When the two signals were combined and used in the pretrained model for EEG TL, the classification accuracy increased by 4.18–4.35% in the control group, and by 2.51–3.00% in the patient group.


2022 ◽  
Vol 12 (1) ◽  
Author(s):  
Vani Rajasekar ◽  
Bratislav Predić ◽  
Muzafer Saracevic ◽  
Mohamed Elhoseny ◽  
Darjan Karabasevic ◽  
...  

AbstractBiometric security is a major emerging concern in the field of data security. In recent years, research initiatives in the field of biometrics have grown at an exponential rate. The multimodal biometric technique with enhanced accuracy and recognition rate for smart cities is still a challenging issue. This paper proposes an enhanced multimodal biometric technique for a smart city that is based on score-level fusion. Specifically, the proposed approach provides a solution to the existing challenges by providing a multimodal fusion technique with an optimized fuzzy genetic algorithm providing enhanced performance. Experiments with different biometric environments reveal significant improvements over existing strategies. The result analysis shows that the proposed approach provides better performance in terms of the false acceptance rate, false rejection rate, equal error rate, precision, recall, and accuracy. The proposed scheme provides a higher accuracy rate of 99.88% and a lower equal error rate of 0.18%. The vital part of this approach is the inclusion of a fuzzy strategy with soft computing techniques known as an optimized fuzzy genetic algorithm.


2022 ◽  
Vol 2022 ◽  
pp. 1-11
Author(s):  
Huijian Deng ◽  
Shijian Cao ◽  
Jingen Tang

In the process of sports, athletes often have aggressive behaviors because of their emotional fluctuations. This violent sports behavior has caused many serious bad effects. In order to reduce and solve this kind of public emergencies, this paper aims to create a swarm intelligence model for predicting people's sports attack behavior, takes the swarm intelligence algorithm as the core technology optimization model, and uses the Internet of Things and other technologies to recognize emotions on physiological signals, predict, and intervene sports attack behavior. The results show the following: (1) After the 50-fold cross-validation method, the results of emotion recognition are good, and the accuracy is high. Compared with other physiological electrical signals, EDA has the worst classification performance. (2) The recognition accuracy of the two methods using multimodal fusion is improved greatly, and the result after comparison is obviously better than that of single mode. (3) Anxiety, anger, surprise, and sadness are the most detected emotions in the model, and the recognition accuracy is higher than 80%. Sports intervention should be carried out in time to calm athletes' emotions. After the experiment, our model runs successfully and performs well, which can be optimized and tested in the next step.


Author(s):  
Mrunal Pathak

Abstract: Smartphones have become a crucial way of storing sensitive information; therefore, the user's privacy needs to be highly secured. This can be accomplished by employing the most reliable and accurate biometric identification system available currently which is, Eye recognition. However, the unimodal eye biometric system is not able to qualify the level of acceptability, speed, and reliability needed. There are other limitations such as constrained authentication in real time applications due to noise in sensed data, spoof attacks, data quality, lack of distinctiveness, restricted amount of freedom, lack of universality and other factors. Therefore, multimodal biometric systems have come into existence in order to increase security as well as to achieve better performance.[1] This paper provides an overview of different multimodal biometric (multibiometric) systems for smartphones being employed till now and also proposes a multimodal biometric system which can possibly overcome the limitations of the current biometric systems. Keywords: Biometrics, Unimodal, Multimodal, Fusion, Multibiometric Systems


2021 ◽  
Author(s):  
Xulin Liu ◽  
Lorraine K Tyler ◽  
James B Rowe ◽  
Kamen A Tsvetanov ◽  

Cognitive ageing is a complex process which requires multimodal approach. Neuroimaging can provide insights into brain morphology, functional organization and vascular dynamics. However, most neuroimaging studies of ageing have focused on each imaging modality separately, limiting the understanding of interrelations between processes identified by different modalities and the interpretation of neural correlates of cognitive decline in ageing. Here, we used linked independent component analysis as a data-driven multimodal approach to jointly analyze magnetic resonance imaging of grey matter density, cerebrovascular, and functional network topographies. Neuroimaging and behavioural data (n = 215) from the Cambridge Centre for Ageing and Neuroscience study were used, containing healthy subjects aged 18 to 88. In the output components, fusion was found between structural and cerebrovascular topographies in multiple components with cognitive-relevance across the lifespan. A component reflecting global atrophy with regional cerebrovascular changes and a component reflecting right frontoparietal network activity were correlated with fluid intelligence over and above age and gender. No meaningful fusion between functional network topography and structural or cerebrovascular signals was observed. We propose that integrating multiple neuroimaging modalities allows to better characterize brain pattern variability and to differentiate brain changes in healthy ageing.


Sensors ◽  
2021 ◽  
Vol 21 (24) ◽  
pp. 8356
Author(s):  
Ha Thi Phuong Thao ◽  
B T Balamurali ◽  
Gemma Roig ◽  
Dorien Herremans

In this paper, we tackle the problem of predicting the affective responses of movie viewers, based on the content of the movies. Current studies on this topic focus on video representation learning and fusion techniques to combine the extracted features for predicting affect. Yet, these typically, while ignoring the correlation between multiple modality inputs, ignore the correlation between temporal inputs (i.e., sequential features). To explore these correlations, a neural network architecture—namely AttendAffectNet (AAN)—uses the self-attention mechanism for predicting the emotions of movie viewers from different input modalities. Particularly, visual, audio, and text features are considered for predicting emotions (and expressed in terms of valence and arousal). We analyze three variants of our proposed AAN: Feature AAN, Temporal AAN, and Mixed AAN. The Feature AAN applies the self-attention mechanism in an innovative way on the features extracted from the different modalities (including video, audio, and movie subtitles) of a whole movie to, thereby, capture the relationships between them. The Temporal AAN takes the time domain of the movies and the sequential dependency of affective responses into account. In the Temporal AAN, self-attention is applied on the concatenated (multimodal) feature vectors representing different subsequent movie segments. In the Mixed AAN, we combine the strong points of the Feature AAN and the Temporal AAN, by applying self-attention first on vectors of features obtained from different modalities in each movie segment and then on the feature representations of all subsequent (temporal) movie segments. We extensively trained and validated our proposed AAN on both the MediaEval 2016 dataset for the Emotional Impact of Movies Task and the extended COGNIMUSE dataset. Our experiments demonstrate that audio features play a more influential role than those extracted from video and movie subtitles when predicting the emotions of movie viewers on these datasets. The models that use all visual, audio, and text features simultaneously as their inputs performed better than those using features extracted from each modality separately. In addition, the Feature AAN outperformed other AAN variants on the above-mentioned datasets, highlighting the importance of taking different features as context to one another when fusing them. The Feature AAN also performed better than the baseline models when predicting the valence dimension.


Complexity ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
Kang Liu ◽  
Xin Gao

The use of multimodal sensors for lane line segmentation has become a growing trend. To achieve robust multimodal fusion, we introduced a new multimodal fusion method and proved its effectiveness in an improved fusion network. Specifically, a multiscale fusion module is proposed to extract effective features from data of different modalities, and a channel attention module is used to adaptively calculate the contribution of the fused feature channels. We verified the effect of multimodal fusion on the KITTI benchmark dataset and A2D2 dataset and proved the effectiveness of the proposed method on the enhanced KITTI dataset. Our method achieves robust lane line segmentation, which is 4.53% higher than the direct fusion on the precision index, and obtains the highest F2 score of 79.72%. We believe that our method introduces an optimization idea of modal data structure level for multimodal fusion.


2021 ◽  
Author(s):  
Lurong Yang ◽  
Zhiquan Feng ◽  
Qingbei Guo ◽  
Jinlan Tian

2021 ◽  
Vol 12 ◽  
Author(s):  
Chenyang Yao ◽  
Na Hu ◽  
Hengyi Cao ◽  
Biqiu Tang ◽  
Wenjing Zhang ◽  
...  

Background: Antipsychotic medications provide limited long-term benefit to ~30% of schizophrenia patients. Multimodal magnetic resonance imaging (MRI) data have been used to investigate brain features between responders and nonresponders to antipsychotic treatment; however, these analytical techniques are unable to weigh the interrelationships between modalities. Here, we used multiset canonical correlation and joint independent component analysis (mCCA + jICA) to fuse MRI data to examine the shared and specific multimodal features between the patients and healthy controls (HCs) and between the responders and non-responders.Method: Resting-state functional and structural MRI data were collected from 55 patients with drug-naïve first-episode schizophrenia (FES) and demographically matched HCs. Based on the decrease in Positive and Negative Syndrome Scale scores from baseline to the 1-year follow-up, FES patients were divided into a responder group (RG) and a non-responder group (NRG). Gray matter volume (GMV), fractional amplitude of low-frequency fluctuation (fALFF), and regional homogeneity (ReHo) maps were used as features in mCCA + jICA.Results: Between FES patients and HCs, there were three modality-specific discriminative independent components (ICs) showing the difference in mixing coefficients (GMV-IC7, GMV-IC8, and fALFF-IC5). The fusion analysis indicated one modality-shared IC (GMV-IC2 and ReHo-IC2) and three modality-specific ICs (GMV-IC1, GMV-IC3, and GMV-IC6) between the RG and NRG. The right postcentral gyrus showed a significant difference in GMV features between FES patients and HCs and modality-shared features (GMV and ReHo) between responders and nonresponders. The modality-shared component findings were highlighted by GMV, mainly in the bilateral temporal gyrus and the right cerebellum associated with ReHo in the right postcentral gyrus.Conclusions: This study suggests that joint anatomical and functional features of the cortices may reflect an early pathophysiological mechanism that is related to a 1-year treatment response.


Sign in / Sign up

Export Citation Format

Share Document