A Novel Attention-based Aggregation Function to Combine Vision and Language

Due to the development of computer vision and natural language processing technologies in recent years, there has been a growing interest in multimodal intelligent tasks that require the ability to concurrently understand various forms of input data such as images and text. Vision-and-language navigation (VLN) require the alignment and grounding of multimodal input data to enable real-time perception of the task status on panoramic images and natural language instruction. This study proposes a novel deep neural network model (JMEBS), with joint multimodal embedding and backtracking search for VLN tasks. The proposed JMEBS model uses a transformer-based joint multimodal embedding module. JMEBS uses both multimodal context and temporal context. It also employs backtracking-enabled greedy local search (BGLS), a novel algorithm with a backtracking feature designed to improve the task success rate and optimize the navigation path, based on the local and global scores related to candidate actions. A novel global scoring method is also used for performance improvement by comparing the partial trajectories searched thus far with a plurality of natural language instructions. The performance of the proposed model on various operations was then experimentally demonstrated and compared with other models using the Matterport3D Simulator and room-to-room (R2R) benchmark datasets.

Download Full-text

Mixed aggregation functions for outliers detection

Journal of Intelligent & Fuzzy Systems ◽

10.3233/jifs-200278 ◽

2021 ◽

pp. 1-14

Author(s):

Hengshan Zhang ◽

Chunru Chen ◽

Tianhua Chen ◽

Zhongmin Wang ◽

Yanping Chen

Keyword(s):

Aggregation Function ◽

Outliers Detection ◽

Aggregation Functions ◽

Novel Approach ◽

Consensus Measure

A scenario that often encounters in the event of aggregating options of different experts for the acquisition of a robust overall consensus is the possible existence of extremely large or small values termed as outliers in this paper, which easily lead to counter-intuitive results in decision aggregation. This paper attempts to devise a novel approach to tackle the consensus outliers especially for non-uniform data, filling the gap in the existing literature. In particular, the concentrate region for a set of non-uniform data is first computed with the proposed searching algorithm such that the domain of aggregation function is partitioned into sub-regions. The aggregation will then operate adaptively with respect to the corresponding sub-regions previously partitioned. Finally, the overall aggregation is operated with a proposed novel consensus measure. To demonstrate the working and efficacy of the proposed approach, several illustrative examples are given in comparison to a number of alternative aggregation functions, with the results achieved being more intuitive and of higher consensus.

Download Full-text