Multiple Context Learning Networks for Visual Question Answering

Mapping Intimacies ◽

10.21203/rs.3.rs-955099/v1 ◽

2021 ◽

Author(s):

Pufen Zhang ◽

Hong Lan

Keyword(s):

Question Answering ◽

Learning Strategy ◽

Learning Networks ◽

Learning Modules ◽

Visual Question Answering ◽

Context Learning ◽

Multiple Context ◽

Proposed Model ◽

Previous State ◽

High Level

Abstract In recently years, some visual question answering (VQA) methods that emphasize the simultaneous understanding of both the context of image and question have been proposed. Despite the effectiveness of these methods, they fail to explore a more comprehensive and generalized context learning tactics. To address this issue, we propose a novel Multiple Context Learning Networks (MCLN) to model the multiple contexts for VQA. Three kinds of contexts are investigated, namely visual context, textual context and a special visual-textual context that ignored by previous methods. Moreover, three corresponding context learning modules are proposed. These modules endow image and text representations with context-aware information based on a uniform context learning strategy. And they work together to form a multiple context learning layer (MCL). Such MCL can be stacked in depth and which describe high-level context information by associating intra-modal contexts with inter-modal context. On the VQA v2.0 datasets, the proposed model achieves 71.05% and 71.48% on test-dev set and test-std set respectively, and gains better performance than the previous state-of-the-art methods. In addition, extensive ablation studies have been carried out to examine the effectiveness of the proposed method.

Download Full-text

Multi-Modality Global Fusion Attention Network for Visual Question Answering

Electronics ◽

10.3390/electronics9111882 ◽

2020 ◽

Vol 9 (11) ◽

pp. 1882

Author(s):

Cheng Yang ◽

Weijia Wu ◽

Yuxing Wang ◽

Hong Zhou

Keyword(s):

Correct Answer ◽

Question Answering ◽

Human Beings ◽

Single Model ◽

Attention Network ◽

Attention Model ◽

Global Perspectives ◽

Visual Question Answering ◽

Previous State ◽

High Level

Visual question answering (VQA) requires a high-level understanding of both questions and images, along with visual reasoning to predict the correct answer. Therefore, it is important to design an effective attention model to associate key regions in an image with key words in a question. Up to now, most attention-based approaches only model the relationships between individual regions in an image and words in a question. It is not enough to predict the correct answer for VQA, as human beings always think in terms of global information, not only local information. In this paper, we propose a novel multi-modality global fusion attention network (MGFAN) consisting of stacked global fusion attention (GFA) blocks, which can capture information from global perspectives. Our proposed method computes co-attention and self-attention at the same time, rather than computing them individually. We validate our proposed method on the two most commonly used benchmarks, the VQA-v2 datasets. Experimental results show that the proposed method outperforms the previous state-of-the-art. Our best single model achieves 70.67% accuracy on the test-dev set of VQA-v2.

Download Full-text

Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33018658 ◽

2019 ◽

Vol 33 ◽

pp. 8658-8665 ◽

Cited By ~ 10

Author(s):

Xiangpeng Li ◽

Jingkuan Song ◽

Lianli Gao ◽

Xianglong Liu ◽

Wenbing Huang ◽

...

Keyword(s):

Question Answering ◽

State Of The Art ◽

Computation Time ◽

Comparable Result ◽

Video Encoding ◽

Visual Question Answering ◽

Proposed Model ◽

Ablation Study ◽

The Given ◽

Video Question Answering

Most of the recent progresses on visual question answering are based on recurrent neural networks (RNNs) with attention. Despite the success, these models are often timeconsuming and having difficulties in modeling long range dependencies due to the sequential nature of RNNs. We propose a new architecture, Positional Self-Attention with Coattention (PSAC), which does not require RNNs for video question answering. Specifically, inspired by the success of self-attention in machine translation task, we propose a Positional Self-Attention to calculate the response at each position by attending to all positions within the same sequence, and then add representations of absolute positions. Therefore, PSAC can exploit the global dependencies of question and temporal information in the video, and make the process of question and video encoding executed in parallel. Furthermore, in addition to attending to the video features relevant to the given questions (i.e., video attention), we utilize the co-attention mechanism by simultaneously modeling “what words to listen to” (question attention). To the best of our knowledge, this is the first work of replacing RNNs with selfattention for the task of visual question answering. Experimental results of four tasks on the benchmark dataset show that our model significantly outperforms the state-of-the-art on three tasks and attains comparable result on the Count task. Our model requires less computation time and achieves better performance compared with the RNNs-based methods. Additional ablation study demonstrates the effect of each component of our proposed model.

Download Full-text

Visual Question Answering Through Adversarial Learning of Multi-modal Representation

10.36227/techrxiv.12731948 ◽

2020 ◽

Author(s):

Iqbal Chowdhury ◽

Kien Nguyen Thanh ◽

Clinton fookes ◽

Sridha Sridharan

Keyword(s):

Natural Language ◽

Question Answering ◽

Feature Representation ◽

Fusion Method ◽

Adversarial Learning ◽

Visual Question Answering ◽

Proposed Model ◽

Fusion Methods ◽

Adversarial Training ◽

Multimodal Representation

Solving the Visual Question Answering (VQA) task is a step towards achieving human-like reasoning capability of the machines. This paper proposes an approach to learn multimodal feature representation with adversarial training. The purpose of the adversarial training allows the model to learn from standard fusion methods in an unsupervised manner. The discriminator model is equipped with a siamese combinatin of two standard fusion method namely multimodal compact bilinear pooling and multimodal tucker fusion. Output multimodal feature representation from generator is a resultant of graph convolutional operation. The resultant multimodal representation of the adversarial training allows the proposed model to infer the correct answers from open-ended natural language questions from the VQA 2.0 dataset. An overall accuracy of 69.86\% demonstrates the accuracy of the proposed model.

Download Full-text

Visual Question Answering Through Adversarial Learning of Multi-modal Representation

10.36227/techrxiv.12731948.v1 ◽

2020 ◽

Author(s):

Iqbal Chowdhury ◽

Kien Nguyen Thanh ◽

Clinton fookes ◽

Sridha Sridharan

Keyword(s):

Natural Language ◽

Question Answering ◽

Feature Representation ◽

Fusion Method ◽

Adversarial Learning ◽

Visual Question Answering ◽

Proposed Model ◽

Fusion Methods ◽

Adversarial Training ◽

Multimodal Representation

Download Full-text

Multi-View Attention Network for Visual Dialog

Applied Sciences ◽

10.3390/app11073009 ◽

2021 ◽

Vol 11 (7) ◽

pp. 3009

Author(s):

Sungjin Park ◽

Taesun Whang ◽

Yeochan Yoon ◽

Heuiseok Lim

Keyword(s):

State Of The Art ◽

Relevant Information ◽

Experimental Results ◽

Multiple Views ◽

Single Model ◽

Attention Network ◽

Proposed Model ◽

Previous State ◽

Multimodal Representations ◽

High Level

Visual dialog is a challenging vision-language task in which a series of questions visually grounded by a given image are answered. To resolve the visual dialog task, a high-level understanding of various multimodal inputs (e.g., question, dialog history, and image) is required. Specifically, it is necessary for an agent to (1) determine the semantic intent of question and (2) align question-relevant textual and visual contents among heterogeneous modality inputs. In this paper, we propose Multi-View Attention Network (MVAN), which leverages multiple views about heterogeneous inputs based on attention mechanisms. MVAN effectively captures the question-relevant information from the dialog history with two complementary modules (i.e., Topic Aggregation and Context Matching), and builds multimodal representations through sequential alignment processes (i.e., Modality Alignment). Experimental results on VisDial v1.0 dataset show the effectiveness of our proposed model, which outperforms previous state-of-the-art methods under both single model and ensemble settings.

Download Full-text

Hybrid Deep Neural Model for Duplicate Question Detection in Transliterated Bi-lingual Data

Recent Patents on Computer Science ◽

10.2174/2213275912666190710152709 ◽

2019 ◽

Vol 12 ◽

Author(s):

Seema Rani ◽

Avadhesh Kumar ◽

Naresh Kumar

Keyword(s):

Question Answering ◽

Neural Model ◽

Manhattan Distance ◽

Semantic Matching ◽

English Only ◽

Detection Model ◽

Proposed Model ◽

Reported Study ◽

Social Media Platforms ◽

Efficient Information

Background: Duplicate content often corrupts the filtering mechanism in online question answering. Moreover, as users are usually more comfortable conversing in their native language questions, transliteration adds to the challenges in detecting duplicate questions. This compromises with the response time and increases the answer overload. Thus, it has now become crucial to build clever, intelligent and semantic filters which semantically match linguistically disparate questions. Objective: Most of the research on duplicate question detection has been done on mono-lingual, majorly English Q&A platforms. The aim is to build a model which extends the cognitive capabilities of machines to interpret, comprehend and learn features for semantic matching in transliterated bi-lingual Hinglish (Hindi + English) data acquired from different Q&A platforms. Method: In the proposed DQDHinglish (Duplicate Question Detection) Model, firstly language transformation (transliteration & translation) is done to convert the bi-lingual transliterated question into a mono-lingual English only text. Next a hybrid of Siamese neural network containing two identical Long-term-Short-memory (LSTM) models and Multi-layer perceptron network is proposed to detect semantically similar question pairs. Manhattan distance function is used as the similarity measure. Result: A dataset was prepared by scrapping 100 question pairs from various social media platforms, such as Quora and TripAdvisor. The performance of the proposed model on the basis of accuracy and F-score. The proposed DQDHinglish achieves a validation accuracy of 82.40%. Conclusion: A deep neural model was introduced to find semantic match between English question and a Hinglish (Hindi + English) question such that similar intent questions can be combined to enable fast and efficient information processing and delivery. A dataset was created and the proposed model was evaluated on the basis of performance accuracy. To the best of our knowledge, this work is the first reported study on transliterated Hinglish semantic question matching.

Download Full-text