AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss

A number of studies point out that current Visual Question Answering (VQA) models are severely affected by the language prior problem, which refers to blindly making predictions based on the language shortcut. Some efforts have been devoted to overcoming this issue with delicate models. However, there is no research to address it from the view of the answer feature space learning, despite the fact that existing VQA methods all cast VQA as a classification task. Inspired by this, in this work, we attempt to tackle the language prior problem from the viewpoint of the feature space learning. An adapted margin cosine loss is designed to discriminate the frequent and the sparse answer feature space under each question type properly. In this way, the limited patterns within the language modality can be largely reduced to eliminate the language priors. We apply this loss function to several baseline models and evaluate its effectiveness on two VQA-CP benchmarks. Experimental results demonstrate that our proposed adapted margin cosine loss can enhance the baseline models with an absolute performance gain of 15\% on average, strongly verifying the potential of tackling the language prior problem in VQA from the angle of the answer feature space learning.

Download Full-text

Question Type Guided Attention in Visual Question Answering

Computer Vision – ECCV 2018 - Lecture Notes in Computer Science ◽

10.1007/978-3-030-01225-0_10 ◽

2018 ◽

pp. 158-175 ◽

Cited By ~ 5

Author(s):

Yang Shi ◽

Tommaso Furlanello ◽

Sheng Zha ◽

Animashree Anandkumar

Keyword(s):

Question Answering ◽

Question Type ◽

Visual Question Answering

Download Full-text

Multiple Interaction Learning with Question-Type Prior Knowledge for Constraining Answer Search Space in Visual Question Answering

Computer Vision – ECCV 2020 Workshops - Lecture Notes in Computer Science ◽

10.1007/978-3-030-66096-3_34 ◽

2020 ◽

pp. 496-510

Author(s):

Tuong Do ◽

Binh X. Nguyen ◽

Huy Tran ◽

Erman Tjiputra ◽

Quang D. Tran ◽

...

Keyword(s):

Prior Knowledge ◽

Question Answering ◽

Search Space ◽

Question Type ◽

Visual Question Answering ◽

Multiple Interaction

Download Full-text

Co-Attention Network With Question Type for Visual Question Answering

IEEE Access ◽

10.1109/access.2019.2908035 ◽

2019 ◽

Vol 7 ◽

pp. 40771-40781 ◽

Cited By ~ 6

Author(s):

Chao Yang ◽

Mengqi Jiang ◽

Bin Jiang ◽

Weixin Zhou ◽

Keqin Li

Keyword(s):

Question Answering ◽

Question Type ◽

Attention Network ◽

Visual Question Answering

Download Full-text

Multimodal encoders and decoders with gate attention for visual question answering

Computer Science and Information Systems ◽

10.2298/csis201120032l ◽

2021 ◽

pp. 32-32

Author(s):

Haiyan Li ◽

Dezhi Han

Keyword(s):

Computer Vision ◽

Natural Language Processing ◽

Language Processing ◽

Question Answering ◽

Visual Features ◽

Modal Interactions ◽

Crossmodal Attention ◽

The Core ◽

Visual Question Answering ◽

Language Modality

Visual Question Answering (VQA) is a multimodal research related to Computer Vision (CV) and Natural Language Processing (NLP). How to better obtain useful information from images and questions and give an accurate answer to the question is the core of the VQA task. This paper presents a VQA model based on multimodal encoders and decoders with gate attention (MEDGA). Each encoder and decoder block in the MEDGA applies not only self-attention and crossmodal attention but also gate attention, so that the new model can better focus on inter-modal and intra-modal interactions simultaneously within visual and language modality. Besides, MEDGA further filters out noise information irrelevant to the results via gate attention and finally outputs attention results that are closely related to visual features and language features, which makes the answer prediction result more accurate. Experimental evaluations on the VQA 2.0 dataset and the ablation experiments under different conditions prove the effectiveness of MEDGA. In addition, the MEDGA accuracy on the test-std dataset has reached 70.11%, which exceeds many existing methods.

Download Full-text

Densely Connected Attention Flow for Visual Question Answering

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2019/122 ◽

2019 ◽

Cited By ~ 1

Author(s):

Fei Liu ◽

Jing Liu ◽

Zhiwei Fang ◽

Richang Hong ◽

Hanqing Lu

Keyword(s):

Question Answering ◽

State Of The Art ◽

Experimental Results ◽

Effective Interactions ◽

Fine Grained ◽

Visual Question Answering ◽

Complex Image ◽

Hierarchical Levels ◽

Answering Questions ◽

Common Defect

Learning effective interactions between multi-modal features is at the heart of visual question answering (VQA). A common defect of the existing VQA approaches is that they only consider a very limited amount of interactions, which may be not enough to model latent complex image-question relations that are necessary for accurately answering questions. Therefore, in this paper, we propose a novel DCAF (Densely Connected Attention Flow) framework for modeling dense interactions. It densely connects all pairwise layers of the network via Attention Connectors, capturing fine-grained interplay between image and question across all hierarchical levels. The proposed Attention Connector efficiently connects the multi-modal features at any two layers with symmetric co-attention, and produces interaction-aware attention features. Experimental results on three publicly available datasets show that the proposed method achieves state-of-the-art performance.

Download Full-text

Overcoming Language Priors with Self-supervised Learning for Visual Question Answering

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2020/151 ◽

2020 ◽

Author(s):

Xi Zhu ◽

Zhendong Mao ◽

Chunxiao Liu ◽

Peng Zhang ◽

Bin Wang ◽

...

Keyword(s):

Supervised Learning ◽

High Frequency ◽

Question Answering ◽

State Of The Art ◽

Experimental Results ◽

Learning Framework ◽

Visual Question Answering ◽

Art Performance

Most Visual Question Answering (VQA) models suffer from the language prior problem, which is caused by inherent data biases. Specifically, VQA models tend to answer questions (e.g., what color is the banana?) based on the high-frequency answers (e.g., yellow) ignoring image contents. Existing approaches tackle this problem by creating delicate models or introducing additional visual annotations to reduce question dependency and strengthen image dependency. However, they are still subject to the language prior problem since the data biases have not been fundamentally addressed. In this paper, we introduce a self-supervised learning framework to solve this problem. Concretely, we first automatically generate labeled data to balance the biased data, and then propose a self-supervised auxiliary task to utilize the balanced data to assist the VQA model to overcome language priors. Our method can compensate for the data biases by generating balanced data without introducing external annotations. Experimental results show that our method achieves state-of-the-art performance, improving the overall accuracy from 49.50% to 57.59% on the most commonly used benchmark VQA-CP v2. In other words, we can increase the performance of annotation-based methods by 16% without using external annotations. Our code is available on GitHub.

Download Full-text

Re-Attention for Visual Question Answering

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i01.5338 ◽

2020 ◽

Vol 34 (01) ◽

pp. 91-98 ◽

Cited By ~ 1

Author(s):

Wenya Guo ◽

Ying Zhang ◽

Xiaoping Wu ◽

Jufeng Yang ◽

Xiangrui Cai ◽

...

Keyword(s):

Key Words ◽

Question Answering ◽

State Of The Art ◽

Feature Space ◽

The State ◽

Well Performance ◽

Visual Objects ◽

Visual Question Answering ◽

Rich Information ◽

Satisfactory Answer

Visual Question Answering~(VQA) requires a simultaneous understanding of images and questions. Existing methods achieve well performance by focusing on both key objects in images and key words in questions. However, the answer also contains rich information which can help to better describe the image and generate more accurate attention maps. In this paper, to utilize the information in answer, we propose a re-attention framework for the VQA task. We first associate image and question by calculating the similarity of each object-word pairs in the feature space. Then, based on the answer, the learned model re-attends the corresponding visual objects in images and reconstructs the initial attention map to produce consistent results. Benefiting from the re-attention procedure, the question can be better understood, and the satisfactory answer is generated. Extensive experiments on the benchmark dataset demonstrate the proposed method performs favorably against the state-of-the-art approaches.

Download Full-text