Image captioning improved visual question answering

2021 ◽  
Author(s):  
Himanshu Sharma ◽  
Anand Singh Jalal
Keyword(s):  
Question Answering ◽  
Image Captioning ◽  
Visual Question Answering

scholarly journals Measuring Machine Intelligence Through Visual Question Answering

AI Magazine ◽  
2016 ◽  
Vol 37 (1) ◽  
pp. 63-72 ◽  
Author(s):  
C. Lawrence Zitnick ◽  
Aishwarya Agrawal ◽  
Stanislaw Antol ◽  
Margaret Mitchell ◽  
Dhruv Batra ◽  
...  
Keyword(s):  
Question Answering ◽  
Machine Intelligence ◽  
Image Captioning ◽  
Visual Question Answering ◽  
Language And Vision ◽  
Measuring Machine

As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence. A common approach is to propose tasks for which a human excels, but one which machines find difficult. However, an ideal task should also be easy to evaluate and not be easily gameable. We begin with a case study exploring the recently popular task of image captioning and its limitations as a task for measuring machine intelligence. An alternative and more promising task is Visual Question Answering that tests a machine’s ability to reason about language and vision. We describe a dataset unprecedented in size created for the task that contains over 760,000 human generated questions about images. Using around 10 million human generated answers, machines may be easily evaluated.

scholarly journals Unified Vision-Language Pre-Training for Image Captioning and VQA

2020 ◽  
Vol 34 (07) ◽  
pp. 13041-13049 ◽  
Author(s):  
Luowei Zhou ◽  
Hamid Palangi ◽  
Lei Zhang ◽  
Houdong Hu ◽  
Jason Corso ◽  
...  
Keyword(s):  
Unsupervised Learning ◽  
Question Answering ◽  
State Of The Art ◽  
Learning Objectives ◽  
Image Captioning ◽  
Language Generation ◽  
Visual Question Answering ◽  
Benchmark Datasets

This paper presents a unified Vision-Language Pre-training (VLP) model. The model is unified in that (1) it can be fine-tuned for either vision-language generation (e.g., image captioning) or understanding (e.g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models. The unified VLP model is pre-trained on a large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. The two tasks differ solely in what context the prediction conditions on. This is controlled by utilizing specific self-attention masks for the shared transformer network. To the best of our knowledge, VLP is the first reported model that achieves state-of-the-art results on both vision-language generation and understanding tasks, as disparate as image captioning and visual question answering, across three challenging benchmark datasets: COCO Captions, Flickr30k Captions, and VQA 2.0. The code and the pre-trained models are available at https://github.com/LuoweiZhou/VLP.

scholarly journals Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

2018 ◽  
Author(s):  
Peter Anderson ◽  
Xiaodong He ◽  
Chris Buehler ◽  
Damien Teney ◽  
Mark Johnson ◽  
...  
Keyword(s):  
Question Answering ◽  
Image Captioning ◽  
Top Down ◽  
Bottom Up ◽  
Visual Question Answering

scholarly journals Dynamic Capsule Attention for Visual Question Answering

2019 ◽  
Vol 33 ◽  
pp. 9324-9331 ◽  
Author(s):  
Yiyi Zhou ◽  
Rongrong Ji ◽  
Jinsong Su ◽  
Xiaoshuai Sun ◽  
Weiqiu Chen
Keyword(s):  
Question Answering ◽  
Visual Features ◽  
Coupling Coefficients ◽  
Image Captioning ◽  
Attention Model ◽  
Visual Question Answering ◽  
Training Examples ◽  
Attention Modeling ◽  
Projection Matrices ◽  
Simple Network

In visual question answering (VQA), recent advances have well advocated the use of attention mechanism to precisely link the question to the potential answer areas. As the difficulty of the question increases, more VQA models adopt multiple attention layers to capture the deeper visual-linguistic correlation. But a negative consequence is the explosion of parameters, which makes the model vulnerable to over-fitting, especially when limited training examples are given. In this paper, we propose an extremely compact alternative to this static multi-layer architecture towards accurate yet efficient attention modeling, termed as Dynamic Capsule Attention (CapsAtt). Inspired by the recent work of Capsule Network, CapsAtt treats visual features as capsules and obtains the attention output via dynamic routing, which updates the attention weights by calculating coupling coefficients between the underlying and output capsules. Meanwhile, CapsAtt also discards redundant projection matrices to make the model much more compact. We quantify CapsAtt on three benchmark VQA datasets, i.e., COCO-QA, VQA1.0 and VQA2.0. Compared to the traditional multi-layer attention model, CapsAtt achieves significant improvements of up to 4.1%, 5.2% and 2.2% on three datasets, respectively. Moreover, with much fewer parameters, our approach also yields competitive results compared to the latest VQA models. To further verify the generalization ability of CapsAtt, we also deploy it on another challenging multi-modal task of image captioning, where state-of-the-art performance is achieved with a simple network structure.

scholarly journals Image Captioning and Visual Question Answering Based on Attributes and External Knowledge

2018 ◽  
Vol 40 (6) ◽  
pp. 1367-1381 ◽  
Author(s):  
Qi Wu ◽  
Chunhua Shen ◽  
Peng Wang ◽  
Anthony Dick ◽  
Anton van den Hengel
Keyword(s):  
Question Answering ◽  
Image Captioning ◽  
External Knowledge ◽  
Visual Question Answering

Visual Question-Answering System Using Integrated Models of Image Captioning and BERT

2021 ◽  
pp. 169-190
Author(s):  
Lavika Goel ◽  
Mohit Dhawan ◽  
Rachit Rathore ◽  
Satyansh Rai ◽  
Aaryan Kapoor ◽  
...  
Keyword(s):  
Question Answering ◽  
Integrated Models ◽  
Image Captioning ◽  
Question Answering System ◽  
Visual Question Answering
Sign in / Sign up

Export Citation Format

Share Document