scholarly journals Unified Vision-Language Pre-Training for Image Captioning and VQA

2020 ◽  
Vol 34 (07) ◽  
pp. 13041-13049 ◽  
Author(s):  
Luowei Zhou ◽  
Hamid Palangi ◽  
Lei Zhang ◽  
Houdong Hu ◽  
Jason Corso ◽  
...  
Keyword(s):  
Unsupervised Learning ◽  
Question Answering ◽  
State Of The Art ◽  
Learning Objectives ◽  
Image Captioning ◽  
Language Generation ◽  
Visual Question Answering ◽  
Benchmark Datasets

This paper presents a unified Vision-Language Pre-training (VLP) model. The model is unified in that (1) it can be fine-tuned for either vision-language generation (e.g., image captioning) or understanding (e.g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models. The unified VLP model is pre-trained on a large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. The two tasks differ solely in what context the prediction conditions on. This is controlled by utilizing specific self-attention masks for the shared transformer network. To the best of our knowledge, VLP is the first reported model that achieves state-of-the-art results on both vision-language generation and understanding tasks, as disparate as image captioning and visual question answering, across three challenging benchmark datasets: COCO Captions, Flickr30k Captions, and VQA 2.0. The code and the pre-trained models are available at https://github.com/LuoweiZhou/VLP.

scholarly journals Measuring Machine Intelligence Through Visual Question Answering

AI Magazine ◽  
2016 ◽  
Vol 37 (1) ◽  
pp. 63-72 ◽  
Author(s):  
C. Lawrence Zitnick ◽  
Aishwarya Agrawal ◽  
Stanislaw Antol ◽  
Margaret Mitchell ◽  
Dhruv Batra ◽  
...  
Keyword(s):  
Question Answering ◽  
Machine Intelligence ◽  
Image Captioning ◽  
Visual Question Answering ◽  
Language And Vision ◽  
Measuring Machine

As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence. A common approach is to propose tasks for which a human excels, but one which machines find difficult. However, an ideal task should also be easy to evaluate and not be easily gameable. We begin with a case study exploring the recently popular task of image captioning and its limitations as a task for measuring machine intelligence. An alternative and more promising task is Visual Question Answering that tests a machine’s ability to reason about language and vision. We describe a dataset unprecedented in size created for the task that contains over 760,000 human generated questions about images. Using around 10 million human generated answers, machines may be easily evaluated.

scholarly journals DiagnosisQA: A semi-automated pipeline for developing clinician validated diagnosis specific QA datasets.

2021 ◽  
Author(s):  
Shreya Mishra ◽  
Raghav Awasthi ◽  
Frank Papay ◽  
Kamal Maheshawari ◽  
Jacek B Cywinski ◽  
...  
Keyword(s):  
Question Answering ◽  
State Of The Art ◽  
Healthcare Providers ◽  
Unstructured Data ◽  
Significant Progress ◽  
Improve Performance ◽  
Healthcare Data ◽  
Research Areas ◽  
Benchmark Datasets ◽  
Automated Pipeline

Question answering (QA) is one of the oldest research areas of AI and Compu- national Linguistics. QA has seen significant progress with the development of state-of-the-art models and benchmark datasets over the last few years. However, pre-trained QA models perform poorly for clinical QA tasks, presumably due to the complexity of electronic healthcare data. With the digitization of healthcare data and the increasing volume of unstructured data, it is extremely important for healthcare providers to have a mechanism to query the data to find appropriate answers. Since diagnosis is central to any decision-making for the clinicians and patients, we have created a pipeline to develop diagnosis-specific QA datasets and curated a QA database for the Cerebrovascular Accident (CVA). CVA, also commonly known as Stroke, is an important and commonly occurring diagnosis amongst critically ill patients. Our method when compared to clinician validation achieved an accuracy of 0.90(with 90% CI [0.82,0.99]). Using our method, we hope to overcome the key challenges of building and validating a highly accurate QA dataset in a semiautomated manner which can help improve performance of QA models.

scholarly journals Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation

2019 ◽  
Vol 33 ◽  
pp. 6626-6633
Author(s):  
Xiang Kong ◽  
Qizhe Xie ◽  
Zihang Dai ◽  
Eduard Hovy
Keyword(s):  
Machine Translation ◽  
State Of The Art ◽  
The State ◽  
Computational Time ◽  
Memory Consumption ◽  
Image Captioning ◽  
Vocabulary Size ◽  
Language Generation ◽  
Practical Applications ◽  
Coding Schemes

Mixture of Softmaxes (MoS) has been shown to be effective at addressing the expressiveness limitation of Softmax-based models. Despite the known advantage, MoS is practically sealed by its large consumption of memory and computational time due to the need of computing multiple Softmaxes. In this work, we set out to unleash the power of MoS in practical applications by investigating improved word coding schemes, which could effectively reduce the vocabulary size and hence relieve the memory and computation burden. We show both BPE and our proposed Hybrid-LightRNN lead to improved encoding mechanisms that can halve the time and memory consumption of MoS without performance losses. With MoS, we achieve an improvement of 1.5 BLEU scores on IWSLT 2014 German-to-English corpus and an improvement of 0.76 CIDEr score on image captioning. Moreover, on the larger WMT 2014 machine translation dataset, our MoSboosted Transformer yields 29.6 BLEU score for English-toGerman and 42.1 BLEU score for English-to-French, outperforming the single-Softmax Transformer by 0.9 and 0.4 BLEU scores respectively and achieving the state-of-the-art result on WMT 2014 English-to-German task.

scholarly journals Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering

2019 ◽  
Vol 33 ◽  
pp. 8658-8665 ◽  
Author(s):  
Xiangpeng Li ◽  
Jingkuan Song ◽  
Lianli Gao ◽  
Xianglong Liu ◽  
Wenbing Huang ◽  
...  
Keyword(s):  
Question Answering ◽  
State Of The Art ◽  
Computation Time ◽  
Comparable Result ◽  
Video Encoding ◽  
Visual Question Answering ◽  
Proposed Model ◽  
Ablation Study ◽  
The Given ◽  
Video Question Answering

Most of the recent progresses on visual question answering are based on recurrent neural networks (RNNs) with attention. Despite the success, these models are often timeconsuming and having difficulties in modeling long range dependencies due to the sequential nature of RNNs. We propose a new architecture, Positional Self-Attention with Coattention (PSAC), which does not require RNNs for video question answering. Specifically, inspired by the success of self-attention in machine translation task, we propose a Positional Self-Attention to calculate the response at each position by attending to all positions within the same sequence, and then add representations of absolute positions. Therefore, PSAC can exploit the global dependencies of question and temporal information in the video, and make the process of question and video encoding executed in parallel. Furthermore, in addition to attending to the video features relevant to the given questions (i.e., video attention), we utilize the co-attention mechanism by simultaneously modeling “what words to listen to” (question attention). To the best of our knowledge, this is the first work of replacing RNNs with selfattention for the task of visual question answering. Experimental results of four tasks on the benchmark dataset show that our model significantly outperforms the state-of-the-art on three tasks and attains comparable result on the Count task. Our model requires less computation time and achieves better performance compared with the RNNs-based methods. Additional ablation study demonstrates the effect of each component of our proposed model.

scholarly journals Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

2018 ◽  
Author(s):  
Peter Anderson ◽  
Xiaodong He ◽  
Chris Buehler ◽  
Damien Teney ◽  
Mark Johnson ◽  
...  
Keyword(s):  
Question Answering ◽  
Image Captioning ◽  
Top Down ◽  
Bottom Up ◽  
Visual Question Answering

Visual question answering: a state-of-the-art review

2020 ◽  
Vol 53 (8) ◽  
pp. 5705-5745
Author(s):  
Sruthy Manmadhan ◽  
Binsu C. Kovoor
Keyword(s):  
Question Answering ◽  
State Of The Art ◽  
Visual Question Answering

scholarly journals HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning

2019 ◽  
Author(s):  
Shiyang Yan ◽  
Jun Xu ◽  
Yuai Liu ◽  
Lin Xu
Keyword(s):  
State Of The Art ◽  
Recurrent Network ◽  
Image Captioning ◽  
Generative Adversarial Network ◽  
Visual Attributes ◽  
Adversarial Network ◽  
Language Representation ◽  
Benchmark Datasets ◽  
Similarity Preserving ◽  
Domain Transfer

Person re-identification (re-ID) aims to recognize a person-of-interest across different cameras with notable appearance variance. Existing research works focused on the capability and robustness of visual representation. In this paper, instead, we propose a novel hierarchical offshoot recurrent network (HorNet) for improving person re-ID via image captioning. Image captions are semantically richer and more consistent than visual attributes, which could significantly alleviate the variance. We use the similarity preserving generative adversarial network (SPGAN) and an image captioner to fulfill domain transfer and language descriptions generation. Then the proposed HorNet can learn the visual and language representation from both the images and captions jointly, and thus enhance the performance of person re-ID. Extensive experiments are conducted on several benchmark datasets with or without image captions, i.e., CUHK03, Market-1501, and Duke-MTMC, demonstrating the superiority of the proposed method. Our method can generate and extract meaningful image captions while achieving state-of-the-art performance.

Sign in / Sign up

Export Citation Format

Share Document