c-RNN: A Fine-Grained Language Model for Image Captioning

Image paragraph generation aims to describe an image with a paragraph in natural language. Compared to image captioning with a single sentence, paragraph generation provides more expressive and fine-grained description for storytelling. Existing approaches mainly optimize paragraph generator towards minimizing word-wise cross entropy loss, which neglects linguistic hierarchy of paragraph and results in ``sparse" supervision for generator learning. In this paper, we propose a novel Densely Supervised Hierarchical Policy-Value (DHPV) network for effective paragraph generation. We design new hierarchical supervisions consisting of hierarchical rewards and values at both sentence and word levels. The joint exploration of hierarchical rewards and values provides dense supervision cues for learning effective paragraph generator. We propose a new hierarchical policy-value architecture which exploits compositionality at token-to-token and sentence-to-sentence levels simultaneously and can preserve the semantic and syntactic constituent integrity. Extensive experiments on the Stanford image-paragraph benchmark have demonstrated the effectiveness of the proposed DHPV approach with performance improvements over multiple state-of-the-art methods.

Download Full-text

High-Quality Image Captioning With Fine-Grained and Semantic-Guided Visual Attention

IEEE Transactions on Multimedia ◽

10.1109/tmm.2018.2888822 ◽

2019 ◽

Vol 21 (7) ◽

pp. 1681-1693 ◽

Cited By ~ 8

Author(s):

Zongjian Zhang ◽

Qiang Wu ◽

Yang Wang ◽

Fang Chen

Keyword(s):

Visual Attention ◽

Image Captioning ◽

High Quality ◽

Fine Grained ◽

High Quality Image ◽

Quality Image

Download Full-text

Fine-Grained Image Captioning with Global-Local Discriminative Objective

IEEE Transactions on Multimedia ◽

10.1109/tmm.2020.3011317 ◽

2020 ◽

pp. 1-1 ◽

Cited By ~ 1

Author(s):

Jie Wu ◽

Tianshui Chen ◽

Hefeng Wu ◽

Zhi Yang ◽

Guangchun Luo ◽

...

Keyword(s):

Image Captioning ◽

Fine Grained

Download Full-text

Exploring and Distilling Cross-Modal Information for Image Captioning

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2019/708 ◽

2019 ◽

Cited By ~ 2

Author(s):

Fenglin Liu ◽

Xuancheng Ren ◽

Yuanxin Liu ◽

Kai Lei ◽

Xu Sun

Keyword(s):

Image Understanding ◽

Great Difficulty ◽

Source Information ◽

Image Captioning ◽

Fine Grained ◽

Deep Image ◽

Word Selection ◽

Global And Local ◽

Accuracy Speed ◽

Vision And Language

Recently, attention-based encoder-decoder models have been used extensively in image captioning. Yet there is still great difficulty for the current methods to achieve deep image understanding. In this work, we argue that such understanding requires visual attention to correlated image regions and semantic attention to coherent attributes of interest. To perform effective attention, we explore image captioning from a cross-modal perspective and propose the Global-and-Local Information Exploring-and-Distilling approach that explores and distills the source information in vision and language. It globally provides the aspect vector, a spatial and relational representation of images based on caption contexts, through the extraction of salient region groupings and attribute collocations, and locally extracts the fine-grained regions and attributes in reference to the aspect vector for word selection. Our fully-attentive model achieves a CIDEr score of 129.3 in offline COCO evaluation with remarkable efficiency in terms of accuracy, speed, and parameter budget.

Download Full-text

MemCap: Memorizing Style Knowledge for Image Captioning

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i07.6998 ◽

2020 ◽

Vol 34 (07) ◽

pp. 12984-12992 ◽

Cited By ~ 2

Author(s):

Wentian Zhao ◽

Xinxiao Wu ◽

Xiaoxun Zhang

Keyword(s):

Language Model ◽

Attention Mechanism ◽

Visual Content ◽

Image Captioning ◽

Memory Module ◽

Training Corpus ◽

Memory Mechanism ◽

Linguistic Style ◽

Related Part ◽

With Memory

Generating stylized captions for images is a challenging task since it requires not only describing the content of the image accurately but also expressing the desired linguistic style appropriately. In this paper, we propose MemCap, a novel stylized image captioning method that explicitly encodes the knowledge about linguistic styles with memory mechanism. Rather than relying heavily on a language model to capture style factors in existing methods, our method resorts to memorizing stylized elements learned from training corpus. Particularly, we design a memory module that comprises a set of embedding vectors for encoding style-related phrases in training corpus. To acquire the style-related phrases, we develop a sentence decomposing algorithm that splits a stylized sentence into a style-related part that reflects the linguistic style and a content-related part that contains the visual content. When generating captions, our MemCap first extracts content-relevant style knowledge from the memory module via an attention mechanism and then incorporates the extracted knowledge into a language model. Extensive experiments on two stylized image captioning datasets (SentiCap and FlickrStyle10K) demonstrate the effectiveness of our method.

Download Full-text

Half-Context Language Models

Computational Linguistics ◽

10.1162/coli_a_00078 ◽

2011 ◽

Vol 37 (4) ◽

pp. 843-865

Author(s):

Hinrich Schütze ◽

Michael Walsh

Keyword(s):

Clustering Algorithm ◽

Language Model ◽

Model Performance ◽

Language Models ◽

Fine Grained ◽

Specific Analysis ◽

Distributional Information ◽

N Gram ◽

Context Specific ◽

Context Models

This article investigates the effects of different degrees of contextual granularity on language model performance. It presents a new language model that combines clustering and half-contextualization, a novel representation of contexts. Half-contextualization is based on the half-context hypothesis that states that the distributional characteristics of a word or bigram are best represented by treating its context distribution to the left and right separately and that only directionally relevant distributional information should be used. Clustering is achieved using a new clustering algorithm for class-based language models that compares favorably to the exchange algorithm. When interpolated with a Kneser-Ney model, half-context models are shown to have better perplexity than commonly used interpolated n-gram models and traditional class-based approaches. A novel, fine-grained, context-specific analysis highlights those contexts in which the model performs well and those which are better treated by existing non-class-based models.

Download Full-text

Cascade Attention Fusion for Fine-Grained Image Captioning Based on Multi-Layer LSTM

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) ◽

10.1109/icassp39728.2021.9413691 ◽

2021 ◽

Author(s):

Shuang Wang ◽

Yun Meng ◽

Yu Gu ◽

Lei Zhang ◽

Xiutiao Ye ◽

...

Keyword(s):

Image Captioning ◽

Fine Grained

Download Full-text

RDMMFET: Representation of Dense Multimodality Fusion Encoder Based on Transformer

Mobile Information Systems ◽

10.1155/2021/2662064 ◽

2021 ◽

Vol 2021 ◽

pp. 1-9

Author(s):

Xu Zhang ◽

DeZhi Han ◽

Chin-Chen Chang

Keyword(s):

Question Answering ◽

Language Model ◽

Visual Images ◽

Text And Image ◽

Data Set ◽

Fine Grained ◽

Natural Language Question ◽

The Relationship ◽

Language Question ◽

Better Than

Visual question answering (VQA) is the natural language question-answering of visual images. The model of VQA needs to make corresponding answers according to specific questions based on understanding images, the most important of which is to understand the relationship between images and language. Therefore, this paper proposes a new model, Representation of Dense Multimodality Fusion Encoder Based on Transformer, for short, RDMMFET, which can learn the related knowledge between vision and language. The RDMMFET model consists of three parts: dense language encoder, image encoder, and multimodality fusion encoder. In addition, we designed three types of pretraining tasks: masked language model, masked image model, and multimodality fusion task. These pretraining tasks can help to understand the fine-grained alignment between text and image regions. Simulation results on the VQA v2.0 data set show that the RDMMFET model can work better than the previous model. Finally, we conducted detailed ablation studies on the RDMMFET model and provided the results of attention visualization, which proves that the RDMMFET model can significantly improve the effect of VQA.

Download Full-text

One for “All”: a unified model for fine-grained sentiment analysis under three tasks

PeerJ Computer Science ◽

10.7717/peerj-cs.816 ◽

2021 ◽

Vol 7 ◽

pp. e816

Author(s):

Heng-yang Lu ◽

Jun Yang ◽

Cong Hu ◽

Wei Fang

Keyword(s):

Sentiment Analysis ◽

Data Augmentation ◽

Language Model ◽

Unified Model ◽

Training Data ◽

Low Resource ◽

Fine Grained ◽

Questions And Answers ◽

Resource Conditions ◽

Media Data

Background Fine-grained sentiment analysis is used to interpret consumers’ sentiments, from their written comments, towards specific entities on specific aspects. Previous researchers have introduced three main tasks in this field (ABSA, TABSA, MEABSA), covering all kinds of social media data (e.g., review specific, questions and answers, and community-based). In this paper, we identify and address two common challenges encountered in these three tasks, including the low-resource problem and the sentiment polarity bias. Methods We propose a unified model called PEA by integrating data augmentation methodology with the pre-trained language model, which is suitable for all the ABSA, TABSA and MEABSA tasks. Two data augmentation methods, which are entity replacement and dual noise injection, are introduced to solve both challenges at the same time. An ensemble method is also introduced to incorporate the results of the basic RNN-based and BERT-based models. Results PEA shows significant improvements on all three fine-grained sentiment analysis tasks when compared with state-of-the-art models. It also achieves comparable results with what the baseline models obtain while using only 20% of their training data, which demonstrates its extraordinary performance under extreme low-resource conditions.

Download Full-text