Image caption generation with high-level image features

In this paper, we build a multi-style generative model for stylish image captioning which uses multi-modality image features, ResNeXt features, and text features generated by DenseCap. We propose the 3M model, a Multi-UPDOWN caption model that encodes multi-modality features and decodes them into captions. We demonstrate the effectiveness of our model on generating human-like captions by examining its performance on two datasets, the PERSONALITY-CAPTIONS dataset, and the FlickrStyle10K dataset. We compare against a variety of state-of-the-art baselines on various automatic NLP metrics such as BLEU, ROUGE-L, CIDEr, SPICE, etc \footnote{code will be available at https://github.com/cici-ai-club/3M}. A qualitative study has also been done to verify our 3M model can be used for generating different stylized captions.

Spatial Relational Attention Using Fully Convolutional Networks for Image Caption Generation

International Journal of Computational Intelligence and Applications ◽

10.1142/s146902682050011x ◽

2020 ◽

Vol 19 (02) ◽

pp. 2050011

Author(s):

Teng Jiang ◽

Liang Gong ◽

Yupu Yang

Keyword(s):

Spatial Relations ◽

Image Features ◽

Attention Mechanism ◽

Convolutional Network ◽

Convolutional Networks ◽

Fully Convolutional Networks ◽

Benchmark Datasets ◽

Visual Concepts ◽

Attention-based encoder–decoder framework has greatly improved image caption generation tasks. The attention mechanism plays a transitional role by transforming static image features into sequential captions. To generate reasonable captions, it is of great significance to detect spatial characteristics of images. In this paper, we propose a spatial relational attention approach to consider spatial positions and attributes. Image features are firstly weighted by the attention mechanism. Then they are concatenated with contextual features to form a spatial–visual tensor. The tensor is feature extracted by a fully convolutional network to produce visual concepts for the decoder network. The fully convolutional layers maintain spatial topology of images. Experiments conducted on the three benchmark datasets, namely Flickr8k, Flickr30k and MSCOCO, demonstrate the effectiveness of our proposed approach. Captions generated by the spatial relational attention method precisely capture spatial relations of objects.

Image Captioning Using Motion-CNN with Object Detection

Sensors ◽

10.3390/s21041270 ◽

2021 ◽

Vol 21 (4) ◽

pp. 1270

Author(s):

Kiyohiko Iwamura ◽

Jun Younes Louhi Kasahara ◽

Alessandro Moro ◽

Atsushi Yamashita ◽

Hajime Asama

Keyword(s):

Visually Impaired ◽

Image Features ◽

The Internet ◽

Image Captioning ◽

Visually Impaired People ◽

Proposed Model ◽

Impaired People ◽

Motion Features ◽

Automatic image captioning has many important applications, such as the depiction of visual contents for visually impaired people or the indexing of images on the internet. Recently, deep learning-based image captioning models have been researched extensively. For caption generation, they learn the relation between image features and words included in the captions. However, image features might not be relevant for certain words such as verbs. Therefore, our earlier reported method included the use of motion features along with image features for generating captions including verbs. However, all the motion features were used. Since not all motion features contributed positively to the captioning process, unnecessary motion features decreased the captioning accuracy. As described herein, we use experiments with motion features for thorough analysis of the reasons for the decline in accuracy. We propose a novel, end-to-end trainable method for image caption generation that alleviates the decreased accuracy of caption generation. Our proposed model was evaluated using three datasets: MSR-VTT2016-Image, MSCOCO, and several copyright-free images. Results demonstrate that our proposed method improves caption generation performance.

From image to language and back again

Natural Language Engineering ◽

10.1017/s1351324918000086 ◽

2018 ◽

Vol 24 (3) ◽

pp. 325-362

Author(s):

A. BELZ ◽

T.L. BERG ◽

L. YU

Keyword(s):

Language Processing ◽

Question Answering ◽

Multimodal Learning ◽

Image Description ◽

The Past ◽

The Neural Network ◽

High Level ◽

Visual Scene Understanding ◽

Work in computer vision and natural language processing involving images and text has been experiencing explosive growth over the past decade, with a particular boost coming from the neural network revolution. The present volume brings together five research articles from several different corners of the area: multilingual multimodal image description (Franket al.), multimodal machine translation (Madhyasthaet al., Franket al.), image caption generation (Madhyasthaet al., Tantiet al.), visual scene understanding (Silbereret al.), and multimodal learning of high-level attributes (Sorodocet al.). In this article, we touch upon all of these topics as we review work involving images and text under the three main headings of image description (Section 2), visually grounded referring expression generation (REG) and comprehension (Section 3), and visual question answering (VQA) (Section 4).

Image Caption Generation Using Deep Learning

International Journal of Computer Sciences and Engineering ◽

10.26438/ijcse/v6si10.5355 ◽

2018 ◽

Vol 06 (10) ◽

pp. 53-55

Author(s):

Sailee P. Pawaskar ◽

J. A. Laxminarayana

Keyword(s):

Deep Learning ◽

Show, Rethink, And Tell: Image Caption Generation With Hierarchical Topic Cues

2021 IEEE International Conference on Multimedia and Expo (ICME) ◽

10.1109/icme51207.2021.9428353 ◽

2021 ◽

Author(s):

Feng Chen ◽

Songxian Xie ◽

Xinyi Li ◽

Jintao Tang ◽

Kunyuan Pang ◽

...

Keyword(s):

Mining association rules between low-level image features and high-level concepts

10.1117/12.421083 ◽

2001 ◽

Cited By ~ 24

Author(s):

Ishwar K. Sethi ◽

Ioana L. Coman ◽

Daniela Stan

Keyword(s):

Association Rules ◽

Image Features ◽

Low Level ◽

Mining Association Rules ◽

High Level

DCNN-based Ship Classification using Enhanced Edge Information and Inception Module

Journal of Imaging Science and Technology ◽

10.2352/j.imagingsci.technol.2022.66.3.030501 ◽

2021 ◽

Author(s):

Bo Wang ◽

Xiaoting Yu ◽

Chengeng Huang ◽

Qinghong Sheng ◽

Yuanyuan Wang ◽

...

Keyword(s):

Neural Networks ◽

Classification Performance ◽

Image Features ◽

Deep Convolutional Neural Networks ◽

Edge Information ◽

Average Accuracy ◽

Ship Classification ◽

Edge Features ◽

High Level ◽

Better Than

The excellent feature extraction ability of deep convolutional neural networks (DCNNs) has been demonstrated in many image processing tasks, by which image classification can achieve high accuracy with only raw input images. However, the specific image features that influence the classification results are not readily determinable and what lies behind the predictions is unclear. This study proposes a method combining the Sobel and Canny operators and an Inception module for ship classification. The Sobel and Canny operators obtain enhanced edge features from the input images. A convolutional layer is replaced with the Inception module, which can automatically select the proper convolution kernel for ship objects in different image regions. The principle is that the high-level features abstracted by the DCNN, and the features obtained by multi-convolution concatenation of the Inception module must ultimately derive from the edge information of the preprocessing input images. This indicates that the classification results are based on the input edge features, which indirectly interpret the classification results to some extent. Experimental results show that the combination of the edge features and the Inception module improves DCNN ship classification performance. The original model with the raw dataset has an average accuracy of 88.72%, while when using enhanced edge features as input, it achieves the best performance of 90.54% among all models. The model that replaces the fifth convolutional layer with the Inception module has the best performance of 89.50%. It performs close to VGG-16 on the raw dataset and is significantly better than other deep neural networks. The results validate the functionality and feasibility of the idea posited.