scholarly journals Boosted Transformer for Image Captioning

2019 ◽  
Vol 9 (16) ◽  
pp. 3260 ◽  
Author(s):  
Jiangyun Li ◽  
Peng Yao ◽  
Longteng Guo ◽  
Weicun Zhang
Keyword(s):  
Visual Information ◽  
State Of The Art ◽  
The Self ◽  
Visual Features ◽  
Image Captioning ◽  
Decoder Architecture ◽  
Semantic Concepts ◽  
Transformer Model ◽  
Internal Relationships ◽  
Auxiliary Module

Image captioning attempts to generate a description given an image, usually taking Convolutional Neural Network as the encoder to extract the visual features and a sequence model, among which the self-attention mechanism has achieved advanced progress recently, as the decoder to generate descriptions. However, this predominant encoder-decoder architecture has some problems to be solved. On the encoder side, without the semantic concepts, the extracted visual features do not make full use of the image information. On the decoder side, the sequence self-attention only relies on word representations, lacking the guidance of visual information and easily influenced by the language prior. In this paper, we propose a novel boosted transformer model with two attention modules for the above-mentioned problems, i.e., “Concept-Guided Attention” (CGA) and “Vision-Guided Attention” (VGA). Our model utilizes CGA in the encoder, to obtain the boosted visual features by integrating the instance-level concepts into the visual features. In the decoder, we stack VGA, which uses the visual information as a bridge to model internal relationships among the sequences and can be an auxiliary module of sequence self-attention. Quantitative and qualitative results on the Microsoft COCO dataset demonstrate the better performance of our model than the state-of-the-art approaches.

scholarly journals Visual Weather Property Prediction by Multi-Task Learning and Two-Dimensional RNNs

Atmosphere ◽  
2021 ◽  
Vol 12 (5) ◽  
pp. 584
Author(s):  
Wei-Ta Chu ◽  
Yu-Hsuan Liang ◽  
Kai-Chia Ho
Keyword(s):  
Neural Networks ◽  
Visual Information ◽  
Temporal Evolution ◽  
State Of The Art ◽  
Image Data ◽  
Visual Features ◽  
Two Dimensional ◽  
Estimation Model ◽  
Property Estimation ◽  
Task Learning

We attempted to employ convolutional neural networks to extract visual features and developed recurrent neural networks for weather property estimation using only image data. Four common weather properties are estimated, i.e., temperature, humidity, visibility, and wind speed. Based on the success of previous works on temperature prediction, we extended them in terms of two aspects. First, by considering the effectiveness of deep multi-task learning, we jointly estimated four weather properties on the basis of the same visual information. Second, we propose that weather property estimations considering temporal evolution can be conducted from two perspectives, i.e., day-wise or hour-wise. A two-dimensional recurrent neural network is thus proposed to unify the two perspectives. In the evaluation, we show that better prediction accuracy can be obtained compared to the state-of-the-art models. We believe that the proposed approach is the first visual weather property estimation model trained based on multi-task learning.

scholarly journals Image Caption Generation via Unified Retrieval and Generation-Based Method

2020 ◽  
Vol 10 (18) ◽  
pp. 6235
Author(s):  
Shanshan Zhao ◽  
Lixiang Li ◽  
Haipeng Peng ◽  
Zihang Yang ◽  
Jiaxuan Zhang
Keyword(s):  
State Of The Art ◽  
Target Language ◽  
Visual Features ◽  
Source Image ◽  
Image Captioning ◽  
Data Set ◽  
Advantages And Disadvantages ◽  
Benchmark Data ◽  
Textual Features ◽  
Almost All

Image captioning is a multi-modal transduction task, translating the source image into the target language. Numerous dominant approaches primarily employed the generation-based or the retrieval-based method. These two kinds of frameworks have their advantages and disadvantages. In this work, we make the best of their respective advantages. We adopt the retrieval-based approach to search the visually similar image and their corresponding captions for each queried image in the MSCOCO data set. Based on the retrieved similar sequences and the visual features of the queried image, the proposed de-noising module yielded a set of attended textual features which brought additional textual information for the generation-based model. Finally, the decoder makes use of not only the visual features but also the textual features to generate the output descriptions. Additionally, the incorporated visual encoder and the de-noising module can be applied as a preprocessing component for the decoder-based attention mechanisms. We evaluate the proposed method on the MSCOCO benchmark data set. Extensive experiment yields state-of-the-art performance, and the incorporated module raises the baseline models in terms of almost all the evaluation metrics.

scholarly journals Improving Image Captioning with Conditional Generative Adversarial Nets

2019 ◽  
Vol 33 ◽  
pp. 8142-8150 ◽  
Author(s):  
Chen Chen ◽  
Shuai Mu ◽  
Wanpeng Xiao ◽  
Zexiong Ye ◽  
Liesi Wu ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
State Of The Art ◽  
Evaluation Metrics ◽  
Training Method ◽  
Image Captioning ◽  
Decoder Architecture ◽  
Evaluation Problem ◽  
Special Case

In this paper, we propose a novel conditional-generativeadversarial-nets-based image captioning framework as an extension of traditional reinforcement-learning (RL)-based encoder-decoder architecture. To deal with the inconsistent evaluation problem among different objective language metrics, we are motivated to design some “discriminator” networks to automatically and progressively determine whether generated caption is human described or machine generated. Two kinds of discriminator architectures (CNN and RNNbased structures) are introduced since each has its own advantages. The proposed algorithm is generic so that it can enhance any existing RL-based image captioning framework and we show that the conventional RL training method is just a special case of our approach. Empirically, we show consistent improvements over all language evaluation metrics for different state-of-the-art image captioning models. In addition, the well-trained discriminators can also be viewed as objective image captioning evaluators.

scholarly journals MAT: A Multimodal Attentive Translator for Image Captioning

2017 ◽  
Author(s):  
Chang Liu ◽  
Fuchun Sun ◽  
Changhu Wang ◽  
Feng Wang ◽  
Alan Yuille
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Visual Information ◽  
State Of The Art ◽  
Input Image ◽  
Target Sequence ◽  
Image Captioning ◽  
Proposed Model ◽  
Image Caption Generation ◽  
Image Caption

In this work we formulate the problem of image captioning as a multimodal translation task. Analogous to machine translation, we present a sequence-to-sequence recurrent neural networks (RNN) model for image caption generation. Different from most existing work where the whole image is represented by convolutional neural network (CNN) feature, we propose to represent the input image as a sequence of detected objects which feeds as the source sequence of the RNN model. In this way, the sequential representation of an image can be naturally translated to a sequence of words, as the target sequence of the RNN model. To represent the image in a sequential way, we extract the objects features in the image and arrange them in a order using convolutional neural networks. To further leverage the visual information from the encoded objects, a sequential attention layer is introduced to selectively attend to the objects that are related to generate corresponding words in the sentences. Extensive experiments are conducted to validate the proposed approach on popular benchmark dataset, i.e., MS COCO, and the proposed model surpasses the state-of-the-art methods in all metrics following the dataset splits of previous work. The proposed approach is also evaluated by the evaluation server of MS COCO captioning challenge, and achieves very competitive results, e.g., a CIDEr of 1.029 (c5) and 1.064 (c40).

scholarly journals Learn and Tell: Learning Priors for Image Caption Generation

2020 ◽  
Vol 10 (19) ◽  
pp. 6942
Author(s):  
Pei Liu ◽  
Dezhong Peng ◽  
Ming Zhang
Keyword(s):  
Visual Information ◽  
Feature Vector ◽  
Extraction Process ◽  
Local Region ◽  
Visual Features ◽  
Image Captioning ◽  
Word Generation ◽  
New Methods ◽  
Part Of Speech ◽  
Current Word

In this work, we propose a novel priors-based attention neural network (PANN) for image captioning, which aims at incorporating two kinds of priors, i.e., the probabilities being mentioned for local region proposals (PBM priors) and part-of-speech clues for caption words (POS priors), into a visual information extraction process at each word prediction. This work was inspired by the intuitions that region proposals have different inherent probabilities for image captioning, and that the POS clues bridge the word class (part-of-speech tag) with the categories of visual features. We propose new methods to extract these two priors, in which the PBM priors are obtained by computing the similarities between the caption feature vector and local feature vectors, while the POS priors are predicated at each step of word generation by taking the hidden state of the decoder as input. After that, these two kinds of priors are further incorporated into the PANN module of the decoder to help the decoder extract more accurate visual information for the current word generation. In our experiments, we qualitatively analyzed the proposed approach and quantitatively evaluated several captioning schemes with our PANN on the MS-COCO dataset. Experimental results demonstrate that our proposed method could achieve better performance as well as the effectiveness of the proposed network for image captioning.

scholarly journals Multi-Evidence and Multi-Modal Fusion Network for Ground-Based Cloud Recognition

2020 ◽  
Vol 12 (3) ◽  
pp. 464
Author(s):  
Shuang Liu ◽  
Mei Li ◽  
Zhong Zhang ◽  
Baihua Xiao ◽  
Tariq S. Durrani
Keyword(s):  
Classification Accuracy ◽  
Visual Information ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Visual Features ◽  
Unified Framework ◽  
Global Features ◽  
Heterogeneous Features ◽  
Novel Method ◽  
Global And Local

In recent times, deep neural networks have drawn much attention in ground-based cloud recognition. Yet such kind of approaches simply center upon learning global features from visual information, which causes incomplete representations for ground-based clouds. In this paper, we propose a novel method named multi-evidence and multi-modal fusion network (MMFN) for ground-based cloud recognition, which could learn extended cloud information by fusing heterogeneous features in a unified framework. Namely, MMFN exploits multiple pieces of evidence, i.e., global and local visual features, from ground-based cloud images using the main network and the attentive network. In the attentive network, local visual features are extracted from attentive maps which are obtained by refining salient patterns from convolutional activation maps. Meanwhile, the multi-modal network in MMFN learns multi-modal features for ground-based cloud. To fully fuse the multi-modal and multi-evidence visual features, we design two fusion layers in MMFN to incorporate multi-modal features with global and local visual features, respectively. Furthermore, we release the first multi-modal ground-based cloud dataset named MGCD which not only contains the ground-based cloud images but also contains the multi-modal information corresponding to each cloud image. The MMFN is evaluated on MGCD and achieves a classification accuracy of 88.63% comparative to the state-of-the-art methods, which validates its effectiveness for ground-based cloud recognition.

scholarly journals Automated Event Detection and Classification in Soccer: The Potential of Using Multiple Modalities

2021 ◽  
Vol 3 (4) ◽  
pp. 1030-1054
Author(s):  
Olav Andre Nergård Rongved ◽  
Markus Stige ◽  
Steven Alexander Hicks ◽  
Vajira Lasantha Thambawita ◽  
Cise Midoglu ◽  
...  
Keyword(s):  
Event Detection ◽  
Visual Information ◽  
Prediction Accuracy ◽  
State Of The Art ◽  
Visual Features ◽  
Complex Task ◽  
Multimodal Approach ◽  
Multiple Modalities ◽  
Audio Features ◽  
Video And Audio

Detecting events in videos is a complex task, and many different approaches, aimed at a large variety of use-cases, have been proposed in the literature. Most approaches, however, are unimodal and only consider the visual information in the videos. This paper presents and evaluates different approaches based on neural networks where we combine visual features with audio features to detect (spot) and classify events in soccer videos. We employ model fusion to combine different modalities such as video and audio, and test these combinations against different state-of-the-art models on the SoccerNet dataset. The results show that a multimodal approach is beneficial. We also analyze how the tolerance for delays in classification and spotting time, and the tolerance for prediction accuracy, influence the results. Our experiments show that using multiple modalities improves event detection performance for certain types of events.

scholarly journals Efficient Rank-Based Diffusion Process with Assured Convergence

2021 ◽  
Vol 7 (3) ◽  
pp. 49
Author(s):  
Daniel Carlos Guimarães Pedronette ◽  
Lucas Pascotti Valem ◽  
Longin Jan Latecki
Keyword(s):  
Diffusion Process ◽  
Learning Strategies ◽  
State Of The Art ◽  
Representation Learning ◽  
Theoretical Background ◽  
High Dimensional ◽  
Visual Features ◽  
Learning Approaches ◽  
Previous Decade ◽  
Asymptotic Complexity

Visual features and representation learning strategies experienced huge advances in the previous decade, mainly supported by deep learning approaches. However, retrieval tasks are still performed mainly based on traditional pairwise dissimilarity measures, while the learned representations lie on high dimensional manifolds. With the aim of going beyond pairwise analysis, post-processing methods have been proposed to replace pairwise measures by globally defined measures, capable of analyzing collections in terms of the underlying data manifold. The most representative approaches are diffusion and ranked-based methods. While the diffusion approaches can be computationally expensive, the rank-based methods lack theoretical background. In this paper, we propose an efficient Rank-based Diffusion Process which combines both approaches and avoids the drawbacks of each one. The obtained method is capable of efficiently approximating a diffusion process by exploiting rank-based information, while assuring its convergence. The algorithm exhibits very low asymptotic complexity and can be computed regionally, being suitable to outside of dataset queries. An experimental evaluation conducted for image retrieval and person re-ID tasks on diverse datasets demonstrates the effectiveness of the proposed approach with results comparable to the state-of-the-art.

scholarly journals Effects of Admixtures on the Self Compacting Concrete State of the Art Report

2020 ◽  
Vol 1006 ◽  
pp. 012038
Author(s):  
S Christopher Gnanaraj ◽  
Ramesh Babu Chokkalingam ◽  
G LiziaThankam
Keyword(s):  
State Of The Art ◽  
The Self ◽  
Self Compacting Concrete
Sign in / Sign up

Export Citation Format

Share Document