scholarly journals Context-Fused Guidance for Image Captioning Using Sequence-Level Training

2022 ◽  
Vol 2022 ◽  
pp. 1-9
Author(s):  
Junlong Feng ◽  
Jianping Zhao
Keyword(s):  
Image Guidance ◽  
Semantic Information ◽  
Semantic Knowledge ◽  
Visual Features ◽  
Image Description ◽  
Visual Concept ◽  
Image Captioning ◽  
Proposed Model ◽  
Fused Image ◽  
Level Training

Recent image captioning models based on the encoder-decoder framework have achieved remarkable success in humanlike sentence generation. However, an explicit separation between encoder and decoder brings out a disconnection between the image and sentence. It usually leads to a rough image description: the generated caption only contains main instances but neglects additional objects and scenes unexpectedly, which reduces the caption consistency of the image. To address this issue, we proposed an image captioning system within context-fused guidance in this paper. It incorporates regional and global image representation as the compositional visual features to learn the objects and attributes in images. To integrate image-level semantic information, the visual concept is employed. To avoid misleading decoding, a context fusion gate is introduced to calculate the textual context by selectively aggregating the information of visual concept and word embedding. Subsequently, the context-fused image guidance is formulated based on the compositional visual features and textual context. It provides the decoder with informative semantic knowledge. Finally, a captioner with a two-layer LSTM architecture is constructed to generate captions. Moreover, to overcome the exposure bias, we train the proposed model through sequence decision-making. The experiments conducted on the MS COCO dataset show the outstanding performance of our work. The linguistic analysis demonstrates that our model improves the caption consistency of the image.

Chinese Image Captioning via Fuzzy Attention-based DenseNet-BiLSTM

2021 ◽  
Vol 17 (1s) ◽  
pp. 1-18 ◽  
Author(s):  
Huimin Lu ◽  
Rui Yang ◽  
Zhenrong Deng ◽  
Yonglin Zhang ◽  
Guangwei Gao ◽  
...  
Keyword(s):  
Feature Extraction ◽  
Contextual Information ◽  
Image Features ◽  
Context Information ◽  
Global Information ◽  
Image Description ◽  
Image Captioning ◽  
Image Content ◽  
Single Feature ◽  
Proposed Model

Chinese image description generation tasks usually have some challenges, such as single-feature extraction, lack of global information, and lack of detailed description of the image content. To address these limitations, we propose a fuzzy attention-based DenseNet-BiLSTM Chinese image captioning method in this article. In the proposed method, we first improve the densely connected network to extract features of the image at different scales and to enhance the model’s ability to capture the weak features. At the same time, a bidirectional LSTM is used as the decoder to enhance the use of context information. The introduction of an improved fuzzy attention mechanism effectively improves the problem of correspondence between image features and contextual information. We conduct experiments on the AI Challenger dataset to evaluate the performance of the model. The results show that compared with other models, our proposed model achieves higher scores in objective quantitative evaluation indicators, including BLEU , BLEU , METEOR, ROUGEl, and CIDEr. The generated description sentence can accurately express the image content.

scholarly journals Progressive Domain-Independent Feature Decomposition Network for Zero-Shot Sketch-Based Image Retrieval

2020 ◽  
Author(s):  
Xinxun Xu ◽  
Muli Yang ◽  
Yanhua Yang ◽  
Hao Wang
Keyword(s):  
Image Retrieval ◽  
Semantic Information ◽  
Semantic Space ◽  
Semantic Knowledge ◽  
Visual Features ◽  
Semantic Features ◽  
Retrieval Task ◽  
Common Space ◽  
Low Dimensional ◽  
Domain Independent

Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) is a specific cross-modal retrieval task for searching natural images given free-hand sketches under the zero-shot scenario. Most existing methods solve this problem by simultaneously projecting visual features and semantic supervision into a low-dimensional common space for efficient retrieval. However, such low-dimensional projection destroys the completeness of semantic knowledge in original semantic space, so that it is unable to transfer useful knowledge well when learning semantic features from different modalities. Moreover, the domain information and semantic information are entangled in visual features, which is not conducive for cross-modal matching since it will hinder the reduction of domain gap between sketch and image. In this paper, we propose a Progressive Domain-independent Feature Decomposition (PDFD) network for ZS-SBIR. Specifically, with the supervision of original semantic knowledge, PDFD decomposes visual features into domain features and semantic ones, and then the semantic features are projected into common space as retrieval features for ZS-SBIR. The progressive projection strategy maintains strong semantic supervision. Besides, to guarantee the retrieval features to capture clean and complete semantic information, the cross-reconstruction loss is introduced to encourage that any combinations of retrieval features and domain features can reconstruct the visual features. Extensive experiments demonstrate the superiority of our PDFD over state-of-the-art competitors.

Integrating Scene Semantic Knowledge into Image Captioning

2021 ◽  
Vol 17 (2) ◽  
pp. 1-22
Author(s):  
Haiyang Wei ◽  
Zhixin Li ◽  
Feicheng Huang ◽  
Canlong Zhang ◽  
Huifang Ma ◽  
...  
Keyword(s):  
Visual Attention ◽  
Visual Information ◽  
Semantic Information ◽  
Language Model ◽  
Semantic Knowledge ◽  
Attention Mechanism ◽  
Image Captioning ◽  
Attention Model ◽  
Visual Attention Mechanism ◽  
Focus Intensity

Most existing image captioning methods use only the visual information of the image to guide the generation of captions, lack the guidance of effective scene semantic information, and the current visual attention mechanism cannot adjust the focus intensity on the image. In this article, we first propose an improved visual attention model. At each timestep, we calculated the focus intensity coefficient of the attention mechanism through the context information of the model, then automatically adjusted the focus intensity of the attention mechanism through the coefficient to extract more accurate visual information. In addition, we represented the scene semantic knowledge of the image through topic words related to the image scene, then added them to the language model. We used the attention mechanism to determine the visual information and scene semantic information that the model pays attention to at each timestep and combined them to enable the model to generate more accurate and scene-specific captions. Finally, we evaluated our model on Microsoft COCO (MSCOCO) and Flickr30k standard datasets. The experimental results show that our approach generates more accurate captions and outperforms many recent advanced models in various evaluation metrics.

A Hindi Image Caption Generation Framework Using Deep Learning

2021 ◽  
Vol 20 (2) ◽  
pp. 1-19
Author(s):  
Santosh Kumar Mishra ◽  
Rijul Dhir ◽  
Sriparna Saha ◽  
Pushpak Bhattacharyya
Keyword(s):  
Computer Vision ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
English Language ◽  
Image Captioning ◽  
Textual Description ◽  
Proposed Model ◽  
Hindi Language ◽  
The Given

Image captioning is the process of generating a textual description of an image that aims to describe the salient parts of the given image. It is an important problem, as it involves computer vision and natural language processing, where computer vision is used for understanding images, and natural language processing is used for language modeling. A lot of works have been done for image captioning for the English language. In this article, we have developed a model for image captioning in the Hindi language. Hindi is the official language of India, and it is the fourth most spoken language in the world, spoken in India and South Asia. To the best of our knowledge, this is the first attempt to generate image captions in the Hindi language. A dataset is manually created by translating well known MSCOCO dataset from English to Hindi. Finally, different types of attention-based architectures are developed for image captioning in the Hindi language. These attention mechanisms are new for the Hindi language, as those have never been used for the Hindi language. The obtained results of the proposed model are compared with several baselines in terms of BLEU scores, and the results show that our model performs better than others. Manual evaluation of the obtained captions in terms of adequacy and fluency also reveals the effectiveness of our proposed approach. Availability of resources : The codes of the article are available at https://github.com/santosh1821cs03/Image_Captioning_Hindi_Language ; The dataset will be made available: http://www.iitp.ac.in/∼ai-nlp-ml/resources.html .

Enhanced Content-Based Image Retrieval Using Information Oriented Angle-Based Local Tri-Directional Weber Patterns

2021 ◽  
pp. 2150046
Author(s):  
Gangavarapu Venkata Satya Kumar ◽  
Pillutla Gopala Krishna Mohan
Keyword(s):  
Image Retrieval ◽  
Computer Applications ◽  
Content Based Image Retrieval ◽  
Visual Features ◽  
The Novel ◽  
Image Content ◽  
Query Image ◽  
Proposed Model ◽  
Maximum Mutual Information ◽  
Basic Features

In diverse computer applications, the analysis of image content plays a key role. This image content might be either textual (like text appearing in the images) or visual (like shape, color, texture). These two image contents consist of image’s basic features and therefore turn out to be as the major advantage for any of the implementation. Many of the art models are based on the visual search or annotated text for Content-Based Image Retrieval (CBIR) models. There is more demand toward multitasking, a new method needs to be introduced with the combination of both textual and visual features. This paper plans to develop the intelligent CBIR system for the collection of different benchmark texture datasets. Here, a new descriptor named Information Oriented Angle-based Local Tri-directional Weber Patterns (IOA-LTriWPs) is adopted. The pattern is operated not only based on tri-direction and eight neighborhood pixels but also based on four angles [Formula: see text], [Formula: see text], [Formula: see text], and [Formula: see text]. Once the patterns concerning tri-direction, eight neighborhood pixels, and four angles are taken, the best patterns are selected based on maximum mutual information. Moreover, the histogram computation of the patterns provides the final feature vector, from which the new weighted feature extraction is performed. As a new contribution, the novel weight function is optimized by the Improved MVO on random basis (IMVO-RB), in such a way that the precision and recall of the retrieved image is high. Further, the proposed model has used the logarithmic similarity called Mean Square Logarithmic Error (MSLE) between the features of the query image and trained images for retrieving the concerned images. The analyses on diverse texture image datasets have validated the accuracy and efficiency of the developed pattern over existing.

Thematic object pairs produce stronger and faster perceptual grouping than taxonomic pairs

2021 ◽  
Author(s):  
Joseph Nah ◽  
Joy Geng
Keyword(s):  
Visual Processing ◽  
Perceptual Grouping ◽  
Semantic Information ◽  
Semantic Knowledge ◽  
Perceptual Unit ◽  
Critical Difference ◽  
Common Event ◽  
Taxonomic Relationships ◽  
Different Types ◽  
Fundamental Units

While objects are fundamental units of vision that convey meaning, how different types of semantic knowledge affect perception is not fully understood. In contrast, the concept literature divides semantic information into taxonomic and thematic types. Taxonomic relationships reflect categorization by similarities (e.g., dog – wolf); thematic groups are based on complementary relationships shared within a common event (e.g., swimsuit – goggles; pool). A critical difference between these two information types is that thematic relationships are learned from the experienced co-occurrence of objects whereas taxonomic relationships are learned abstractly. In two studies, we test the hypothesis that visual processing of thematically related objects is more rapid because they serve as mutual visual primes and form a perceptual unit. The results demonstrate that learned co-occurrence not only shapes semantic knowledge, but also affects low level visual processing, revealing a link between how information is acquired (e.g., experienced vs. unobserved) and how it modulates perception.

scholarly journals Variational Autoencoder-Based Multiple Image Captioning Using a Caption Attention Map

2019 ◽  
Vol 9 (13) ◽  
pp. 2699 ◽  
Author(s):  
Boeun Kim ◽  
Saim Shin ◽  
Hyedong Jung
Keyword(s):  
Generative Models ◽  
Research Topic ◽  
Video Data ◽  
Image Feature ◽  
Image Captioning ◽  
Multiple Image ◽  
Visually Impaired People ◽  
Proposed Model ◽  
Variational Autoencoder ◽  
Latent Distribution

Image captioning is a promising research topic that is applicable to services that search for desired content in a large amount of video data and a situation explanation service for visually impaired people. Previous research on image captioning has been focused on generating one caption per image. However, to increase usability in applications, it is necessary to generate several different captions that contain various representations for an image. We propose a method to generate multiple captions using a variational autoencoder, which is one of the generative models. Because an image feature plays an important role when generating captions, a method to extract a Caption Attention Map (CAM) of the image is proposed, and CAMs are projected to a latent distribution. In addition, methods for the evaluation of multiple image captioning tasks are proposed that have not yet been actively researched. The proposed model outperforms in the aspect of diversity compared with the base model when the accuracy is comparable. Moreover, it is verified that the model using CAM generates detailed captions describing various content in the image.

scholarly journals Video Description Model Based on Temporal-Spatial and Channel Multi-Attention Mechanisms

2020 ◽  
Vol 10 (12) ◽  
pp. 4312 ◽  
Author(s):  
Jie Xu ◽  
Haoliang Wei ◽  
Linke Li ◽  
Qiuru Fu ◽  
Jinhong Guo
Keyword(s):  
Neural Network ◽  
Spatial Attention ◽  
Semantic Information ◽  
Attention Mechanism ◽  
Visual Features ◽  
Feature Maps ◽  
Global Features ◽  
Model Based ◽  
Video Description ◽  
Video Visualization

Video description plays an important role in the field of intelligent imaging technology. Attention perception mechanisms are extensively applied in video description models based on deep learning. Most existing models use a temporal-spatial attention mechanism to enhance the accuracy of models. Temporal attention mechanisms can obtain the global features of a video, whereas spatial attention mechanisms obtain local features. Nevertheless, because each channel of the convolutional neural network (CNN) feature maps has certain spatial semantic information, it is insufficient to merely divide the CNN features into regions and then apply a spatial attention mechanism. In this paper, we propose a temporal-spatial and channel attention mechanism that enables the model to take advantage of various video features and ensures the consistency of visual features between sentence descriptions to enhance the effect of the model. Meanwhile, in order to prove the effectiveness of the attention mechanism, this paper proposes a video visualization model based on the video description. Experimental results show that, our model has achieved good performance on the Microsoft Video Description (MSVD) dataset and a certain improvement on the Microsoft Research-Video to Text (MSR-VTT) dataset.

scholarly journals Image quality classification algorithm based on InceptionV3 and SVM

2019 ◽  
Vol 277 ◽  
pp. 02036
Author(s):  
Yu Li ◽  
Lizhuang Liu
Keyword(s):  
Image Quality ◽  
Data Augmentation ◽  
Classification Problem ◽  
Support Vector ◽  
Image Size ◽  
Image Description ◽  
Training Strategy ◽  
Video Images ◽  
Quality Classification ◽  
Proposed Model

In this work we investigate the use of deep learning for image quality classification problem. We use a pre-trained Convolutional Neural Network (CNN) for image description, and the Support Vector Machine (SVM) model is trained as an image quality classifier whose inputs are normalized features extracted by the CNN model. We report on different design choices, ranging from the use of various CNN architectures to the use of features extracted from different layers of a CNN model. To cope with the problem of a lack of adequate amounts of distorted picture data, a novel training strategy of multi-scale training, which is selecting a new image size for training after several batches, combined with data augmentation is introduced. The experimental results tested on the actual monitoring video images shows that the proposed model can accurately classify distorted images.

Sign in / Sign up

Export Citation Format

Share Document