scholarly journals Densely Supervised Hierarchical Policy-Value Network for Image Paragraph Generation

2019 ◽  
Author(s):  
Siying Wu ◽  
Zheng-Jun Zha ◽  
Zilei Wang ◽  
Houqiang Li ◽  
Feng Wu
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
Cross Entropy ◽  
Image Captioning ◽  
Value Network ◽  
Entropy Loss ◽  
Fine Grained ◽  
Performance Improvements ◽  
Single Sentence ◽  
Multiple State

Image paragraph generation aims to describe an image with a paragraph in natural language. Compared to image captioning with a single sentence, paragraph generation provides more expressive and fine-grained description for storytelling. Existing approaches mainly optimize paragraph generator towards minimizing word-wise cross entropy loss, which neglects linguistic hierarchy of paragraph and results in ``sparse" supervision for generator learning. In this paper, we propose a novel Densely Supervised Hierarchical Policy-Value (DHPV) network for effective paragraph generation. We design new hierarchical supervisions consisting of hierarchical rewards and values at both sentence and word levels. The joint exploration of hierarchical rewards and values provides dense supervision cues for learning effective paragraph generator. We propose a new hierarchical policy-value architecture which exploits compositionality at token-to-token and sentence-to-sentence levels simultaneously and can preserve the semantic and syntactic constituent integrity. Extensive experiments on the Stanford image-paragraph benchmark have demonstrated the effectiveness of the proposed DHPV approach with performance improvements over multiple state-of-the-art methods.

scholarly journals Show, Recall, and Tell: Image Captioning with Recall Mechanism

2020 ◽  
Vol 34 (07) ◽  
pp. 12176-12183
Author(s):  
Li Wang ◽  
Zechen Bai ◽  
Yonghua Zhang ◽  
Hongtao Lu
Keyword(s):  
State Of The Art ◽  
Text Retrieval ◽  
Text Summarization ◽  
Cross Entropy ◽  
Image Captioning ◽  
Soft Switch ◽  
Entropy Loss ◽  
Human Conduct ◽  
Art Methods ◽  
The Way

Generating natural and accurate descriptions in image captioning has always been a challenge. In this paper, we propose a novel recall mechanism to imitate the way human conduct captioning. There are three parts in our recall mechanism : recall unit, semantic guide (SG) and recalled-word slot (RWS). Recall unit is a text-retrieval module designed to retrieve recalled words for images. SG and RWS are designed for the best use of recalled words. SG branch can generate a recalled context, which can guide the process of generating caption. RWS branch is responsible for copying recalled words to the caption. Inspired by pointing mechanism in text summarization, we adopt a soft switch to balance the generated-word probabilities between SG and RWS. In the CIDEr optimization step, we also introduce an individual recalled-word reward (WR) to boost training. Our proposed methods (SG+RWS+WR) achieve BLEU-4 / CIDEr / SPICE scores of 36.6 / 116.9 / 21.3 with cross-entropy loss and 38.7 / 129.1 / 22.4 with CIDEr optimization on MSCOCO Karpathy test split, which surpass the results of other state-of-the-art methods.

scholarly journals Deep Learning with Taxonomic Loss for Plant Identification

2019 ◽  
Vol 2019 ◽  
pp. 1-8
Author(s):  
Danzi Wu ◽  
Xue Han ◽  
Guan Wang ◽  
Yu Sun ◽  
Haiyan Zhang ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Cross Entropy ◽  
Classification Task ◽  
Plant Identification ◽  
Entropy Loss ◽  
Fine Grained ◽  
Performance Improvements ◽  
The Hierarchical Structure ◽  
Significant Performance

Plant identification is a fine-grained classification task which aims to identify the family, genus, and species according to plant appearance features. Inspired by the hierarchical structure of taxonomic tree, the taxonomic loss was proposed, which could encode the hierarchical relationships among multilevel labels into the deep learning objective function by simple group and sum operation. By training various neural networks on PlantCLEF 2015 and PlantCLEF 2017 datasets, the experimental results demonstrated that the proposed loss function was easy to implement and outperformed the most commonly adopted cross-entropy loss. Eight neural networks were trained, respectively, by two different loss functions on PlantCLEF 2015 dataset, and the models trained by taxonomic loss led to significant performance improvements. On PlantCLEF 2017 dataset with 10,000 species, the SENet-154 model trained by taxonomic loss achieved the accuracies of 84.07%, 79.97%, and 73.61% at family, genus and species levels, which improved those of model trained by cross-entropy loss by 2.23%, 1.34%, and 1.08%, respectively. The taxonomic loss could further facilitate the fine-grained classification task with hierarchical labels.

scholarly journals Biomedical Concept Recognition Using Deep Neural Sequence Models

2019 ◽  
Author(s):  
Negacy D. Hailu ◽  
Michael Bada ◽  
Asmelash Teka Hadgu ◽  
Lawrence E. Hunter
Keyword(s):  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Machine Translation ◽  
Language Processing ◽  
State Of The Art ◽  
Conditional Random Field ◽  
Concept Recognition ◽  
Performance Improvements ◽  
Art Performance

AbstractBackgroundthe automated identification of mentions of ontological concepts in natural language texts is a central task in biomedical information extraction. Despite more than a decade of effort, performance in this task remains below the level necessary for many applications.Resultsrecently, applications of deep learning in natural language processing have demonstrated striking improvements over previously state-of-the-art performance in many related natural language processing tasks. Here we demonstrate similarly striking performance improvements in recognizing biomedical ontology concepts in full text journal articles using deep learning techniques originally developed for machine translation. For example, our best performing system improves the performance of the previous state-of-the-art in recognizing terms in the Gene Ontology Biological Process hierarchy, from a previous best F1 score of 0.40 to an F1 of 0.70, nearly halving the error rate. Nearly all other ontologies show similar performance improvements.ConclusionsA two-stage concept recognition system, which is a conditional random field model for span detection followed by a deep neural sequence model for normalization, improves the state-of-the-art performance for biomedical concept recognition. Treating the biomedical concept normalization task as a sequence-to-sequence mapping task similar to neural machine translation improves performance.

scholarly journals Cross Entropy of Neural Language Models at Infinity—A New Bound of the Entropy Rate

Entropy ◽  
2018 ◽  
Vol 20 (11) ◽  
pp. 839 ◽  
Author(s):  
Shuntaro Takahashi ◽  
Kumiko Tanaka-Ishii
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
Training Data ◽  
Language Models ◽  
Cross Entropy ◽  
Entropy Rate ◽  
Natural Language Text ◽  
Power Law Decay ◽  
The Cross ◽  
Two Parameters

Neural language models have drawn a lot of attention for their strong ability to predict natural language text. In this paper, we estimate the entropy rate of natural language with state-of-the-art neural language models. To obtain the estimate, we consider the cross entropy, a measure of the prediction accuracy of neural language models, under the theoretically ideal conditions that they are trained with an infinitely large dataset and receive an infinitely long context for prediction. We empirically verify that the effects of the two parameters, the training data size and context length, on the cross entropy consistently obey a power-law decay with a positive constant for two different state-of-the-art neural language models with different language datasets. Based on the verification, we obtained 1.12 bits per character for English by extrapolating the two parameters to infinity. This result suggests that the upper bound of the entropy rate of natural language is potentially smaller than the previously reported values.

Fine-Grained Image Retrieval via Piecewise Cross Entropy loss

2020 ◽  
Vol 93 ◽  
pp. 103820 ◽  
Author(s):  
Xianxian Zeng ◽  
Yun Zhang ◽  
Xiaodong Wang ◽  
Kairui Chen ◽  
Dong Li ◽  
...  
Keyword(s):  
Image Retrieval ◽  
Cross Entropy ◽  
Entropy Loss ◽  
Fine Grained

scholarly journals Panoptic Segmentation-Based Attention for Image Captioning

2020 ◽  
Vol 10 (1) ◽  
pp. 391
Author(s):  
Wenjie Cai ◽  
Zheng Xiong ◽  
Xianfang Sun ◽  
Paul L. Rosin ◽  
Longcun Jin ◽  
...  
Keyword(s):  
Main Part ◽  
State Of The Art ◽  
Image Representation ◽  
Experimental Results ◽  
Competitive Performance ◽  
Image Captioning ◽  
Feature Vectors ◽  
Fine Grained ◽  
Art Methods

Image captioning is the task of generating textual descriptions of images. In order to obtain a better image representation, attention mechanisms have been widely adopted in image captioning. However, in existing models with detection-based attention, the rectangular attention regions are not fine-grained, as they contain irrelevant regions (e.g., background or overlapped regions) around the object, making the model generate inaccurate captions. To address this issue, we propose panoptic segmentation-based attention that performs attention at a mask-level (i.e., the shape of the main part of an instance). Our approach extracts feature vectors from the corresponding segmentation regions, which is more fine-grained than current attention mechanisms. Moreover, in order to process features of different classes independently, we propose a dual-attention module which is generic and can be applied to other frameworks. Experimental results showed that our model could recognize the overlapped objects and understand the scene better. Our approach achieved competitive performance against state-of-the-art methods. We made our code available.

Dual Cross-Entropy Loss for Small-Sample Fine-Grained Vehicle Classification

2019 ◽  
Vol 68 (5) ◽  
pp. 4204-4212 ◽  
Author(s):  
Xiaoxu Li ◽  
Liyun Yu ◽  
Dongliang Chang ◽  
Zhanyu Ma ◽  
Jie Cao
Keyword(s):  
Small Sample ◽  
Cross Entropy ◽  
Vehicle Classification ◽  
Entropy Loss ◽  
Fine Grained

scholarly journals Multi-Point Semantic Representation for Intent Classification

2020 ◽  
Vol 34 (05) ◽  
pp. 9531-9538
Author(s):  
Jinghan Zhang ◽  
Yuxiao Ye ◽  
Yue Zhang ◽  
Likun Qiu ◽  
Bin Fu ◽  
...  
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
Semantic Representation ◽  
Coarse Grained ◽  
Semantic Parsing ◽  
Language Understanding ◽  
Attention Model ◽  
Object Condition ◽  
Fine Grained ◽  
Query Type

Detecting user intents from utterances is the basis of natural language understanding (NLU) task. To understand the meaning of utterances, some work focuses on fully representing utterances via semantic parsing in which annotation cost is labor-intentsive. While some researchers simply view this as intent classification or frequently asked questions (FAQs) retrieval, they do not leverage the shared utterances among different intents. We propose a simple and novel multi-point semantic representation framework with relatively low annotation cost to leverage the fine-grained factor information, decomposing queries into four factors, i.e., topic, predicate, object/condition, query type. Besides, we propose a compositional intent bi-attention model under multi-task learning with three kinds of attention mechanisms among queries, labels and factors, which jointly combines coarse-grained intent and fine-grained factor information. Extensive experiments show that our framework and model significantly outperform several state-of-the-art approaches with an improvement of 1.35%-2.47% in terms of accuracy.

Sign in / Sign up

Export Citation Format

Share Document