scholarly journals TCIC: Theme Concepts Learning Cross Language and Vision for Image Captioning

2021 ◽  
Author(s):  
Zhihao Fan ◽  
Zhongyu Wei ◽  
Siyuan Wang ◽  
Ruize Wang ◽  
Zejun Li ◽  
...  
Keyword(s):  
State Of The Art ◽  
Representation Learning ◽  
Experimental Results ◽  
Text Representation ◽  
Image Captioning ◽  
Scene Graph ◽  
Low Level ◽  
Language And Vision ◽  
High Level ◽  
Cross Language

Existing research for image captioning usually represents an image using a scene graph with low-level facts (objects and relations) and fails to capture the high-level semantics. In this paper, we propose a Theme Concepts extended Image Captioning (TCIC) framework that incorporates theme concepts to represent high-level cross-modality semantics. In practice, we model theme concepts as memory vectors and propose Transformer with Theme Nodes (TTN) to incorporate those vectors for image captioning. Considering that theme concepts can be learned from both images and captions, we propose two settings for their representations learning based on TTN. On the vision side, TTN is configured to take both scene graph based features and theme concepts as input for visual representation learning. On the language side, TTN is configured to take both captions and theme concepts as input for text representation re-construction. Both settings aim to generate target captions with the same transformer-based decoder. During the training, we further align representations of theme concepts learned from images and corresponding captions to enforce the cross-modality learning. Experimental results on MS COCO show the effectiveness of our approach compared to some state-of-the-art models.

scholarly journals Image Cationing with Visual-Semantic LSTM

2018 ◽  
Author(s):  
Nannan Li ◽  
Zhenzhong Chen
Keyword(s):  
Visual Processing ◽  
State Of The Art ◽  
Sampling Strategy ◽  
Experimental Results ◽  
Visual Cell ◽  
Semantic Features ◽  
Training Process ◽  
Image Captioning ◽  
Low Level ◽  
High Level

In this paper, a novel image captioning approach is proposed to describe the content of images. Inspired by the visual processing of our cognitive system, we propose a visual-semantic LSTM model to locate the attention objects with their low-level features in the visual cell, and then successively extract high-level semantic features in the semantic cell. In addition, a state perturbation term is introduced to the word sampling strategy in the REINFORCE based method to explore proper vocabularies in the training process. Experimental results on MS COCO and Flickr30K validate the effectiveness of our approach when compared to the state-of-the-art methods.

scholarly journals Deep ChaosNet for Action Recognition in Videos

Complexity ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-5
Author(s):  
Huafeng Chen ◽  
Maosheng Zhang ◽  
Zhengming Gao ◽  
Yunhong Zhao
Keyword(s):  
Neural Network ◽  
Action Recognition ◽  
Deep Neural Network ◽  
Recognition Accuracy ◽  
State Of The Art ◽  
Experimental Results ◽  
Low Level ◽  
Hidden Layer ◽  
High Level ◽  
Standard Action

Current methods of chaos-based action recognition in videos are limited to the artificial feature causing the low recognition accuracy. In this paper, we improve ChaosNet to the deep neural network and apply it to action recognition. First, we extend ChaosNet to deep ChaosNet for extracting action features. Then, we send the features to the low-level LSTM encoder and high-level LSTM encoder for obtaining low-level coding output and high-level coding results, respectively. The agent is a behavior recognizer for producing recognition results. The manager is a hidden layer, responsible for giving behavioral segmentation targets at the high level. Our experiments are executed on two standard action datasets: UCF101 and HMDB51. The experimental results show that the proposed algorithm outperforms the state of the art.

scholarly journals Saliency Detection by Multilevel Deep Pyramid Model

2018 ◽  
Vol 2018 ◽  
pp. 1-11 ◽  
Author(s):  
Hai Wang ◽  
Lei Dai ◽  
Yingfeng Cai ◽  
Long Chen ◽  
Yong Zhang
Keyword(s):  
Background Noise ◽  
State Of The Art ◽  
Saliency Detection ◽  
Saliency Map ◽  
Multiple Features ◽  
Low Level ◽  
Pyramid Model ◽  
High Level ◽  
Different Levels ◽  
Better Than

Traditional salient object detection models are divided into several classes based on low-level features and contrast between pixels. In this paper, we propose a model based on a multilevel deep pyramid (MLDP), which involves fusing multiple features on different levels. Firstly, the MLDP uses the original image as the input for a VGG16 model to extract high-level features and form an initial saliency map. Next, the MLDP further extracts high-level features to form a saliency map based on a deep pyramid. Then, the MLDP obtains the salient map fused with superpixels by extracting low-level features. After that, the MLDP applies background noise filtering to the saliency map fused with superpixels in order to filter out the interference of background noise and form a saliency map based on the foreground. Lastly, the MLDP combines the saliency map fused with the superpixels with the saliency map based on the foreground, which results in the final saliency map. The MLDP is not limited to low-level features while it fuses multiple features and achieves good results when extracting salient targets. As can be seen in our experiment section, the MLDP is better than the other 7 state-of-the-art models across three different public saliency datasets. Therefore, the MLDP has superiority and wide applicability in extraction of salient targets.

scholarly journals Towards High-Level Intrinsic Exploration in Reinforcement Learning

2020 ◽  
Author(s):  
Nicolas Bougie ◽  
Ryutaro Ichise
Keyword(s):  
Reinforcement Learning ◽  
Time Horizon ◽  
State Of The Art ◽  
Experimental Results ◽  
Prior Work ◽  
Extrinsic Rewards ◽  
Intrinsic Reward ◽  
Long Time ◽  
End To End ◽  
High Level

Deep reinforcement learning (DRL) methods traditionally struggle with tasks where environment rewards are sparse or delayed, which entails that exploration remains one of the key challenges of DRL. Instead of solely relying on extrinsic rewards, many state-of-the-art methods use intrinsic curiosity as exploration signal. While they hold promise of better local exploration, discovering global exploration strategies is beyond the reach of current methods. We propose a novel end-to-end intrinsic reward formulation that introduces high-level exploration in reinforcement learning. Our curiosity signal is driven by a fast reward that deals with local exploration and a slow reward that incentivizes long-time horizon exploration strategies. We formulate curiosity as the error in an agent’s ability to reconstruct the observations given their contexts. Experimental results show that this high-level exploration enables our agents to outperform prior work in several Atari games.

A Visual Saliency Detection Approach by Fusing Low-Level Priors With High-Level Priors

2019 ◽  
Vol 9 (3) ◽  
pp. 23-37
Author(s):  
Monika Singh ◽  
Anand Singh Singh Jalal ◽  
Ruchira Manke ◽  
Aamir Khan
Keyword(s):  
Minimum Distance ◽  
Saliency Detection ◽  
Visual Saliency ◽  
Research Area ◽  
Experimental Results ◽  
Low Level ◽  
Detection Approach ◽  
Art Methods ◽  
High Level ◽  
Visual Saliency Detection

Saliency detection has always been a challenging and interesting research area for researchers. The existing methodologies either focus on foreground regions or background regions of an image by computing low-level features. However, considering only low-level features did not produce worthy results. In this paper, low-level features, which are extracted using super pixels, are embodied with high-level priors. The background features are assumed as the low-level prior due to the similarity in the background areas and boundary of an image which are interconnected and have minimum distance in between them. High-level priors such as location, color, and semantic prior are incorporated with low-level prior to spotlight the salient area in the image. The experimental results illustrate that the proposed approach outperform the sate-of-the-art methods.

scholarly journals Universal Word Segmentation: Implementation and Interpretation

2018 ◽  
Vol 6 ◽  
pp. 421-435 ◽  
Author(s):  
Yan Shao ◽  
Christian Hardmeier ◽  
Joakim Nivre
Keyword(s):  
State Of The Art ◽  
Word Segmentation ◽  
Experimental Results ◽  
Word Boundary ◽  
Writing Systems ◽  
Low Level ◽  
Wide Range ◽  
Segmentation Accuracy ◽  
Small Set

Word segmentation is a low-level NLP task that is non-trivial for a considerable number of languages. In this paper, we present a sequence tagging framework and apply it to word segmentation for a wide range of languages with different writing systems and typological characteristics. Additionally, we investigate the correlations between various typological factors and word segmentation accuracy. The experimental results indicate that segmentation accuracy is positively related to word boundary markers and negatively to the number of unique non-segmental terms. Based on the analysis, we design a small set of language-specific settings and extensively evaluate the segmentation system on the Universal Dependencies datasets. Our model obtains state-of-the-art accuracies on all the UD languages. It performs substantially better on languages that are non-trivial to segment, such as Chinese, Japanese, Arabic and Hebrew, when compared to previous work.

scholarly journals Towards Explanatory Interactive Image Captioning Using Top-Down and Bottom-Up Features, Beam Search and Re-ranking

2020 ◽  
Vol 34 (4) ◽  
pp. 571-584
Author(s):  
Rajarshi Biswas ◽  
Michael Barz ◽  
Daniel Sonntag
Keyword(s):  
State Of The Art ◽  
Input Image ◽  
The State ◽  
Beam Search ◽  
Image Captioning ◽  
Bottom Up ◽  
Interactive Machine Learning ◽  
Joint Embedding ◽  
Bounding Boxes ◽  
High Level

AbstractImage captioning is a challenging multimodal task. Significant improvements could be obtained by deep learning. Yet, captions generated by humans are still considered better, which makes it an interesting application for interactive machine learning and explainable artificial intelligence methods. In this work, we aim at improving the performance and explainability of the state-of-the-art method Show, Attend and Tell by augmenting their attention mechanism using additional bottom-up features. We compute visual attention on the joint embedding space formed by the union of high-level features and the low-level features obtained from the object specific salient regions of the input image. We embed the content of bounding boxes from a pre-trained Mask R-CNN model. This delivers state-of-the-art performance, while it provides explanatory features. Further, we discuss how interactive model improvement can be realized through re-ranking caption candidates using beam search decoders and explanatory features. We show that interactive re-ranking of beam search candidates has the potential to outperform the state-of-the-art in image captioning.

scholarly journals Backbone Cannot Be Trained at Once: Rolling Back to Pre-Trained Network for Person Re-Identification

2019 ◽  
Vol 33 ◽  
pp. 8859-8867 ◽  
Author(s):  
Youngmin Ro ◽  
Jongwon Choi ◽  
Dae Ung Jo ◽  
Byeongho Heo ◽  
Jongin Lim ◽  
...  
Keyword(s):  
Network Architecture ◽  
State Of The Art ◽  
Fine Tuning ◽  
Neural Network Architecture ◽  
Large Dataset ◽  
Low Level ◽  
Tuning Method ◽  
Improved Performance ◽  
High Level ◽  
Tuning Strategy

In person re-identification (ReID) task, because of its shortage of trainable dataset, it is common to utilize fine-tuning method using a classification network pre-trained on a large dataset. However, it is relatively difficult to sufficiently finetune the low-level layers of the network due to the gradient vanishing problem. In this work, we propose a novel fine-tuning strategy that allows low-level layers to be sufficiently trained by rolling back the weights of high-level layers to their initial pre-trained weights. Our strategy alleviates the problem of gradient vanishing in low-level layers and robustly trains the low-level layers to fit the ReID dataset, thereby increasing the performance of ReID tasks. The improved performance of the proposed strategy is validated via several experiments. Furthermore, without any addons such as pose estimation or segmentation, our strategy exhibits state-of-the-art performance using only vanilla deep convolutional neural network architecture.

scholarly journals JSOX: A Justly Simple Objectization for XML

2014 ◽  
Author(s):  
Steven J. DeRose
Keyword(s):  
The Other ◽  
Low Level ◽  
Other Hand ◽  
High Level ◽  
Cross Language

XML can be as easy to work with as JSON. However, this has not been obvious until now. JSON is easy because it supports only datatypes that are already native to Javascript and uses the same syntax to access them (such as [1:10], ["x"], and “.” notation). XML, on the other hand, supports additional datatypes, and is most commonly handled via SAX or DOM, both of which are low-level and meant to be cross-language. Typical developers want high-level access that feels “native” in the language they are using. These shortcomings have little or nothing to do with XML, and can be remedied by a different API. Software that demonstrates this is presented and described. It uses Python's richer set of abstract datatypes (such as tuples and sets), and provides native Python style syntax with richer semantics than JSON or Javascript.

scholarly journals Context-Aware Image Inpainting with Learned Semantic Priors

2021 ◽  
Author(s):  
Wendong Zhang ◽  
Junwei Zhu ◽  
Ying Tai ◽  
Yunbo Wang ◽  
Wenqing Chu ◽  
...  
Keyword(s):  
State Of The Art ◽  
Contextual Information ◽  
Image Inpainting ◽  
Context Aware ◽  
Global Context ◽  
Low Level ◽  
Complex Scenes ◽  
Knowledge Distillation ◽  
Image Generator ◽  
High Level

Recent advances in image inpainting have shown impressive results for generating plausible visual details on rather simple backgrounds. However, for complex scenes, it is still challenging to restore reasonable contents as the contextual information within the missing regions tends to be ambiguous. To tackle this problem, we introduce pretext tasks that are semantically meaningful to estimating the missing contents. In particular, we perform knowledge distillation on pretext models and adapt the features to image inpainting. The learned semantic priors ought to be partially invariant between the high-level pretext task and low-level image inpainting, which not only help to understand the global context but also provide structural guidance for the restoration of local textures. Based on the semantic priors, we further propose a context-aware image inpainting model, which adaptively integrates global semantics and local features in a unified image generator. The semantic learner and the image generator are trained in an end-to-end manner. We name the model SPL to highlight its ability to learn and leverage semantic priors. It achieves the state of the art on Places2, CelebA, and Paris StreetView datasets

Sign in / Sign up

Export Citation Format

Share Document