scholarly journals Leveraging Human Attention in Novel Object Captioning

2021 ◽  
Author(s):  
Xianyu Chen ◽  
Ming Jiang ◽  
Qi Zhao
Keyword(s):  
State Of The Art ◽  
Source Code ◽  
Training Data ◽  
Training Method ◽  
Image Captioning ◽  
Novel Object ◽  
Text Corpora ◽  
Human Attention ◽  
Gating Mechanism ◽  
Novel Objects

Image captioning models depend on training with paired image-text corpora, which poses various challenges in describing images containing novel objects absent from the training data. While previous novel object captioning methods rely on external image taggers or object detectors to describe novel objects, we present the Attention-based Novel Object Captioner (ANOC) that complements novel object captioners with human attention features that characterize generally important information independent of tasks. It introduces a gating mechanism that adaptively incorporates human attention with self-learned machine attention, with a Constrained Self-Critical Sequence Training method to address the exposure bias while maintaining constraints of novel object descriptions. Extensive experiments conducted on the nocaps and Held-Out COCO datasets demonstrate that our method considerably outperforms the state-of-the-art novel object captioners. Our source code is available at https://github.com/chenxy99/ANOC.

scholarly journals Feature Deformation Meta-Networks in Image Captioning of Novel Objects

2020 ◽  
Vol 34 (07) ◽  
pp. 10494-10501
Author(s):  
Tingjia Cao ◽  
Ke Han ◽  
Xiaomei Wang ◽  
Lin Ma ◽  
Yanwei Fu ◽  
...  
Keyword(s):  
Image Features ◽  
The Novel ◽  
Image Captioning ◽  
Novel Object ◽  
Detection Model ◽  
Novel Objects ◽  
Visual Concepts ◽  
Set Sharing ◽  
Ablation Study ◽  
Object Features

This paper studies the task of image captioning with novel objects, which only exist in testing images. Intrinsically, this task can reflect the generalization ability of models in understanding and captioning the semantic meanings of visual concepts and objects unseen in training set, sharing the similarity to one/zero-shot learning. The critical difficulty thus comes from that no paired images and sentences of the novel objects can be used to help train the captioning model. Inspired by recent work (Chen et al. 2019b) that boosts one-shot learning by learning to generate various image deformations, we propose learning meta-networks for deforming features for novel object captioning. To this end, we introduce the feature deformation meta-networks (FDM-net), which is trained on source data, and learn to adapt to the novel object features detected by the auxiliary detection model. FDM-net includes two sub-nets: feature deformation, and scene graph sentence reconstruction, which produce the augmented image features and corresponding sentences, respectively. Thus, rather than directly deforming images, FDM-net can efficiently and dynamically enlarge the paired images and texts by learning to deform image features. Extensive experiments are conducted on the widely used novel object captioning dataset, and the results show the effectiveness of our FDM-net. Ablation study and qualitative visualization further give insights of our model.

scholarly journals A Girl Has No Name: Automated Authorship Obfuscation using Mutant-X

2019 ◽  
Vol 2019 (4) ◽  
pp. 54-71
Author(s):  
Asad Mahmood ◽  
Faizan Ahmad ◽  
Zubair Shafiq ◽  
Padmini Srinivasan ◽  
Fareed Zaffar
Keyword(s):  
State Of The Art ◽  
Fitness Function ◽  
Random Search ◽  
Training Data ◽  
Authorship Attribution ◽  
Future Research ◽  
Original Text ◽  
Text Corpora ◽  
Semantic Relevance ◽  
Better Than

Abstract Stylometric authorship attribution aims to identify an anonymous or disputed document’s author by examining its writing style. The development of powerful machine learning based stylometric authorship attribution methods presents a serious privacy threat for individuals such as journalists and activists who wish to publish anonymously. Researchers have proposed several authorship obfuscation approaches that try to make appropriate changes (e.g. word/phrase replacements) to evade attribution while preserving semantics. Unfortunately, existing authorship obfuscation approaches are lacking because they either require some manual effort, require significant training data, or do not work for long documents. To address these limitations, we propose a genetic algorithm based random search framework called Mutant-X which can automatically obfuscate text to successfully evade attribution while keeping the semantics of the obfuscated text similar to the original text. Specifically, Mutant-X sequentially makes changes in the text using mutation and crossover techniques while being guided by a fitness function that takes into account both attribution probability and semantic relevance. While Mutant-X requires black-box knowledge of the adversary’s classifier, it does not require any additional training data and also works on documents of any length. We evaluate Mutant-X against a variety of authorship attribution methods on two different text corpora. Our results show that Mutant-X can decrease the accuracy of state-of-the-art authorship attribution methods by as much as 64% while preserving the semantics much better than existing automated authorship obfuscation approaches. While Mutant-X advances the state-of-the-art in automated authorship obfuscation, we find that it does not generalize to a stronger threat model where the adversary uses a different attribution classifier than what Mutant-X assumes. Our findings warrant the need for future research to improve the generalizability (or transferability) of automated authorship obfuscation approaches.

scholarly journals Discriminative non-negative representation based classifier for image recognition

2021 ◽  
Vol 15 ◽  
pp. 174830262110449
Author(s):  
Kai-Jun Hu ◽  
He-Feng Yin ◽  
Jun Sun
Keyword(s):  
Pattern Classification ◽  
State Of The Art ◽  
Source Code ◽  
Classification Performance ◽  
Training Data ◽  
Practical Applications ◽  
The Past ◽  
Benchmark Datasets ◽  
Classification Tasks ◽  
Negative Representation

During the past decade, representation based classification method has received considerable attention in the community of pattern recognition. The recently proposed non-negative representation based classifier achieved superb recognition results in diverse pattern classification tasks. Unfortunately, discriminative information of training data is not fully exploited in non-negative representation based classifier, which undermines its classification performance in practical applications. To address this problem, we introduce a decorrelation regularizer into the formulation of non-negative representation based classifier and propose a discriminative non-negative representation based classifier for pattern classification. The decorrelation regularizer is able to reduce the correlation of representation results of different classes, thus promoting the competition among them. Experimental results on benchmark datasets validate the efficacy of the proposed discriminative non-negative representation based classifier, and it can outperform some state-of-the-art deep learning based methods. The source code of our proposed discriminative non-negative representation based classifier is accessible at https://github.com/yinhefeng/DNRC .

scholarly journals Improving Image Captioning with Conditional Generative Adversarial Nets

2019 ◽  
Vol 33 ◽  
pp. 8142-8150 ◽  
Author(s):  
Chen Chen ◽  
Shuai Mu ◽  
Wanpeng Xiao ◽  
Zexiong Ye ◽  
Liesi Wu ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
State Of The Art ◽  
Evaluation Metrics ◽  
Training Method ◽  
Image Captioning ◽  
Decoder Architecture ◽  
Evaluation Problem ◽  
Special Case

In this paper, we propose a novel conditional-generativeadversarial-nets-based image captioning framework as an extension of traditional reinforcement-learning (RL)-based encoder-decoder architecture. To deal with the inconsistent evaluation problem among different objective language metrics, we are motivated to design some “discriminator” networks to automatically and progressively determine whether generated caption is human described or machine generated. Two kinds of discriminator architectures (CNN and RNNbased structures) are introduced since each has its own advantages. The proposed algorithm is generic so that it can enhance any existing RL-based image captioning framework and we show that the conventional RL training method is just a special case of our approach. Empirically, we show consistent improvements over all language evaluation metrics for different state-of-the-art image captioning models. In addition, the well-trained discriminators can also be viewed as objective image captioning evaluators.

scholarly journals Kelantan and Sarawak Malay Dialects: Parallel Dialect Text Collection and Alignment Using Hybrid Distance-Statistical-Based Phrase Alignment Algorithm

2021 ◽  
Vol 12 (3) ◽  
pp. 2163-2171
Author(s):  
Khaw, Jasmina Yen Min Et.al
Keyword(s):  
State Of The Art ◽  
Training Data ◽  
Alignment Algorithm ◽  
Text Corpora ◽  
Multilingual Information Retrieval ◽  
Alignment Algorithms ◽  
Parallel Text ◽  
Hybrid Distance ◽  
Phrase Alignment ◽  
Parallel Texts

Parallel texts corpora are essential resources especially in translation and multilingual information retrieval. However, the publicly available parallel text corpora are limited to certain types and domains.  Besides, Malay dialects are not standardized in term of writing. The existing alignment algorithms that is used to analayze the writing will require a large training data to obtain a good result. The paper describes our methodology in acquiring a parallel text corpus of Standard Malay and Malay dialects, particularly Kelantan Malay and Sarawak Malay. Second, we propose a hybrid of distance-based and statistical-based alignment algorithm to align words and phrases of the parallel text. The proposed approach has a better precision and recall than the state-of-the-art GIZA++. In the paper, the alignment obtained were also compared to find out the lexical similarities and differences between SM and the two dialects.

scholarly journals Spatiotemporal Super-Resolution with Cross-Task Consistency and Its Semi-supervised Extension

2020 ◽  
Author(s):  
Han-Yi Lin ◽  
Pi-Cheng Hsiu ◽  
Tei-Wei Kuo ◽  
Yen-Yu Lin
Keyword(s):  
High Resolution ◽  
State Of The Art ◽  
Source Code ◽  
Super Resolution ◽  
The Other ◽  
Training Data ◽  
Frame Rate ◽  
Stream Network ◽  
Network Training ◽  
Temporal Dimensions

Spatiotemporal super-resolution (SR) aims to upscale both the spatial and temporal dimensions of input videos, and produces videos with higher frame resolutions and rates. It involves two essential sub-tasks: spatial SR and temporal SR. We design a two-stream network for spatiotemporal SR in this work. One stream contains a temporal SR module followed by a spatial SR module, while the other stream has the same two modules in the reverse order. Based on the interchangeability of performing the two sub-tasks, the two network streams are supposed to produce consistent spatiotemporal SR results. Thus, we present a cross-stream consistency to enforce the similarity between the outputs of the two streams. In this way, the training of the two streams is correlated, which allows the two SR modules to share their supervisory signals and improve each other. In addition, the proposed cross-stream consistency does not consume labeled training data and can guide network training in an unsupervised manner. We leverage this property to carry out semi-supervised spatiotemporal SR. It turns out that our method makes the most of training data, and can derive an effective model with few high-resolution and high-frame-rate videos, achieving the state-of-the-art performance. The source code of this work is available at https://hankweb.github.io/STSRwithCrossTask/.

scholarly journals Improving Semi-Supervised Learning for Audio Classification with FixMatch

Electronics ◽  
2021 ◽  
Vol 10 (15) ◽  
pp. 1807
Author(s):  
Sascha Grollmisch ◽  
Estefanía Cano
Keyword(s):  
Neural Networks ◽  
Supervised Learning ◽  
Transfer Learning ◽  
Data Transfer ◽  
State Of The Art ◽  
Training Data ◽  
Audio Classification ◽  
Image Domain ◽  
Full Dataset ◽  
Audio Data

Including unlabeled data in the training process of neural networks using Semi-Supervised Learning (SSL) has shown impressive results in the image domain, where state-of-the-art results were obtained with only a fraction of the labeled data. The commonality between recent SSL methods is that they strongly rely on the augmentation of unannotated data. This is vastly unexplored for audio data. In this work, SSL using the state-of-the-art FixMatch approach is evaluated on three audio classification tasks, including music, industrial sounds, and acoustic scenes. The performance of FixMatch is compared to Convolutional Neural Networks (CNN) trained from scratch, Transfer Learning, and SSL using the Mean Teacher approach. Additionally, a simple yet effective approach for selecting suitable augmentation methods for FixMatch is introduced. FixMatch with the proposed modifications always outperformed Mean Teacher and the CNNs trained from scratch. For the industrial sounds and music datasets, the CNN baseline performance using the full dataset was reached with less than 5% of the initial training data, demonstrating the potential of recent SSL methods for audio data. Transfer Learning outperformed FixMatch only for the most challenging dataset from acoustic scene classification, showing that there is still room for improvement.

scholarly journals Image Restoration by Learning Morphological Opening-Closing Network

2020 ◽  
Vol 4 (1) ◽  
pp. 87-107
Author(s):  
Ranjan Mondal ◽  
Moni Shankar Dey ◽  
Bhabatosh Chanda
Keyword(s):  
Neural Network ◽  
Image Restoration ◽  
State Of The Art ◽  
Source Code ◽  
Back Propagation ◽  
Image Features ◽  
Main Difficulty ◽  
The Right ◽  
Right Order ◽  
Morphological Opening

AbstractMathematical morphology is a powerful tool for image processing tasks. The main difficulty in designing mathematical morphological algorithm is deciding the order of operators/filters and the corresponding structuring elements (SEs). In this work, we develop morphological network composed of alternate sequences of dilation and erosion layers, which depending on learned SEs, may form opening or closing layers. These layers in the right order along with linear combination (of their outputs) are useful in extracting image features and processing them. Structuring elements in the network are learned by back-propagation method guided by minimization of the loss function. Efficacy of the proposed network is established by applying it to two interesting image restoration problems, namely de-raining and de-hazing. Results are comparable to that of many state-of-the-art algorithms for most of the images. It is also worth mentioning that the number of network parameters to handle is much less than that of popular convolutional neural network for similar tasks. The source code can be found here https://github.com/ranjanZ/Mophological-Opening-Closing-Net

scholarly journals Transcription Alignment of Historical Vietnamese Manuscripts without Human-Annotated Learning Samples

2021 ◽  
Vol 11 (11) ◽  
pp. 4894
Author(s):  
Anna Scius-Bertrand ◽  
Michael Jungo ◽  
Beat Wolf ◽  
Andreas Fischer ◽  
Marc Bui
Keyword(s):  
Object Detection ◽  
State Of The Art ◽  
Positive Impact ◽  
Detection System ◽  
Training Data ◽  
Detection Accuracy ◽  
Current State ◽  
Alignment Task ◽  
Scanned Image ◽  
Automatic Transcription

The current state of the art for automatic transcription of historical manuscripts is typically limited by the requirement of human-annotated learning samples, which are are necessary to train specific machine learning models for specific languages and scripts. Transcription alignment is a simpler task that aims to find a correspondence between text in the scanned image and its existing Unicode counterpart, a correspondence which can then be used as training data. The alignment task can be approached with heuristic methods dedicated to certain types of manuscripts, or with weakly trained systems reducing the required amount of annotations. In this article, we propose a novel learning-based alignment method based on fully convolutional object detection that does not require any human annotation at all. Instead, the object detection system is initially trained on synthetic printed pages using a font and then adapted to the real manuscripts by means of self-training. On a dataset of historical Vietnamese handwriting, we demonstrate the feasibility of annotation-free alignment as well as the positive impact of self-training on the character detection accuracy, reaching a detection accuracy of 96.4% with a YOLOv5m model without using any human annotation.

scholarly journals New polyp image classification technique using transfer learning of network-in-network structure in endoscopic images

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Young Jae Kim ◽  
Jang Pyo Bae ◽  
Jun-Won Chung ◽  
Dong Kyun Park ◽  
Kwang Gi Kim ◽  
...  
Keyword(s):  
Colorectal Cancer ◽  
Transfer Learning ◽  
Test Data ◽  
State Of The Art ◽  
Early Stage ◽  
Statistical Significance ◽  
Recall Rate ◽  
Training Data ◽  
Fine Tuning ◽  
Accuracy Evaluation

AbstractWhile colorectal cancer is known to occur in the gastrointestinal tract. It is the third most common form of cancer of 27 major types of cancer in South Korea and worldwide. Colorectal polyps are known to increase the potential of developing colorectal cancer. Detected polyps need to be resected to reduce the risk of developing cancer. This research improved the performance of polyp classification through the fine-tuning of Network-in-Network (NIN) after applying a pre-trained model of the ImageNet database. Random shuffling is performed 20 times on 1000 colonoscopy images. Each set of data are divided into 800 images of training data and 200 images of test data. An accuracy evaluation is performed on 200 images of test data in 20 experiments. Three compared methods were constructed from AlexNet by transferring the weights trained by three different state-of-the-art databases. A normal AlexNet based method without transfer learning was also compared. The accuracy of the proposed method was higher in statistical significance than the accuracy of four other state-of-the-art methods, and showed an 18.9% improvement over the normal AlexNet based method. The area under the curve was approximately 0.930 ± 0.020, and the recall rate was 0.929 ± 0.029. An automatic algorithm can assist endoscopists in identifying polyps that are adenomatous by considering a high recall rate and accuracy. This system can enable the timely resection of polyps at an early stage.

Sign in / Sign up

Export Citation Format

Share Document