scholarly journals Graph Convolutional Networks for Text Classification

2019 ◽  
Vol 33 ◽  
pp. 7370-7377 ◽  
Author(s):  
Liang Yao ◽  
Chengsheng Mao ◽  
Yuan Luo
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Language Processing ◽  
Text Classification ◽  
State Of The Art ◽  
Classical Problem ◽  
Experimental Results ◽  
Training Data ◽  
Convolutional Networks ◽  
Single Text

Text classification is an important and classical problem in natural language processing. There have been a number of studies that applied convolutional neural networks (convolution on regular grid, e.g., sequence) to classification. However, only a limited number of studies have explored the more flexible graph convolutional neural networks (convolution on non-grid, e.g., arbitrary graph) for the task. In this work, we propose to use graph convolutional networks for text classification. We build a single text graph for a corpus based on word co-occurrence and document word relations, then learn a Text Graph Convolutional Network (Text GCN) for the corpus. Our Text GCN is initialized with one-hot representation for word and document, it then jointly learns the embeddings for both words and documents, as supervised by the known class labels for documents. Our experimental results on multiple benchmark datasets demonstrate that a vanilla Text GCN without any external word embeddings or knowledge outperforms state-of-the-art methods for text classification. On the other hand, Text GCN also learns predictive word and document embeddings. In addition, experimental results show that the improvement of Text GCN over state-of-the-art comparison methods become more prominent as we lower the percentage of training data, suggesting the robustness of Text GCN to less training data in text classification.

scholarly journals Combining Knowledge with Deep Convolutional Neural Networks for Short Text Classification

2017 ◽  
Author(s):  
Jin Wang ◽  
Zhongyuan Wang ◽  
Dawei Zhang ◽  
Jun Yan
Keyword(s):  
Neural Networks ◽  
Knowledge Base ◽  
Convolutional Neural Networks ◽  
Text Classification ◽  
State Of The Art ◽  
Experimental Results ◽  
Text Representation ◽  
Deep Convolutional Neural Networks ◽  
Short Text ◽  
Fine Grained

Text classification is a fundamental task in NLP applications. Most existing work relied on either explicit or implicit text representation to address this problem. While these techniques work well for sentences, they can not easily be applied to short text because of its shortness and sparsity. In this paper, we propose a framework based on convolutional neural networks that combines explicit and implicit representations of short text for classification. We first conceptualize a short text as a set of relevant concepts using a large taxonomy knowledge base. We then obtain the embedding of short text by coalescing the words and relevant concepts on top of pre-trained word vectors. We further incorporate character level features into our model to capture fine-grained subword information. Experimental results on five commonly used datasets show that our proposed method significantly outperforms state-of-the-art methods.

scholarly journals Sentiment Classification Using Convolutional Neural Networks

2019 ◽  
Vol 9 (11) ◽  
pp. 2347 ◽  
Author(s):  
Hannah Kim ◽  
Young-Seob Jeong
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Text Classification ◽  
State Of The Art ◽  
Sentiment Classification ◽  
Learning Models ◽  
Text Data ◽  
Textual Data ◽  
Better Than

As the number of textual data is exponentially increasing, it becomes more important to develop models to analyze the text data automatically. The texts may contain various labels such as gender, age, country, sentiment, and so forth. Using such labels may bring benefits to some industrial fields, so many studies of text classification have appeared. Recently, the Convolutional Neural Network (CNN) has been adopted for the task of text classification and has shown quite successful results. In this paper, we propose convolutional neural networks for the task of sentiment classification. Through experiments with three well-known datasets, we show that employing consecutive convolutional layers is effective for relatively longer texts, and our networks are better than other state-of-the-art deep learning models.

scholarly journals Convolutional Neural Networks Inference Memory Optimization with Receptive Field-Based InputTiling

2021 ◽  
Author(s):  
Weihao Zhuang ◽  
Tristan Hascoet ◽  
Xunquan Chen ◽  
Ryoichi Takashima ◽  
Tetsuya Takiguchi ◽  
...  
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Convolutional Neural Networks ◽  
Language Processing ◽  
State Of The Art ◽  
Input Image ◽  
Memory Consumption ◽  
Excellent Performance ◽  
Conceptual Approach ◽  
Recent Developments

Abstract Currently, deep learning plays an indispensable role in many fields, including computer vision, natural language processing, and speech recognition. Convolutional Neural Networks (CNNs) have demonstrated excellent performance in computer vision tasks thanks to their powerful feature extraction capability. However, as the larger models have shown higher accuracy, recent developments have led to state-of-the-art CNN models with increasing resource consumption. This paper investigates a conceptual approach to reduce the memory consumption of CNN inference. Our method consists of processing the input image in a sequence of carefully designed tiles within the lower subnetwork of the CNN, so as to minimize its peak memory consumption, while keeping the end-to-end computation unchanged. This method introduces a trade-off between memory consumption and computations, which is particularly suitable for high-resolution inputs. Our experimental results show that MobileNetV2 memory consumption can be reduced by up to 5.3 times with our proposed method. For ResNet50, one of the most commonly used CNN models in computer vision tasks, memory can be optimized by up to 2.3 times.

scholarly journals A Study of The Convolutional Neural Networks Applications

2019 ◽  
Vol 3 (2) ◽  
pp. 31-40 ◽  
Author(s):  
Ahmed Shamsaldin ◽  
Polla Fattah ◽  
Tarik Rashid ◽  
Nawzad Al-Salihi
Keyword(s):  
Neural Networks ◽  
Computer Vision ◽  
Deep Learning ◽  
Natural Language Processing ◽  
Face Recognition ◽  
Convolutional Neural Networks ◽  
Language Processing ◽  
Text Classification ◽  
Scene Labeling ◽  
Real World Problems

At present, deep learning is widely used in a broad range of arenas. A convolutional neural networks (CNN) is becoming the star of deep learning as it gives the best and most precise results when cracking real-world problems. In this work, a brief description of the applications of CNNs in two areas will be presented: First, in computer vision, generally, that is, scene labeling, face recognition, action recognition, and image classification; Second, in natural language processing, that is, the fields of speech recognition and text classification.

scholarly journals Grapheme-to-Phoneme Conversion with Convolutional Neural Networks

2019 ◽  
Vol 9 (6) ◽  
pp. 1143 ◽  
Author(s):  
Sevinj Yolchuyeva ◽  
Géza Németh ◽  
Bálint Gyires-Tóth
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Network ◽  
Convolutional Neural Networks ◽  
Language Processing ◽  
Speech Synthesis ◽  
State Of The Art ◽  
Error Rates ◽  
Written Form ◽  
G2p Conversion

Grapheme-to-phoneme (G2P) conversion is the process of generating pronunciation for words based on their written form. It has a highly essential role for natural language processing, text-to-speech synthesis and automatic speech recognition systems. In this paper, we investigate convolutional neural networks (CNN) for G2P conversion. We propose a novel CNN-based sequence-to-sequence (seq2seq) architecture for G2P conversion. Our approach includes an end-to-end CNN G2P conversion with residual connections and, furthermore, a model that utilizes a convolutional neural network (with and without residual connections) as encoder and Bi-LSTM as a decoder. We compare our approach with state-of-the-art methods, including Encoder-Decoder LSTM and Encoder-Decoder Bi-LSTM. Training and inference times, phoneme and word error rates were evaluated on the public CMUDict dataset for US English, and the best performing convolutional neural network-based architecture was also evaluated on the NetTalk dataset. Our method approaches the accuracy of previous state-of-the-art results in terms of phoneme error rate.

Sanitizing hidden activations for improving adversarial robustness of convolutional neural networks

2021 ◽  
pp. 1-11
Author(s):  
Tianshi Mu ◽  
Kequan Lin ◽  
Huabing Zhang ◽  
Jian Wang
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
State Of The Art ◽  
Black Box ◽  
Experimental Results ◽  
Amplification Effect ◽  
Wide Range ◽  
Adversarial Examples

Deep learning is gaining significant traction in a wide range of areas. Whereas, recent studies have demonstrated that deep learning exhibits the fatal weakness on adversarial examples. Due to the black-box nature and un-transparency problem of deep learning, it is difficult to explain the reason for the existence of adversarial examples and also hard to defend against them. This study focuses on improving the adversarial robustness of convolutional neural networks. We first explore how adversarial examples behave inside the network through visualization. We find that adversarial examples produce perturbations in hidden activations, which forms an amplification effect to fool the network. Motivated by this observation, we propose an approach, termed as sanitizing hidden activations, to help the network correctly recognize adversarial examples by eliminating or reducing the perturbations in hidden activations. To demonstrate the effectiveness of our approach, we conduct experiments on three widely used datasets: MNIST, CIFAR-10 and ImageNet, and also compare with state-of-the-art defense techniques. The experimental results show that our sanitizing approach is more generalized to defend against different kinds of attacks and can effectively improve the adversarial robustness of convolutional neural networks.

scholarly journals AAR-CNNs: Auto Adaptive Regularized Convolutional Neural Networks

2018 ◽  
Author(s):  
Yao Lu ◽  
Guangming Lu ◽  
Yuanrong Xu ◽  
Bob Zhang
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
State Of The Art ◽  
Experimental Results ◽  
Training Phase ◽  
Low Resolution ◽  
Adaptive Regularization ◽  
End To End ◽  
Overfitting Problem

In order to address the overfitting problem caused by the small or simple training datasets and the large model’s size in Convolutional Neural Networks (CNNs), a novel Auto Adaptive Regularization (AAR) method is proposed in this paper. The relevant networks can be called AAR-CNNs. AAR is the first method using the “abstraction extent” (predicted by AE net) and a tiny learnable module (SE net) to auto adaptively predict more accurate and individualized regularization information. The AAR module can be directly inserted into every stage of any popular networks and trained end to end to improve the networks’ flexibility. This method can not only regularize the network at both the forward and the backward processes in the training phase, but also regularize the network on a more refined level (channel or pixel level) depending on the abstraction extent’s form. Comparative experiments are performed on low resolution ImageNet, CIFAR and SVHN datasets. Experimental results show that the AAR-CNNs can achieve state-of-the-art performances on these datasets.

scholarly journals Informing Piano Multi-Pitch Estimation with Inferred Local Polyphony Based on Convolutional Neural Networks

Electronics ◽  
2021 ◽  
Vol 10 (7) ◽  
pp. 851
Author(s):  
Michael Taenzer ◽  
Stylianos I. Mimilakis ◽  
Jakob Abeßer
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
State Of The Art ◽  
Piano Music ◽  
Experimental Results ◽  
Pitch Estimation ◽  
Feature Representations ◽  
Series Of Experiments ◽  
Music Recordings ◽  
Recent Extension

In this work, we propose considering the information from a polyphony for multi-pitch estimation (MPE) in piano music recordings. To that aim, we propose a method for local polyphony estimation (LPE), which is based on convolutional neural networks (CNNs) trained in a supervised fashion to explicitly predict the degree of polyphony. We investigate two feature representations as inputs to our method, in particular, the Constant-Q Transform (CQT) and its recent extension Folded-CQT (F-CQT). To evaluate the performance of our method, we conduct a series of experiments on real and synthetic piano recordings based on the MIDI Aligned Piano Sounds (MAPS) and the Saarland Music Data (SMD) datasets. We compare our approaches with a state-of-the art piano transcription method by informing said method with the LPE knowledge in a postprocessing stage. The experimental results suggest that using explicit LPE information can refine MPE predictions. Furthermore, it is shown that, on average, the CQT representation is preferred over F-CQT for LPE.

Common Kernels and Convolutions in Binary- and Ternary-Weight Neural Networks

2020 ◽  
pp. 2150158
Author(s):  
Byungmin Ahn ◽  
Taewhan Kim
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Optimization Algorithm ◽  
State Of The Art ◽  
The State ◽  
Memory Access ◽  
Experimental Results ◽  
Hardware Platform ◽  
Access Latency ◽  
The Common

A new algorithm for extracting common kernels and convolutions to maximally eliminate the redundant operations among the convolutions in binary- and ternary-weight convolutional neural networks is presented. Precisely, we propose (1) a new algorithm of common kernel extraction to overcome the local and limited exploration of common kernel candidates by the existing method, and subsequently apply (2) a new concept of common convolution extraction to maximally eliminate the redundancy in the convolution operations. In addition, our algorithm is able to (3) tune in minimizing the number of resulting kernels for convolutions, thereby saving the total memory access latency for kernels. Experimental results on ternary-weight VGG-16 demonstrate that our convolution optimization algorithm is very effective, reducing the total number of operations for all convolutions by [Formula: see text], thereby reducing the total number of execution cycles on hardware platform by 22.4% while using [Formula: see text] fewer kernels over that of the convolution utilizing the common kernels extracted by the state-of-the-art algorithm.

Recurrent convolutional neural networks for poet identification

2020 ◽  
Author(s):  
Dariush Salami ◽  
Saeedeh Momtazi
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Language Processing ◽  
Short Term Memory ◽  
State Of The Art ◽  
The State ◽  
Support Vector ◽  
Neural Network Models ◽  
Spatial Features ◽  
Input Text

Abstract Deep neural networks have been widely used in various language processing tasks. Recurrent neural networks (RNNs) and convolutional neural networks (CNN) are two common types of neural networks that have a successful history in capturing temporal and spatial features of texts. By using RNN, we can encode input text to a lower space of semantic features while considering the sequential behavior of words. By using CNN, we can transfer the representation of input text to a flat structure to be used for classifying text. In this article, we proposed a novel recurrent CNN model to capture not only the temporal but also the spatial features of the input poem/verse to be used for poet identification. Considering the shortcomings of the normal RNNs, we try both long short-term memory and gated recurrent unit units in the proposed architecture and apply them to the poet identification task. There are a large number of poems in the history of literature whose poets are unknown. Considering the importance of the task in the information processing field, a great variety of methods from traditional learning models, such as support vector machine and logistic regression, to deep neural network models, such as CNN, have been proposed to address this problem. Our experiments show that the proposed model significantly outperforms the state-of-the-art models for poet identification by receiving either a poem or a single verse as input. In comparison to the state-of-the-art CNN model, we achieved 9% and 4% improvements in f-measure for poem- and verse-based tasks, respectively.

Sign in / Sign up

Export Citation Format

Share Document