Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification

Graph neural network (GNN) and label propagation algorithm (LPA) are both message passing algorithms, which have achieved superior performance in semi-supervised classification. GNN performs feature propagation by a neural network to make predictions, while LPA uses label propagation across graph adjacency matrix to get results. However, there is still no effective way to directly combine these two kinds of algorithms. To address this issue, we propose a novel Unified Message Passaging Model (UniMP) that can incorporate feature and label propagation at both training and inference time. First, UniMP adopts a Graph Transformer network, taking feature embedding and label embedding as input information for propagation. Second, to train the network without overfitting in self-loop input label information, UniMP introduces a masked label prediction strategy, in which some percentage of input label information are masked at random, and then predicted. UniMP conceptually unifies feature propagation and label propagation and is empirically powerful. It obtains new state-of-the-art semi-supervised classification results in Open Graph Benchmark (OGB).

Download Full-text

Combining Graph Convolutional Neural Networks and Label Propagation

ACM Transactions on Information Systems ◽

10.1145/3490478 ◽

2022 ◽

Vol 40 (4) ◽

pp. 1-27

Author(s):

Hongwei Wang ◽

Jure Leskovec

Keyword(s):

Neural Networks ◽

Convolutional Neural Networks ◽

Message Passing ◽

Label Propagation ◽

Unified Framework ◽

Label Information ◽

Message Passing Algorithms ◽

Node Classification ◽

Improved Performance ◽

Node Labels

Label Propagation Algorithm (LPA) and Graph Convolutional Neural Networks (GCN) are both message passing algorithms on graphs. Both solve the task of node classification, but LPA propagates node label information across the edges of the graph, while GCN propagates and transforms node feature information. However, while conceptually similar, theoretical relationship between LPA and GCN has not yet been systematically investigated. Moreover, it is unclear how LPA and GCN can be combined under a unified framework to improve the performance. Here we study the relationship between LPA and GCN in terms of feature/label influence , in which we characterize how much the initial feature/label of one node influences the final feature/label of another node in GCN/LPA. Based on our theoretical analysis, we propose an end-to-end model that combines GCN and LPA. In our unified model, edge weights are learnable, and the LPA serves as regularization to assist the GCN in learning proper edge weights that lead to improved performance. Our model can also be seen as learning the weights of edges based on node labels, which is more direct and efficient than existing feature-based attention models or topology-based diffusion models. In a number of experiments for semi-supervised node classification and knowledge-graph-aware recommendation, our model shows superiority over state-of-the-art baselines.

Download Full-text

An Analysis of State-of-the-art Activation Functions For Supervised Deep Neural Network

10.31219/osf.io/2zk6a ◽

2021 ◽

Author(s):

Anh Nguyen ◽

Khoa Pham ◽

Dat Ngo ◽

Thanh Ngo ◽

Lam Pham

Keyword(s):

Neural Network ◽

Supervised Classification ◽

Deep Neural Network ◽

State Of The Art ◽

Network Architectures ◽

Activation Functions ◽

Scene Classification ◽

Learning Network ◽

Deep Learning Network

This paper provides an analysis of state-of-the-art activation functions with respect to supervised classification of deep neural network. These activation functions comprise of Rectified Linear Units (ReLU), Exponential Linear Unit (ELU), Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit (GELU), and the Inverse Square Root Linear Unit (ISRLU). To evaluate, experiments over two deep learning network architectures integrating these activation functions are conducted. The first model, basing on Multilayer Perceptron (MLP), is evaluated with MNIST dataset to perform these activation functions.Meanwhile, the second model, likely VGGish-based architecture, is applied for Acoustic Scene Classification (ASC) Task 1A in DCASE 2018 challenge, thus evaluate whether these activation functions work well in different datasets as well as different network architectures.

Download Full-text

WTRPNet: An Explainable Graph Feature Convolutional Neural Network for Epileptic EEG Classification

ACM Transactions on Multimedia Computing Communications and Applications ◽

10.1145/3460522 ◽

2021 ◽

Vol 17 (3s) ◽

pp. 1-18

Author(s):

Qi Xin ◽

Shaohao Hu ◽

Shuaiqi Liu ◽

Ling Zhao ◽

Shuihua Wang

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

State Of The Art ◽

Traumatic Injury ◽

Recurrence Plot ◽

Superior Performance ◽

Eeg Classification ◽

Epilepsy Diagnosis ◽

Electroencephalogram Eeg

As one of the important tools of epilepsy diagnosis, the electroencephalogram (EEG) is noninvasive and presents no traumatic injury to patients. It contains a lot of physiological and pathological information that is easy to obtain. The automatic classification of epileptic EEG is important in the diagnosis and therapeutic efficacy of epileptics. In this article, an explainable graph feature convolutional neural network named WTRPNet is proposed for epileptic EEG classification. Since WTRPNet is constructed by a recurrence plot in the wavelet domain, it can fully obtain the graph feature of the EEG signal, which is established by an explainable graph features extracted layer called WTRP block . The proposed method shows superior performance over state-of-the-art methods. Experimental results show that our algorithm has achieved an accuracy of 99.67% in classification of focal and nonfocal epileptic EEG, which proves the effectiveness of the classification and detection of epileptic EEG.

Download Full-text

Robust place recognition based on salient landmarks screening and convolutional neural network features

International Journal of Advanced Robotic Systems ◽

10.1177/1729881420966966 ◽

2020 ◽

Vol 17 (6) ◽

pp. 172988142096696

Author(s):

Jie Niu ◽

Kun Qian

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

State Of The Art ◽

Environmental Modeling ◽

Superior Performance ◽

Similarity Measurement ◽

Natural Environments ◽

Place Recognition ◽

Average Precision ◽

Specific Object

In this work, we propose a robust place recognition measurement in natural environments based on salient landmark screening and convolutional neural network (CNN) features. First, the salient objects in the image are segmented as candidate landmarks. Then, a category screening network is designed to remove specific object types that are not suitable for environmental modeling. Finally, a three-layer CNN is used to get highly representative features of the salient landmarks. In the similarity measurement, a Siamese network is chosen to calculate the similarity between images. Experiments were conducted on three challenging benchmark place recognition datasets and superior performance was achieved compared to other state-of-the-art methods, including FABMAP, SeqSLAM, SeqCNNSLAM, and PlaceCNN. Our method obtains the best results on the precision–recall curves, and the average precision reaches 78.43%, which is the best of the comparison methods. This demonstrates that the CNN features on the screened salient landmarks can be against a strong viewpoint and condition variations.

Download Full-text

CAGAN: Consistent Adversarial Training Enhanced GANs

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2018/359 ◽

2018 ◽

Cited By ~ 1

Author(s):

Yao Ni ◽

Dandan Song ◽

Xi Zhang ◽

Hao Wu ◽

Lejian Liao

Keyword(s):

Neural Network ◽

Parameter Space ◽

Supervised Classification ◽

State Of The Art ◽

Generative Adversarial Networks ◽

Image Generation ◽

Real Samples ◽

Adversarial Networks ◽

Novel Approach ◽

Adversarial Training

Generative adversarial networks (GANs) have shown impressive results, however, the generator and the discriminator are optimized in finite parameter space which means their performance still need to be improved. In this paper, we propose a novel approach of adversarial training between one generator and an exponential number of critics which are sampled from the original discriminative neural network via dropout. As discrepancy between outputs of different sub-networks of a same sample can measure the consistency of these critics, we encourage the critics to be consistent to real samples and inconsistent to generated samples during training, while the generator is trained to generate consistent samples for different critics. Experimental results demonstrate that our method can obtain state-of-the-art Inception scores of 9.17 and 10.02 on supervised CIFAR-10 and unsupervised STL-10 image generation tasks, respectively, as well as achieve competitive semi-supervised classification results on several benchmarks. Importantly, we demonstrate that our method can maintain stability in training and alleviate mode collapse.

Download Full-text

Exploiting Heterogeneous Graph Neural Networks with Latent Worker/Task Correlation Information for Label Aggregation in Crowdsourcing

ACM Transactions on Knowledge Discovery from Data ◽

10.1145/3460865 ◽

2022 ◽

Vol 16 (2) ◽

pp. 1-18

Author(s):

Hanlu Wu ◽

Tengfei Ma ◽

Lingfei Wu ◽

Fangli Xu ◽

Shouling Ji

Keyword(s):

Neural Network ◽

Neural Networks ◽

State Of The Art ◽

Superior Performance ◽

True Label ◽

Label Aggregation ◽

Correlation Information ◽

Real World Datasets ◽

Graph Neural Networks ◽

High Level

Crowdsourcing has attracted much attention for its convenience to collect labels from non-expert workers instead of experts. However, due to the high level of noise from the non-experts, a label aggregation model that infers the true label from noisy crowdsourced labels is required. In this article, we propose a novel framework based on graph neural networks for aggregating crowd labels. We construct a heterogeneous graph between workers and tasks and derive a new graph neural network to learn the representations of nodes and the true labels. Besides, we exploit the unknown latent interaction between the same type of nodes (workers or tasks) by adding a homogeneous attention layer in the graph neural networks. Experimental results on 13 real-world datasets show superior performance over state-of-the-art models.

Download Full-text

A Robust and Scalable Graph Neural Network for Accurate Single Cell Classification

10.1101/2021.06.24.449752 ◽

2021 ◽

Author(s):

Yuansong Zeng ◽

Xiang Zhou ◽

Zixiang Pan ◽

Yutong Lu ◽

Yuedong Yang

Keyword(s):

Neural Network ◽

Single Cell ◽

Message Passing ◽

High Speed ◽

Large Scale ◽

Cellular Heterogeneity ◽

Superior Performance ◽

Marker Genes ◽

Cell Classification ◽

High Resolution Data

Single-cell RNA sequencing (scRNA-seq) techniques provide high-resolution data on cellular heterogeneity in diverse tissues, and a critical step for the data analysis is cell type identification. Traditional methods usually cluster the cells and manually identify cell clusters through marker genes, which is time-consuming and subjective. With the launch of several large-scale single-cell projects, millions of sequenced cells have been annotated and it is promising to transfer labels from the annotated datasets to newly generated datasets. One powerful way for the transferring is to learn cell relations through the graph neural network (GNN), while vanilla GNN is difficult to process millions of cells due to the expensive costs of the message-passing procedure at each training epoch. Here, we have developed a robust and scalable GNN-based method for accurate single cell classification (GraphCS), where the graph is constructed to connect similar cells within and between labelled and unlabelled scRNA-seq datasets for propagation of shared information. To overcome the slow information propagation of GNN at each training epoch, the diffused information is pre-calculated via the approximate Generalized PageRank algorithm, enabling sublinear complexity for a high speed and scalability on millions of cells. Compared with existing methods, GraphCS demonstrates better performance on simulated, cross-platform, and cross-species scRNA-seq datasets. More importantly, our model can achieve superior performance on a large dataset with one million cells within 50 minutes.

Download Full-text

An Empirical Study on Deep Neural Network Models for Chinese Dialogue Generation

Symmetry ◽

10.3390/sym12111756 ◽

2020 ◽

Vol 12 (11) ◽

pp. 1756

Author(s):

Zhe Li ◽

Mieradilijiang Maimaiti ◽

Jiabao Sheng ◽

Zunwang Ke ◽

Wushour Silamu ◽

...

Keyword(s):

Neural Network ◽

Empirical Study ◽

Deep Neural Network ◽

Question Answering ◽

State Of The Art ◽

Network Models ◽

Superior Performance ◽

Research Progress ◽

Neural Network Models ◽

Wide Range

The task of dialogue generation has attracted increasing attention due to its diverse downstream applications, such as question-answering systems and chatbots. Recently, the deep neural network (DNN)-based dialogue generation models have achieved superior performance against conventional models utilizing statistical machine learning methods. However, despite that an enormous number of state-of-the-art DNN-based models have been proposed, there lacks detailed empirical comparative analysis for them on the open Chinese corpus. As a result, relevant researchers and engineers might find it hard to get an intuitive understanding of the current research progress. To address this challenge, we conducted an empirical study for state-of-the-art DNN-based dialogue generation models in various Chinese corpora. Specifically, extensive experiments were performed on several well-known single-turn and multi-turn dialogue corpora, including KdConv, Weibo, and Douban, to evaluate a wide range of dialogue generation models that are based on the symmetrical architecture of Seq2Seq, RNNSearch, transformer, generative adversarial nets, and reinforcement learning respectively. Moreover, we paid special attention to the prevalent pre-trained model for the quality of dialogue generation. Their performances were evaluated by four widely-used metrics in this area: BLEU, pseudo, distinct, and rouge. Finally, we report a case study to show example responses generated by these models separately.

Download Full-text

Ensemble Neural Relation Extraction with Adaptive Boosting

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2018/630 ◽

2018 ◽

Cited By ~ 6

Author(s):

Dongdong Yang ◽

Senzhang Wang ◽

Zhoujun Li

Keyword(s):

Neural Network ◽

State Of The Art ◽

Noisy Data ◽

Relation Extraction ◽

Extraction Methods ◽

Superior Performance ◽

Semantic Meaning ◽

Adaptive Boosting ◽

Proposed Model ◽

Ensemble Neural Networks

Relation extraction has been widely studied to extract new relational facts from open corpus. Previous relation extraction methods are faced with the problem of wrong labels and noisy data, which substantially decrease the performance of the model. In this paper, we propose an ensemble neural network model - Adaptive Boosting LSTMs with Attention, to more effectively perform relation extraction. Specifically, our model first employs the recursive neural network LSTMs to embed each sentence. Then we import attention into LSTMs by considering that the words in a sentence do not contribute equally to the semantic meaning of the sentence. Next via adaptive boosting, we build strategically several such neural classifiers. By ensembling multiple such LSTM classifiers with adaptive boosting, we could build a more effective and robust joint ensemble neural networks based relation extractor. Experiment results on real dataset demonstrate the superior performance of the proposed model, improving F1-score by about 8% compared to the state-of-the-art models.

Download Full-text

MILDMS: Multiple Instance Learning via DD Constraint and Multiple Part Similarity

Symmetry ◽

10.3390/sym11091080 ◽

2019 ◽

Vol 11 (9) ◽

pp. 1080

Author(s):

Chao Wen ◽

Zhan Li ◽

Jian Qu ◽

Qingchen Fan ◽

Aiping Li

Keyword(s):

Neural Network ◽

State Of The Art ◽

Subject Area ◽

Multiple Instance Learning ◽

Combination Method ◽

Learning Problem ◽

Label Information ◽

Weakly Supervised ◽

Instance Space ◽

Diverse Density

As a subject area of symmetry, multiple instance learning (MIL) is a special form of a weakly supervised learning problem where the label is related to the bag, not the instances contained in it. The difficulty of MIL lies in the incomplete label information of instances. To resolve this problem, in this paper, we propose a novel diverse density (DD) and multiple part similarity combination method for multiple instance learning, named MILDMS. First, we model the target concepts optimization with a DD function constraint on positive and negative instance space, which can greatly improve the robustness to label noise problem. Next, we combine the positive and negative instances in the bag (generated by hand-crafted and convolutional neural network features) with multiple part similarities to construct an MIL kernel. We evaluate the proposed approach on the MUSK dataset, whose results MUSK1 (91.9%) and MUSK2 (92.2%) show our method is comparable to other MIL algorithms. To further demonstrate generality, we also present experimental results on the PASCAL VOC 2007 and 2012 (46.5% and 42.2%) and COREL (78.6%) that significantly outperforms the state-of-the-art algorithms including deep MIL and other non-deep MIL algorithms.

Download Full-text