scholarly journals Compressing deep graph convolution network with multi-staged knowledge distillation

PLoS ONE ◽  
2021 ◽  
Vol 16 (8) ◽  
pp. e0256187
Author(s):  
Junghun Kim ◽  
Jinhong Jung ◽  
U. Kang
Keyword(s):  
Embedded Systems ◽  
Real World ◽  
State Of The Art ◽  
Significant Loss ◽  
Second Best ◽  
Novel Approach ◽  
Feature Aggregation ◽  
Knowledge Distillation ◽  
Real World Datasets ◽  
Effective Layer

Given a trained deep graph convolution network (GCN), how can we effectively compress it into a compact network without significant loss of accuracy? Compressing a trained deep GCN into a compact GCN is of great importance for implementing the model to environments such as mobile or embedded systems, which have limited computing resources. However, previous works for compressing deep GCNs do not consider the multi-hop aggregation of the deep GCNs, though it is the main purpose for their multiple GCN layers. In this work, we propose MustaD (Multi-staged knowledge Distillation), a novel approach for compressing deep GCNs to single-layered GCNs through multi-staged knowledge distillation (KD). MustaD distills the knowledge of 1) the aggregation from multiple GCN layers as well as 2) task prediction while preserving the multi-hop feature aggregation of deep GCNs by a single effective layer. Extensive experiments on four real-world datasets show that MustaD provides the state-of-the-art performance compared to other KD based methods. Specifically, MustaD presents up to 4.21%p improvement of accuracy compared to the second-best KD models.

scholarly journals Density Guarantee on Finding Multiple Subgraphs and Subtensors

2021 ◽  
Vol 15 (5) ◽  
pp. 1-32
Author(s):  
Quang-huy Duong ◽  
Heri Ramampiaro ◽  
Kjetil Nørvåg ◽  
Thu-lan Dam
Keyword(s):  
Lower Bound ◽  
State Of The Art ◽  
The State ◽  
The Other ◽  
Exact Methods ◽  
Practical Solution ◽  
Novel Approach ◽  
Wide Range ◽  
Real World Datasets ◽  
Tensor Data

Dense subregion (subgraph & subtensor) detection is a well-studied area, with a wide range of applications, and numerous efficient approaches and algorithms have been proposed. Approximation approaches are commonly used for detecting dense subregions due to the complexity of the exact methods. Existing algorithms are generally efficient for dense subtensor and subgraph detection, and can perform well in many applications. However, most of the existing works utilize the state-or-the-art greedy 2-approximation algorithm to capably provide solutions with a loose theoretical density guarantee. The main drawback of most of these algorithms is that they can estimate only one subtensor, or subgraph, at a time, with a low guarantee on its density. While some methods can, on the other hand, estimate multiple subtensors, they can give a guarantee on the density with respect to the input tensor for the first estimated subsensor only. We address these drawbacks by providing both theoretical and practical solution for estimating multiple dense subtensors in tensor data and giving a higher lower bound of the density. In particular, we guarantee and prove a higher bound of the lower-bound density of the estimated subgraph and subtensors. We also propose a novel approach to show that there are multiple dense subtensors with a guarantee on its density that is greater than the lower bound used in the state-of-the-art algorithms. We evaluate our approach with extensive experiments on several real-world datasets, which demonstrates its efficiency and feasibility.

scholarly journals Robust CNN Compression Framework for Security-Sensitive Embedded Systems

2021 ◽  
Vol 11 (3) ◽  
pp. 1093
Author(s):  
Jeonghyun Lee ◽  
Sangkyun Lee
Keyword(s):  
Embedded Systems ◽  
Optimization Problem ◽  
State Of The Art ◽  
Classification Problems ◽  
Proximal Gradient Method ◽  
Knowledge Distillation ◽  
New Type ◽  
Adversarial Examples ◽  
Adversarial Training ◽  
Memory Efficient

Convolutional neural networks (CNNs) have achieved tremendous success in solving complex classification problems. Motivated by this success, there have been proposed various compression methods for downsizing the CNNs to deploy them on resource-constrained embedded systems. However, a new type of vulnerability of compressed CNNs known as the adversarial examples has been discovered recently, which is critical for security-sensitive systems because the adversarial examples can cause malfunction of CNNs and can be crafted easily in many cases. In this paper, we proposed a compression framework to produce compressed CNNs robust against such adversarial examples. To achieve the goal, our framework uses both pruning and knowledge distillation with adversarial training. We formulate our framework as an optimization problem and provide a solution algorithm based on the proximal gradient method, which is more memory-efficient than the popular ADMM-based compression approaches. In experiments, we show that our framework can improve the trade-off between adversarial robustness and compression rate compared to the existing state-of-the-art adversarial pruning approach.

scholarly journals Efficient Heterogeneous Collaborative Filtering without Negative Sampling for Recommendation

2020 ◽  
Vol 34 (01) ◽  
pp. 19-26 ◽  
Author(s):  
Chong Chen ◽  
Min Zhang ◽  
Yongfeng Zhang ◽  
Weizhi Ma ◽  
Yiqun Liu ◽  
...  
Keyword(s):  
Collaborative Filtering ◽  
Real World ◽  
Large Scale ◽  
State Of The Art ◽  
Heterogeneous Data ◽  
Model Parameters ◽  
Online Systems ◽  
Practical Applications ◽  
Real World Datasets ◽  
Primary Type

Recent studies on recommendation have largely focused on exploring state-of-the-art neural networks to improve the expressiveness of models, while typically apply the Negative Sampling (NS) strategy for efficient learning. Despite effectiveness, two important issues have not been well-considered in existing methods: 1) NS suffers from dramatic fluctuation, making sampling-based methods difficult to achieve the optimal ranking performance in practical applications; 2) although heterogeneous feedback (e.g., view, click, and purchase) is widespread in many online systems, most existing methods leverage only one primary type of user feedback such as purchase. In this work, we propose a novel non-sampling transfer learning solution, named Efficient Heterogeneous Collaborative Filtering (EHCF) for Top-N recommendation. It can not only model fine-grained user-item relations, but also efficiently learn model parameters from the whole heterogeneous data (including all unlabeled data) with a rather low time complexity. Extensive experiments on three real-world datasets show that EHCF significantly outperforms state-of-the-art recommendation methods in both traditional (single-behavior) and heterogeneous scenarios. Moreover, EHCF shows significant improvements in training efficiency, making it more applicable to real-world large-scale systems. Our implementation has been released 1 to facilitate further developments on efficient whole-data based neural methods.

scholarly journals Which Attributes Matter the Most for Loan Origination? A Neural Attention Approach

2020 ◽  
Author(s):  
Antonios Alexos ◽  
Sotirios Chatzis
Keyword(s):  
Deep Learning ◽  
Real World ◽  
Individual Case ◽  
Learning Model ◽  
Attention Mechanism ◽  
Novel Approach ◽  
Specific Individual ◽  
Real World Datasets ◽  
Deep Learning Model

In this paper we address the understanding of the problem, of why a deep learning model decides that an individual is eligible for a loan or not. Here we propose a novel approach for inferring, which attributes matter the most, for making a decision in each specific individual case. Specifically we leverage concepts from neural attention to devise a novel feature wise attention mechanism. As we show, using real world datasets, our approach offers unique insights into the importance of various features, by producing a decision explanation for each specific loan case. At the same time, we observe that our novel mechanism, generates decisions which are much closer to the decisions generated by human experts, compared to the existent competitors.

scholarly journals Discrete Trust-aware Matrix Factorization for Fast Recommendation

2019 ◽  
Author(s):  
Guibing Guo ◽  
Enneng Yang ◽  
Li Shen ◽  
Xiaochun Yang ◽  
Xiaodong He
Keyword(s):  
Social Influence ◽  
Collaborative Filtering ◽  
Recommender Systems ◽  
Social Relations ◽  
Real World ◽  
Matrix Factorization ◽  
State Of The Art ◽  
Proposed Model ◽  
Hamming Space ◽  
Real World Datasets

Trust-aware recommender systems have received much attention recently for their abilities to capture the influence among connected users. However, they suffer from the efficiency issue due to large amount of data and time-consuming real-valued operations. Although existing discrete collaborative filtering may alleviate this issue to some extent, it is unable to accommodate social influence. In this paper we propose a discrete trust-aware matrix factorization (DTMF) model to take dual advantages of both social relations and discrete technique for fast recommendation. Specifically, we map the latent representation of users and items into a joint hamming space by recovering the rating and trust interactions between users and items. We adopt a sophisticated discrete coordinate descent (DCD) approach to optimize our proposed model. In addition, experiments on two real-world datasets demonstrate the superiority of our approach against other state-of-the-art approaches in terms of ranking accuracy and efficiency.

scholarly journals EA Reader: Enhance Attentive Reader for Cloze-Style Question Answering via Multi-Space Context Fusion

2019 ◽  
Vol 33 ◽  
pp. 6375-6382
Author(s):  
Chengzhen Fu ◽  
Yan Zhang
Keyword(s):  
Real World ◽  
Question Answering ◽  
State Of The Art ◽  
Unified Model ◽  
Inference Process ◽  
Context Vector ◽  
Attentive Reader ◽  
Semantic Spaces ◽  
Real World Datasets

Query-document semantic interactions are essential for the success of many cloze-style question answering models. Recently, researchers have proposed several attention-based methods to predict the answer by focusing on appropriate subparts of the context document. In this paper, we design a novel module to produce the query-aware context vector, named Multi-Space based Context Fusion (MSCF), with the following considerations: (1) interactions are applied across multiple latent semantic spaces; (2) attention is measured at bit level, not at token level. Moreover, we extend MSCF to the multi-hop architecture. This unified model is called Enhanced Attentive Reader (EA Reader). During the iterative inference process, the reader is equipped with a novel memory update rule and maintains the understanding of documents through read, update and write operations. We conduct extensive experiments on four real-world datasets. Our results demonstrate that EA Reader outperforms state-of-the-art models.

scholarly journals Exploring Periodicity and Interactivity in Multi-Interest Framework for Sequential Recommendation

2021 ◽  
Author(s):  
Gaode Chen ◽  
Xinghua Zhang ◽  
Yanyan Zhao ◽  
Cong Xue ◽  
Ji Xiang
Keyword(s):  
Real World ◽  
Information Overload ◽  
State Of The Art ◽  
Recommendation Systems ◽  
Time Interval ◽  
Interest Representation ◽  
Novel Method ◽  
Real World Datasets ◽  
Item Representation ◽  
Global And Local

Sequential recommendation systems alleviate the problem of information overload, and have attracted increasing attention in the literature. Most prior works usually obtain an overall representation based on the user’s behavior sequence, which can not sufficiently reflect the multiple interests of the user. To this end, we propose a novel method called PIMI to mitigate this issue. PIMI can model the user’s multi-interest representation effectively by considering both the periodicity and interactivity in the item sequence. Specifically, we design a periodicity-aware module to utilize the time interval information between user’s behaviors. Meanwhile, an ingenious graph is proposed to enhance the interactivity between items in user’s behavior sequence, which can capture both global and local item features. Finally, a multi-interest extraction module is applied to describe user’s multiple interests based on the obtained item representation. Extensive experiments on two real-world datasets Amazon and Taobao show that PIMI outperforms state-of-the-art methods consistently.

scholarly journals Partial Label Learning with Self-Guided Retraining

2019 ◽  
Vol 33 ◽  
pp. 3542-3549 ◽  
Author(s):  
Lei Feng ◽  
Bo An
Keyword(s):  
Real World ◽  
Optimization Problem ◽  
State Of The Art ◽  
Ground Truth ◽  
Learning Approaches ◽  
High Confidence ◽  
Infinity Norm ◽  
Real World Datasets ◽  
Partial Label Learning ◽  
Optimization Efficiency

Partial label learning deals with the problem where each training instance is assigned a set of candidate labels, only one of which is correct. This paper provides the first attempt to leverage the idea of self-training for dealing with partially labeled examples. Specifically, we propose a unified formulation with proper constraints to train the desired model and perform pseudo-labeling jointly. For pseudo-labeling, unlike traditional self-training that manually differentiates the ground-truth label with enough high confidence, we introduce the maximum infinity norm regularization on the modeling outputs to automatically achieve this consideratum, which results in a convex-concave optimization problem. We show that optimizing this convex-concave problem is equivalent to solving a set of quadratic programming (QP) problems. By proposing an upper-bound surrogate objective function, we turn to solving only one QP problem for improving the optimization efficiency. Extensive experiments on synthesized and real-world datasets demonstrate that the proposed approach significantly outperforms the state-of-the-art partial label learning approaches.

scholarly journals Multi-Aspect Embedding for Attribute-Aware Trajectories

Symmetry ◽  
2019 ◽  
Vol 11 (9) ◽  
pp. 1149
Author(s):  
Thapana Boonchoo ◽  
Xiang Ao ◽  
Qing He
Keyword(s):  
Real World ◽  
Execution Time ◽  
State Of The Art ◽  
Representation Learning ◽  
Learning Approach ◽  
Trajectory Data ◽  
Trajectory Mining ◽  
Trajectory Similarity ◽  
Effectiveness And Efficiency ◽  
Real World Datasets

Motivated by the proliferation of trajectory data produced by advanced GPS-enabled devices, trajectory is gaining in complexity and beginning to embroil additional attributes beyond simply the coordinates. As a consequence, this creates the potential to define the similarity between two attribute-aware trajectories. However, most existing trajectory similarity approaches focus only on location based proximities and fail to capture the semantic similarities encompassed by these additional asymmetric attributes (aspects) of trajectories. In this paper, we propose multi-aspect embedding for attribute-aware trajectories (MAEAT), a representation learning approach for trajectories that simultaneously models the similarities according to their multiple aspects. MAEAT is built upon a sentence embedding algorithm and directly learns whole trajectory embedding via predicting the context aspect tokens when given a trajectory. Two kinds of token generation methods are proposed to extract multiple aspects from the raw trajectories, and a regularization is devised to control the importance among aspects. Extensive experiments on the benchmark and real-world datasets show the effectiveness and efficiency of the proposed MAEAT compared to the state-of-the-art and baseline methods. The results of MAEAT can well support representative downstream trajectory mining and management tasks, and the algorithm outperforms other compared methods in execution time by at least two orders of magnitude.

scholarly journals Who Are Controlled by The Same User? Multiple Identities Deception Detection via Social Interaction Activity (Student Abstract)

2020 ◽  
Vol 34 (10) ◽  
pp. 13853-13854
Author(s):  
Jiacheng Li ◽  
Chunyuan Yuan ◽  
Wei Zhou ◽  
Jingli Wang ◽  
Songlin Hu
Keyword(s):  
Social Media ◽  
Social Interaction ◽  
Social Interactions ◽  
Real World ◽  
Deception Detection ◽  
State Of The Art ◽  
Multiple Identities ◽  
The Social ◽  
Real World Datasets ◽  
Multiple Accounts

Social media has become a preferential place for sharing information. However, some users may create multiple accounts and manipulate them to deceive legitimate users. Most previous studies utilize verbal or behavior features based methods to solve this problem, but they are only designed for some particular platforms, leading to low universalness.In this paper, to support multiple platforms, we construct interaction tree for each account based on their social interactions which is common characteristic of social platforms. Then we propose a new method to calculate the social interaction entropy of each account and detect the accounts which are controlled by the same user. Experimental results on two real-world datasets show that the method has robust superiority over state-of-the-art methods.

Sign in / Sign up

Export Citation Format

Share Document