scholarly journals Bert-Enhanced Text Graph Neural Network for Classification

Entropy ◽  
2021 ◽  
Vol 23 (11) ◽  
pp. 1536
Author(s):  
Yiping Yang ◽  
Xiaohui Cui
Keyword(s):  
Neural Network ◽  
Semantic Information ◽  
Structural Information ◽  
Text Processing ◽  
Language Model ◽  
Research Direction ◽  
Semantic Features ◽  
Structure Information ◽  
Single Text ◽  
Graph Neural Networks

Text classification is a fundamental research direction, aims to assign tags to text units. Recently, graph neural networks (GNN) have exhibited some excellent properties in textual information processing. Furthermore, the pre-trained language model also realized promising effects in many tasks. However, many text processing methods cannot model a single text unit’s structure or ignore the semantic features. To solve these problems and comprehensively utilize the text’s structure information and semantic information, we propose a Bert-Enhanced text Graph Neural Network model (BEGNN). For each text, we construct a text graph separately according to the co-occurrence relationship of words and use GNN to extract text features. Moreover, we employ Bert to extract semantic features. The former part can take into account the structural information, and the latter can focus on modeling the semantic information. Finally, we interact and aggregate these two features of different granularity to get a more effective representation. Experiments on standard datasets demonstrate the effectiveness of BEGNN.

scholarly journals SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

2021 ◽  
Vol 40 (3) ◽  
pp. 1-13
Author(s):  
Lumin Yang ◽  
Jiajie Zhuang ◽  
Hongbo Fu ◽  
Xiangzhi Wei ◽  
Kun Zhou ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Large Scale ◽  
State Of The Art ◽  
Semantic Segmentation ◽  
Structure Information ◽  
Graph Neural Networks ◽  
Node Labels ◽  
Point Level

We introduce SketchGNN , a convolutional graph neural network for semantic segmentation and labeling of freehand vector sketches. We treat an input stroke-based sketch as a graph with nodes representing the sampled points along input strokes and edges encoding the stroke structure information. To predict the per-node labels, our SketchGNN uses graph convolution and a static-dynamic branching network architecture to extract the features at three levels, i.e., point-level, stroke-level, and sketch-level. SketchGNN significantly improves the accuracy of the state-of-the-art methods for semantic sketch segmentation (by 11.2% in the pixel-based metric and 18.2% in the component-based metric over a large-scale challenging SPG dataset) and has magnitudes fewer parameters than both image-based and sequence-based methods.

scholarly journals Network Embedding via a Bi-Mode and Deep Neural Network Model

2017 ◽  
Author(s):  
Yang Fang ◽  
Xiang Zhao ◽  
Zhen Tan
Keyword(s):  
Neural Network ◽  
Deep Neural Network ◽  
Semantic Information ◽  
Dimensional Space ◽  
Relation Extraction ◽  
Network Embedding ◽  
Structure Information ◽  
Second Mode ◽  
Real World Datasets ◽  
Low Dimensional

Network Embedding (NE) is an important method to learn the representations of network via a low-dimensional space. Conventional NE models focus on capturing the structure information and semantic information of vertices while neglecting such information for edges. In this work, we propose a novel NE model named BimoNet to capture both the structure and semantic information of edges. BimoNet is composed of two parts, i.e., the bi-mode embedding part and the deep neural network part. For bi-mode embedding part, the first mode named add-mode is used to express the entity-shared features of edges and the second mode named subtract-mode is employed to represent the entity-specific features of edges. These features actually reflect the semantic information. For deep neural network part, we firstly regard the edges in a network as nodes, and the vertices as links, which will not change the overall structure of the whole network. Then we take the nodes' adjacent matrix as the input of the deep neural network as it can obtain similar representations for nodes with similar structure. Afterwards, by jointly optimizing the objective function of these two parts, BimoNet could preserve both the semantic and structure information of edges. In experiments, we evaluate BimoNet on three real-world datasets and task of relation extraction, and BimoNet is demonstrated to outperform state-of-the-art baseline models consistently and significantly.

scholarly journals A Hybrid Neural Network BERT-Cap Based on Pre-Trained Language Model and Capsule Network for User Intent Classification

Complexity ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-11
Author(s):  
Hai Liu ◽  
Yuanxia Liu ◽  
Leung-Pun Wong ◽  
Lap-Kei Lee ◽  
Tianyong Hao
Keyword(s):  
Neural Network ◽  
Neural Network Model ◽  
Question Answering ◽  
Semantic Information ◽  
Language Model ◽  
Dialogue System ◽  
Hybrid Neural Network ◽  
Question Answering System ◽  
User Intent ◽  
Vital Component

User intent classification is a vital component of a question-answering system or a task-based dialogue system. In order to understand the goals of users’ questions or discourses, the system categorizes user text into a set of pre-defined user intent categories. User questions or discourses are usually short in length and lack sufficient context; thus, it is difficult to extract deep semantic information from these types of text and the accuracy of user intent classification may be affected. To better identify user intents, this paper proposes a BERT-Cap hybrid neural network model with focal loss for user intent classification to capture user intents in dialogue. The model uses multiple transformer encoder blocks to encode user utterances and initializes encoder parameters with a pre-trained BERT. Then, it extracts essential features using a capsule network with dynamic routing after utterances encoding. Experiment results on four publicly available datasets show that our model BERT-Cap achieves a F1 score of 0.967 and an accuracy of 0.967, outperforming a number of baseline methods, indicating its effectiveness in user intent classification.

scholarly journals Evidence for a deep, distributed and dynamic code for animacy in human ventral anterior temporal cortex

eLife ◽  
2021 ◽  
Vol 10 ◽  
Author(s):  
Timothy T Rogers ◽  
Christopher R Cox ◽  
Qihong Lu ◽  
Akihiro Shimotake ◽  
Takayuki Kikuch ◽  
...  
Keyword(s):  
Neural Network ◽  
Semantic Information ◽  
Semantic Network ◽  
Temporal Cortex ◽  
Semantic Features ◽  
Temporal Lobes ◽  
Correct Information ◽  
Neural Populations ◽  
Ventral Temporal Cortex ◽  
Multivariate Pattern

How does the human brain encode semantic information about objects? This paper reconciles two seemingly contradictory views. The first proposes that local neural populations independently encode semantic features; the second, that semantic representations arise as a dynamic distributed code that changes radically with stimulus processing. Combining simulations with a well-known neural network model of semantic memory, multivariate pattern classification, and human electrocorticography, we find that both views are partially correct: information about the animacy of a depicted stimulus is distributed across ventral temporal cortex in a dynamic code possessing feature-like elements posteriorly but with elements that change rapidly and nonlinearly in anterior regions. This pattern is consistent with the view that anterior temporal lobes serve as a deep cross-modal ‘hub’ in an interactive semantic network, and more generally suggests that tertiary association cortices may adopt dynamic distributed codes difficult to detect with common brain imaging methods.

Randomness Evaluation for Scrambled Image Using Intersecting Cortical Model Neural Network

2014 ◽  
Vol 530-531 ◽  
pp. 434-442
Author(s):  
Jing Jing Zhao ◽  
Guang Zhu Xu ◽  
Bang Jun Lei ◽  
Chun Lin Li
Keyword(s):  
Neural Network ◽  
Structural Information ◽  
Cortical Model ◽  
Image Scrambling ◽  
Human Visual Perception ◽  
L1 Norm ◽  
Structure Information ◽  
Binary Images ◽  
Input Field ◽  
Evaluation Algorithm

A bio-inspired visual neural network named ICMNN was adopted to extract image structural information for image scrambling evaluation. In order to describe image structure more effectively with ICMNN, image bitmap was introduced into ICMNN input field. First, the original image was decomposed into eight binary images and each was scrambled with Arnold transformation without loss of generality. Then, ICMNN was adopted to extract the structural feature sequence of bitmap images and their corresponding scrambled ones. Last, L1 norm of the structure change sequence between them was calculated to evaluate the scrambling degree of the scrambling images. Results show that combining image bitmap decomposition with ICMNN can effectively evaluate image scrambling degree and describe the change of the structure information, which agrees with human visual perception. This evaluation algorithm is also independent of the scrambling algorithm and has a good versatility.

scholarly journals Graph Neural Networks with Multiple Feature Extraction Paths for Chemical Property Estimation

Molecules ◽  
2021 ◽  
Vol 26 (11) ◽  
pp. 3125
Author(s):  
Sho Ishida ◽  
Tomo Miyazaki ◽  
Yoshihiro Sugaya ◽  
Shinichiro Omachi
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Feature Extraction ◽  
Chemical Property ◽  
Structural Information ◽  
Three Dimensional ◽  
Property Estimation ◽  
Multiple Feature ◽  
Multiple Structures ◽  
Graph Neural Networks

Feature extraction is essential for chemical property estimation of molecules using machine learning. Recently, graph neural networks have attracted attention for feature extraction from molecules. However, existing methods focus only on specific structural information, such as node relationship. In this paper, we propose a novel graph convolutional neural network that performs feature extraction with simultaneously considering multiple structures. Specifically, we propose feature extraction paths specialized in node, edge, and three-dimensional structures. Moreover, we propose an attention mechanism to aggregate the features extracted by the paths. The attention aggregation enables us to select useful features dynamically. The experimental results showed that the proposed method outperformed previous methods.

scholarly journals A Novel Approach for Malicious URL Detection Based on the Joint Model

2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
JianTing Yuan ◽  
YiPeng Liu ◽  
Long Yu
Keyword(s):  
Neural Network ◽  
Semantic Information ◽  
Attention Mechanism ◽  
Semantic Features ◽  
Uniform Resource Locator ◽  
Word Level ◽  
Network Algorithm ◽  
Model Combining ◽  
Novel Approach ◽  
Neural Network Algorithm

The number of malicious websites is increasing yearly, and many companies and individuals worldwide have suffered losses. Therefore, the detection of malicious websites is a task that needs continuous development. In this study, a joint neural network algorithm model combining the attention mechanism, bidirectional independent recurrent neural network (Bi-IndRNN), and capsule network (CapsNet) is proposed. The word vector tool word2vec trains the character- and word-level uniform resource locator (URL) static embedding vector features. At the same time, the algorithm will also extract texture fingerprint features that can compare the content differences of different malicious web URL binary files. Then, the extracted features are fused and input into the joint neural network algorithm model. First, the multihead attention mechanism is used to extract contextual semantic features by adjusting weights and Bi-IndRNN. Second, CapsNet with dynamic routing is used to extract deep semantic information. Finally, the sigmoid classifier is used for classification. This study uses different methods from different angles to extract more comprehensive features. From the experimental results, the method proposed in this study improves the classification accuracy of malicious web page detection compared with other researchers.

scholarly journals A Source Code Similarity Based on Siamese Neural Network

2020 ◽  
Vol 10 (21) ◽  
pp. 7519
Author(s):  
Chunli Xie ◽  
Xia Wang ◽  
Cheng Qian ◽  
Mengqi Wang
Keyword(s):  
Neural Network ◽  
Semantic Information ◽  
Language Model ◽  
Similarity Score ◽  
Statistical Language Model ◽  
Inverse Document Frequency ◽  
Continuous Space ◽  
Document Frequency ◽  
Space Vectors ◽  
Similar Code

Finding similar code snippets is a fundamental task in the field of software engineering. Several approaches have been proposed for this task by using statistical language model which focuses on syntax and structure of codes rather than deep semantic information underlying codes. In this paper, a Siamese Neural Network is proposed that maps codes into continuous space vectors and try to capture their semantic meaning. Firstly, an unsupervised pre-trained method that models code snippets as a weighted series of word vectors. The weights of the series are fitted by the Term Frequency-Inverse Document Frequency (TF-IDF). Then, a Siamese Neural Network trained model is constructed to learn semantic vector representation of code snippets. Finally, the cosine similarity is provided to measure the similarity score between pairs of code snippets. Moreover, we have implemented our approach on a dataset of functionally similar code. The experimental results show that our method improves some performance over single word embedding method.

scholarly journals Bidirectional Recurrent Neural Network Language Model: Cross Entropy Churn Metrics for Defect Prediction Modelling

2019 ◽  
Vol 8 (6) ◽  
pp. 2792-2800
Keyword(s):  
Neural Network ◽  
Prediction Models ◽  
Dynamic Error ◽  
Language Model ◽  
Source Code ◽  
Cross Entropy ◽  
Machine Language ◽  
Defect Prediction ◽  
Semantic Features

Software Defect Prediction (SDP) plays an active area in many research domain of Software Quality of Assurance (SQA). Many existing research studies are based on software traditional metric sets and defect prediction models are built in machine language to detect the bug for limited source code line. Inspired by the above existing system. In this paper, defect prediction is focused on predicting defects in source code. The aim of this dissertation is to enhance the quality of the software for precise prediction of defects. So, that it helps the developer to find the bug and fix the issue, to make better use of a resource which reduces the test effort, minimize the cost and improve the quality of software. A new approach is introduced to improve the prediction performance of Bidirectional RNNLM in Deep Neural Network. To build the defect prediction model a defect learner framework is proposed and first it need to build a Neural Language Model. Using this Language Model it helps to learn to deep semantic features in source code and it train & test the model. Based on language model it combined with software traditional metric sets to measure the code and find the defect. The probability of language model and metric set Cross-Entropy with Abstract Syntax Tree (CE-AST) metric is used to evaluate the defect proneness and set as a metric label. For classification the metric label K-NN classifier is used. BPTT algorithm for learning RNN will provide additional improvement, it improves the predictions performance to find the dynamic error.

scholarly journals A Feature Combination-Based Graph Convolutional Neural Network Model for Relation Extraction

Symmetry ◽  
2021 ◽  
Vol 13 (8) ◽  
pp. 1458
Author(s):  
Jinling Xu ◽  
Yanping Chen ◽  
Yongbin Qin ◽  
Ruizhang Huang ◽  
Qinghua Zheng
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Prior Knowledge ◽  
Network Model ◽  
Neural Network Model ◽  
Structural Information ◽  
Relation Extraction ◽  
Feature Combination ◽  
Named Entities ◽  
Graph Neural Networks

The task to extract relations tries to identify relationships between two named entities in a sentence. Because a sentence usually contains several named entities, capturing structural information of a sentence is important to support this task. Currently, graph neural networks are widely implemented to support relation extraction, in which dependency trees are employed to generate adjacent matrices for encoding structural information of a sentence. Because parsing a sentence is error-prone, it influences the performance of a graph neural network. On the other hand, a sentence is structuralized by several named entities, which precisely segment a sentence into several parts. Different features can be combined by prior knowledge and experience, which are effective to initialize a symmetric adjacent matrix for a graph neural network. Based on this phenomenon, we proposed a feature combination-based graph convolutional neural network model (FC-GCN). It has the advantages of encoding structural information of a sentence, considering prior knowledge, and avoiding errors caused by parsing. In the experiments, the results show significant improvement, which outperform existing state-of-the-art performances.

Sign in / Sign up

Export Citation Format

Share Document