Improving Text Matching with Semantic Dependency Graph via Message Passing Neural Network

2021 ◽  
pp. 395-406
Author(s):  
Yongkang Song ◽  
Dianqing Liu ◽  
Dazhan Mao ◽  
Yanqiu Shao
Keyword(s):  
Neural Network ◽  
Message Passing ◽  
Dependency Graph ◽  
Semantic Dependency ◽  
Text Matching

scholarly journals JAMPI: Efficient Matrix Multiplication in Spark Using Barrier Execution Mode

2020 ◽  
Author(s):  
Tamas Foldi ◽  
Chris von Csefalvay ◽  
Nicolas A. Perez
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Message Passing ◽  
Matrix Multiplication ◽  
Map Reduce ◽  
Distributed Training ◽  
Learning Tasks ◽  
Memory Footprint ◽  
Asynchronous Network ◽  
Execution Mode

The new barrier mode in Apache Spark allows embedding distributed deep learning training as a Spark stage to simplify the distributed training workflow. In Spark, a task in a stage doesn’t depend on any other tasks in the same stage, and hence it can be scheduled independently. However, several algorithms require more sophisticated inter-task communications, similar to the MPI paradigm. By combining distributed message passing (using asynchronous network IO), OpenJDK’s new auto-vectorization and Spark’s barrier execution mode, we can add non-map/reduce based algorithms, such as Cannon’s distributed matrix multiplication to Spark. We document an efficient distributed matrix multiplication using Cannon’s algorithm, which improves significantly on the performance of the existing MLlib implementation. Used within a barrier task, the algorithm described herein results in an up to 24% performance increase on a 10,000x10,000 square matrix with a significantly lower memory footprint. Applications of efficient matrix multiplication include, among others, accelerating the training and implementation of deep convolutional neural network based workloads, and thus such efficient algorithms can play a ground-breaking role in faster, more efficient execution of even the most complicated machine learning tasks

Matrix inference and estimation in multi-layer models*

2021 ◽  
Vol 2021 (12) ◽  
pp. 124004
Author(s):  
Parthe Pandit ◽  
Mojtaba Sahraee-Ardakan ◽  
Sundeep Rangan ◽  
Philip Schniter ◽  
Alyson K Fletcher
Keyword(s):  
Neural Network ◽  
Message Passing ◽  
Hidden Variables ◽  
Signal Recovery ◽  
Inference Problem ◽  
Neural Network Learning ◽  
Approximate Message Passing ◽  
Message Passing Algorithm ◽  
Training Samples ◽  
Mixed Regression

Abstract We consider the problem of estimating the input and hidden variables of a stochastic multi-layer neural network (NN) from an observation of the output. The hidden variables in each layer are represented as matrices with statistical interactions along both rows as well as columns. This problem applies to matrix imputation, signal recovery via deep generative prior models, multi-task and mixed regression, and learning certain classes of two-layer NNs. We extend a recently-developed algorithm—multi-layer vector approximate message passing, for this matrix-valued inference problem. It is shown that the performance of the proposed multi-layer matrix vector approximate message passing algorithm can be exactly predicted in a certain random large-system limit, where the dimensions N × d of the unknown quantities grow as N → ∞ with d fixed. In the two-layer neural-network learning problem, this scaling corresponds to the case where the number of input features as well as training samples grow to infinity but the number of hidden nodes stays fixed. The analysis enables a precise prediction of the parameter and test error of the learning.

Research on Neural Networks Used in Parallel Computing

2013 ◽  
Vol 411-414 ◽  
pp. 1998-2001
Author(s):  
Liang Cheng
Keyword(s):  
Neural Network ◽  
Message Passing ◽  
Paper Study ◽  
Optimal Solutions ◽  
Inference System ◽  
Iterative Processing ◽  
Computational Environment ◽  
The Neural Network ◽  
Fuzzy Neural ◽  
Expected Increase

The parallel programming approaches were in the focus of research efforts due to an expected increase in efficiency of iterative processing in the parallel computational environment. On this end the parallel evolutionary asymmetric subset-hood product fuzzy-neural inference system has been developed to take advantage of parallelization in message passing. This paper study the structure of the neural network and the time series forecasting with neural network, the results could help us to obtain the optimal solutions to higher complexity of the problem.

scholarly journals CapsTM: capsule network for Chinese medical text matching

2021 ◽  
Vol 21 (S2) ◽  
Author(s):  
Xiaoming Yu ◽  
Yedan Shen ◽  
Yuan Ni ◽  
Xiaowei Huang ◽  
Xiaolong Wang ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Network Architecture ◽  
State Of The Art ◽  
Interaction Matrix ◽  
Application Systems ◽  
Input Layer ◽  
Art Methods ◽  
Text Matching

Abstract Background Text Matching (TM) is a fundamental task of natural language processing widely used in many application systems such as information retrieval, automatic question answering, machine translation, dialogue system, reading comprehension, etc. In recent years, a large number of deep learning neural networks have been applied to TM, and have refreshed benchmarks of TM repeatedly. Among the deep learning neural networks, convolutional neural network (CNN) is one of the most popular networks, which suffers from difficulties in dealing with small samples and keeping relative structures of features. In this paper, we propose a novel deep learning architecture based on capsule network for TM, called CapsTM, where capsule network is a new type of neural network architecture proposed to address some of the short comings of CNN and shows great potential in many tasks. Methods CapsTM is a five-layer neural network, including an input layer, a representation layer, an aggregation layer, a capsule layer and a prediction layer. In CapsTM, two pieces of text are first individually converted into sequences of embeddings and are further transformed by a highway network in the input layer. Then, Bidirectional Long Short-Term Memory (BiLSTM) is used to represent each piece of text and attention-based interaction matrix is used to represent interactive information of the two pieces of text in the representation layer. Subsequently, the two kinds of representations are fused together by BiLSTM in the aggregation layer, and are further represented with capsules (vectors) in the capsule layer. Finally, the prediction layer is a connected network used for classification. CapsTM is an extension of ESIM by adding a capsule layer before the prediction layer. Results We construct a corpus of Chinese medical question matching, which contains 36,360 question pairs. This corpus is randomly split into three parts: a training set of 32,360 question pairs, a development set of 2000 question pairs and a test set of 2000 question pairs. On this corpus, we conduct a series of experiments to evaluate the proposed CapsTM and compare it with other state-of-the-art methods. CapsTM achieves the highest F-score of 0.8666. Conclusion The experimental results demonstrate that CapsTM is effective for Chinese medical question matching and outperforms other state-of-the-art methods for comparison.

scholarly journals Graph Neural Network for Object Reconstruction in Liquid Argon Time Projection Chambers

2021 ◽  
Vol 251 ◽  
pp. 03054 ◽  
Author(s):  
Jeremy Hewes ◽  
Adam Aurisano ◽  
Giuseppe Cerati ◽  
Jim Kowalkowski ◽  
Claire Lee ◽  
...  
Keyword(s):  
Neural Network ◽  
Message Passing ◽  
Liquid Argon ◽  
Great Promise ◽  
Muon Track ◽  
Particle Type ◽  
Novel Technique ◽  
Time Projection Chambers ◽  
Time Projection ◽  
The Relationship

This paper presents a graph neural network (GNN) technique for low-level reconstruction of neutrino interactions in a Liquid Argon Time Projection Chamber (LArTPC). GNNs are still a relatively novel technique, and have shown great promise for similar reconstruction tasks in the LHC. In this paper, a multihead attention message passing network is used to classify the relationship between detector hits by labelling graph edges, determining whether hits were produced by the same underlying particle, and if so, the particle type. The trained model is 84% accurate overall, and performs best on the EM shower and muon track classes. The model’s strengths and weaknesses are discussed, and plans for developing this technique further are summarised.

scholarly journals Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification

2021 ◽  
Author(s):  
Yunsheng Shi ◽  
Zhengjie Huang ◽  
Shikun Feng ◽  
Hui Zhong ◽  
Wenjing Wang ◽  
...  
Keyword(s):  
Neural Network ◽  
Message Passing ◽  
Supervised Classification ◽  
State Of The Art ◽  
Label Propagation ◽  
Superior Performance ◽  
Label Prediction ◽  
Label Information ◽  
Message Passing Algorithms ◽  
Prediction Strategy

Graph neural network (GNN) and label propagation algorithm (LPA) are both message passing algorithms, which have achieved superior performance in semi-supervised classification. GNN performs feature propagation by a neural network to make predictions, while LPA uses label propagation across graph adjacency matrix to get results. However, there is still no effective way to directly combine these two kinds of algorithms. To address this issue, we propose a novel Unified Message Passaging Model (UniMP) that can incorporate feature and label propagation at both training and inference time. First, UniMP adopts a Graph Transformer network, taking feature embedding and label embedding as input information for propagation. Second, to train the network without overfitting in self-loop input label information, UniMP introduces a masked label prediction strategy, in which some percentage of input label information are masked at random, and then predicted. UniMP conceptually unifies feature propagation and label propagation and is empirically powerful. It obtains new state-of-the-art semi-supervised classification results in Open Graph Benchmark (OGB).

Modeling Data Center Networks with Message Passing Neural Network and Multi-task Learning

2021 ◽  
pp. 96-112
Author(s):  
Kai Zhang ◽  
Xueyong Xu ◽  
Chenchen Fu ◽  
Xiumin Wang ◽  
Weiwei Wu
Keyword(s):  
Neural Network ◽  
Data Center ◽  
Message Passing ◽  
Data Center Networks ◽  
Task Learning ◽  
Modeling Data
Sign in / Sign up

Export Citation Format

Share Document