Recent Trends in the Use of Graph Neural Network Models for Natural Language Processing

2020 ◽  
pp. 274-289
Author(s):  
BURCU YILMAZ ◽  
Hilal Genc ◽  
Mustafa Agriman ◽  
Bugra Kaan Demirdover ◽  
Mert Erdemir ◽  
...  
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Language Processing ◽  
Network Models ◽  
Feature Engineering ◽  
Graph Structure ◽  
Neural Network Models ◽  
Graph Models ◽  
Learning Tasks ◽  
Low Dimensional

Graphs are powerful data structures that allow us to represent varying relationships within data. In the past, due to the difficulties related to the time complexities of processing graph models, graphs rarely involved machine learning tasks. In recent years, especially with the new advances in deep learning techniques, increasing number of graph models related to the feature engineering and machine learning are proposed. Recently, there has been an increase in approaches that automatically learn to encode graph structure into low dimensional embedding. These approaches are accompanied by models for machine learning tasks, and they fall into two categories. The first one focuses on feature engineering techniques on graphs. The second group of models assembles graph structure to learn a graph neighborhood in the machine learning model. In this chapter, the authors focus on the advances in applications of graphs on NLP using the recent deep learning models.

scholarly journals Comparison of rule-based and neural network models for negation detection in radiology reports

2020 ◽  
pp. 1-22 ◽  
Author(s):  
D. Sykes ◽  
A. Grivas ◽  
C. Grover ◽  
R. Tobin ◽  
C. Sudlow ◽  
...  
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Language Processing ◽  
Network Models ◽  
Neural Network Models ◽  
Test Set ◽  
Rule Based ◽  
Radiology Reports ◽  
The Neural Network ◽  
Negation Detection

Abstract Using natural language processing, it is possible to extract structured information from raw text in the electronic health record (EHR) at reasonably high accuracy. However, the accurate distinction between negated and non-negated mentions of clinical terms remains a challenge. EHR text includes cases where diseases are stated not to be present or only hypothesised, meaning a disease can be mentioned in a report when it is not being reported as present. This makes tasks such as document classification and summarisation more difficult. We have developed the rule-based EdIE-R-Neg, part of an existing text mining pipeline called EdIE-R (Edinburgh Information Extraction for Radiology reports), developed to process brain imaging reports, (https://www.ltg.ed.ac.uk/software/edie-r/) and two machine learning approaches; one using a bidirectional long short-term memory network and another using a feedforward neural network. These were developed on data from the Edinburgh Stroke Study (ESS) and tested on data from routine reports from NHS Tayside (Tayside). Both datasets consist of written reports from medical scans. These models are compared with two existing rule-based models: pyConText (Harkema et al. 2009. Journal of Biomedical Informatics42(5), 839–851), a python implementation of a generalisation of NegEx, and NegBio (Peng et al. 2017. NegBio: A high-performance tool for negation and uncertainty detection in radiology reports. arXiv e-prints, p. arXiv:1712.05898), which identifies negation scopes through patterns applied to a syntactic representation of the sentence. On both the test set of the dataset from which our models were developed, as well as the largely similar Tayside test set, the neural network models and our custom-built rule-based system outperformed the existing methods. EdIE-R-Neg scored highest on F1 score, particularly on the test set of the Tayside dataset, from which no development data were used in these experiments, showing the power of custom-built rule-based systems for negation detection on datasets of this size. The performance gap of the machine learning models to EdIE-R-Neg on the Tayside test set was reduced through adding development Tayside data into the ESS training set, demonstrating the adaptability of the neural network models.

Data processing using deep learning of the generative-adversarial neural network (GAN)

2021 ◽  
Author(s):  
V.Y. Ilichev ◽  
I.V. Chukhraev
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Deep Learning ◽  
Effective Means ◽  
Network Models ◽  
Neural Network Models ◽  
High Quality ◽  
Python Language ◽  
And Training

The article is devoted to the consideration of one of the areas of application of modern and promising computer technology – machine learning. This direction is based on the creation of models consisting of neural networks and their deep learning. At present, there is a need to generate new, not yet existing, images of objects of different types. Most often, text files or images act as such objects. To achieve a high quality of results, a generation method based on the adversarial work of two neural networks (generator and discriminator) was once worked out. This class of neural network models is distinguished by the complexity of topography, since it is necessary to correctly organize the structure of neural layers in order to achieve maximum accuracy and minimal error. The described program is created using the Python language and special libraries that extend the set of commands for performing additional functions: working with neural networks Keras (main library), integrating with the operating system Os, outputting graphs Matplotlib, working with data arrays Numpy and others. A description is given of the type and features of each neural layer, as well as the use of library connection functions, input of initial data, compilation and training of the obtained model. Next, the implementation of the procedure for outputting the results of evaluating the errors of the generator and discriminator and the accuracy achieved by the model depending on the number of cycles (eras) of its training is considered. Based on the results of the work, conclusions were drawn and recommendations were made for the use and development of the considered methodology for creating and training generative and adversarial neural networks. Studies have demonstrated the procedure for operating with comparatively simple and accessible, but effective means of a universal Python language with the Keras library to create and teach a complex neural network model. In fact, it has been proved that the use of this method allows to achieve high-quality results of machine learning, previously achievable only when using special software systems for working with neural networks.

scholarly journals Arabic text summarization using deep learning approach

2020 ◽  
Vol 7 (1) ◽  
Author(s):  
Molham Al-Maleh ◽  
Said Desouki
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Language Processing ◽  
Network Models ◽  
Arabic Language ◽  
Text Summarization ◽  
Neural Network Models ◽  
Data Set ◽  
Learning Techniques ◽  
Arabic Text Summarization

AbstractNatural language processing has witnessed remarkable progress with the advent of deep learning techniques. Text summarization, along other tasks like text translation and sentiment analysis, used deep neural network models to enhance results. The new methods of text summarization are subject to a sequence-to-sequence framework of encoder–decoder model, which is composed of neural networks trained jointly on both input and output. Deep neural networks take advantage of big datasets to improve their results. These networks are supported by the attention mechanism, which can deal with long texts more efficiently by identifying focus points in the text. They are also supported by the copy mechanism that allows the model to copy words from the source to the summary directly. In this research, we are re-implementing the basic summarization model that applies the sequence-to-sequence framework on the Arabic language, which has not witnessed the employment of this model in the text summarization before. Initially, we build an Arabic data set of summarized article headlines. This data set consists of approximately 300 thousand entries, each consisting of an article introduction and the headline corresponding to this introduction. We then apply baseline summarization models to the previous data set and compare the results using the ROUGE scale.

scholarly journals Recommender System for Telecom Packages Based on the Deep & Cross Network

2021 ◽  
Author(s):  
Congming Shi ◽  
Wen Wang ◽  
Shoulin Wei ◽  
Feiya Lv
Keyword(s):  
Deep Learning ◽  
Recommender System ◽  
Network Models ◽  
Feature Engineering ◽  
Neural Network Models ◽  
Learning Abilities ◽  
Powerful Learning ◽  
Text Information ◽  
Real World Datasets ◽  
Click Through Rate

Abstract With the evolution of 5G technology, the telecom industry has influenced the livelihood of people and impacted the development of the national economy significantly. To increase revenue per customer and secure long-term contracts of users, telecommunications firms and enterprises have launched different types of telecom packages to satisfy the varying requirements of users. Although several recommender systems have been proposed in recent years for telecommunication package recommendation, extracting effective feature information from large and complex consumption data remains challenging. Considering the telecom package recommendation problems, traditional recommendation methods either use complex expert feature engineering or fail to perform end-to-end deep learning training. In this study, a recommender system based on deep & cross network (DCN), deep belief network (DBN), Embedding, and Word2Vec is proposed using the powerful learning abilities of deep learning. The proposed system fits the telecom package recommender system in terms of click-through rate prediction to provide a potential solution for the recommendation challenges faced by telecom enterprises. The proposed model can effectively capture the finite order interactional features and deep hidden features. Additionally, the text information in the data is completely used to further improve the recommendation ability of the model. Moreover, the proposed method does not require feature engineering. We conducted comprehensive experiments using real-world datasets, and the results verify that our method can generate improved recommendation accuracy in comparison with those observed in DBN, DCN, deep factorization machine, and deep neural network models individually.

scholarly journals Machine Learning in Agriculture Application: Algorithms and Techniques

2020 ◽  
Vol 9 (6) ◽  
pp. 1140-1146
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
High Performance ◽  
Network Models ◽  
Processing Technique ◽  
Machine Learning Techniques ◽  
Image Processing Technique ◽  
Neural Network Models ◽  
Survey Paper ◽  
Learning Techniques

Machine learning techniques with high performance computing technologies can create various new opportunities in the agriculture domain. This paper does comprehensivereview of various papers which are concentrating on machine learning (ML) and deep learning application in agriculture. This paper is categorized into three sections a) Yield prediction using machine learning technique b) Price prediction c) Leaf disease detection using neural networks. In this paper we study the comparison of neural network models with existing models. The findings of this survey paper indicate Deep learning models give high accuracy and outperform traditional image processing technique and ML techniques outperforms various traditional techniques in prediction.

scholarly journals Skin Disease Detection: Machine Learning vs Deep Learning

2021 ◽  
Author(s):  
Samir Bandyopadhyay ◽  
Amiya Bhaumik ◽  
Sandeep Poddar
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Learning Process ◽  
Skin Disease ◽  
Nearest Neighbor ◽  
Network Models ◽  
Support Vector ◽  
Common Disease ◽  
K Nearest Neighbor ◽  
Neural Network Models

Skin disease is a very common disease for humans. In the medical industry detecting skin disease and recognizing its type is a very challenging task. Due to the complexity of human skin texture and the visual closeness effect of the diseases, sometimes it is really difficult to detect the exact type. Therefore, it is necessary to detect and recognize the skin disease at its very first observation. In today's era, artificial intelligence (AI) is rapidly growing in medical fields. Different machine learning (ML) and deep learning(DL) algorithms are used for diagnostic purposes. These methods drastically improve the diagnosis process and also speed up the process. In this paper, a brief comparison between the machine learning process and the deep learning process was discussed. In both processes, three different and popular algorithms are used. For the machine Learning process Bagged Tree Ensemble, K-Nearest Neighbor (KNN), and Support Vector Machine(SVM) algorithms were used. For the deep learning process three pre-trained deep neural network models

scholarly journals A COMPREHENSIVE STUDY ON APPLICATION OF DEEP LEARNING IN BRAIN TUMOR DETECTION

2021 ◽  
Vol 23 (07) ◽  
pp. 977-994
Author(s):  
Josmy Mathew ◽  
◽  
Dr. N. Srinivasan ◽  
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Brain Tumours ◽  
Data Augmentation ◽  
State Of The Art ◽  
Network Models ◽  
Data Sets ◽  
Learning Approaches ◽  
Neural Network Models ◽  
Comprehensive Study

Deep Learning is an area of machine learning which, because of its capability to handle a large quantity of data, has demonstrated amazing achievements in each field, notably in biomedicine. Its potential and abilities were evaluated and utilised with an effective prognosis in the identification of brain tumours with MRI pictures. The diagnosis of MRI images by computer-aided brain tumours includes tumour identification, segmentation and classification. Many types of research have concentrated in recent years on conventional or basic machine learning approaches in the detection of brain tumours. Throughout this overview, we offer a comprehensive assessment of the surveys that have been reported so far and the current approaches for detecting tumours. Our review examines the major processes in deep learning approaches for detecting brain tumours including preprocessing, extraction of features and classification and their performance and limitations. We also explore state-of-the-art neural network models to identify brain tumours through extensive trials with and without data augmentation. This review also discusses existing data sets for brain tumour detection assessments.

scholarly journals Neural Models of Text Normalization for Speech Applications

2019 ◽  
Vol 45 (2) ◽  
pp. 293-337 ◽  
Author(s):  
Hao Zhang ◽  
Richard Sproat ◽  
Axel H. Ng ◽  
Felix Stahlberg ◽  
Xiaochang Peng ◽  
...  
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Language Processing ◽  
Speech Synthesis ◽  
Network Models ◽  
Neural Models ◽  
Neural Network Models ◽  
Sentential Context ◽  
Finite State ◽  
Text Normalization

Machine learning, including neural network techniques, have been applied to virtually every domain in natural language processing. One problem that has been somewhat resistant to effective machine learning solutions is text normalization for speech applications such as text-to-speech synthesis (TTS). In this application, one must decide, for example, that 123 is verbalized as one hundred twenty three in 123 pages but as one twenty three in 123 King Ave. For this task, state-of-the-art industrial systems depend heavily on hand-written language-specific grammars. We propose neural network models that treat text normalization for TTS as a sequence-to-sequence problem, in which the input is a text token in context, and the output is the verbalization of that token. We find that the most effective model, in accuracy and efficiency, is one where the sentential context is computed once and the results of that computation are combined with the computation of each token in sequence to compute the verbalization. This model allows for a great deal of flexibility in terms of representing the context, and also allows us to integrate tagging and segmentation into the process. These models perform very well overall, but occasionally they will predict wildly inappropriate verbalizations, such as reading 3 cm as three kilometers. Although rare, such verbalizations are a major issue for TTS applications. We thus use finite-state covering grammars to guide the neural models, either during training and decoding, or just during decoding, away from such “unrecoverable” errors. Such grammars can largely be learned from data.

Machine Learning Crowdfunding

2020 ◽  
Vol 10 (2) ◽  
pp. 1-11
Author(s):  
Evangelos Katsamakas ◽  
Hao Sun
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Language Processing ◽  
Topic Modeling ◽  
Network Models ◽  
Unstructured Data ◽  
Text Analytics ◽  
Learning Models ◽  
Neural Network Models ◽  
Machine Learning Models

Crowdfunding is a novel and important economic mechanism for funding projects and promoting innovation in the digital economy. This article explores most recent structured and unstructured data from a crowdfunding platform. It provides an in-depth exploration of the data using text analytics techniques, such as sentiment analysis and topic modeling. It uses novel natural language processing to represent project descriptions, and evaluates machine learning models, including neural network models, to predict project fundraising success. It discusses the findings of the performance evaluation, and summarizes lessons for crowdfunding platforms and their users.

scholarly journals A Survey on Silicon Photonics for Deep Learning

2021 ◽  
Vol 17 (4) ◽  
pp. 1-57
Author(s):  
Febin P. Sunny ◽  
Ebadollah Taheri ◽  
Mahdi Nikdast ◽  
Sudeep Pasricha
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Language Processing ◽  
Integrated Circuit ◽  
Silicon Photonics ◽  
General Pattern ◽  
Network Models ◽  
Neural Network Models ◽  
Data Movement ◽  
Improved Performance

Deep learning has led to unprecedented successes in solving some very difficult problems in domains such as computer vision, natural language processing, and general pattern recognition. These achievements are the culmination of decades-long research into better training techniques and deeper neural network models, as well as improvements in hardware platforms that are used to train and execute the deep neural network models. Many application-specific integrated circuit (ASIC) hardware accelerators for deep learning have garnered interest in recent years due to their improved performance and energy-efficiency over conventional CPU and GPU architectures. However, these accelerators are constrained by fundamental bottlenecks due to (1) the slowdown in CMOS scaling, which has limited computational and performance-per-watt capabilities of emerging electronic processors; and (2) the use of metallic interconnects for data movement, which do not scale well and are a major cause of bandwidth, latency, and energy inefficiencies in almost every contemporary processor. Silicon photonics has emerged as a promising CMOS-compatible alternative to realize a new generation of deep learning accelerators that can use light for both communication and computation. This article surveys the landscape of silicon photonics to accelerate deep learning, with a coverage of developments across design abstractions in a bottom-up manner, to convey both the capabilities and limitations of the silicon photonics paradigm in the context of deep learning acceleration.

Sign in / Sign up

Export Citation Format

Share Document