LBERT: Lexically aware Transformer-based Bidirectional Encoder Representation model for learning universal bio-entity relations

2020 ◽  
Author(s):  
Neha Warikoo ◽  
Yung-Chun Chang ◽  
Wen-Lian Hsu
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Predictive Analytics ◽  
Relation Extraction ◽  
Data Representation ◽  
Supplementary Information ◽  
Biomedical Domain ◽  
Critical Function ◽  
Representation Model ◽  
Classification Tasks

Abstract Motivation Natural Language Processing techniques are constantly being advanced to accommodate the influx of data as well as to provide exhaustive and structured knowledge dissemination. Within the biomedical domain, relation detection between bio-entities known as the Bio-Entity Relation Extraction (BRE) task has a critical function in knowledge structuring. Although recent advances in deep learning-based biomedical domain embedding have improved BRE predictive analytics, these works are often task selective or use external knowledge-based pre-/post-processing. In addition, deep learning-based models do not account for local syntactic contexts, which have improved data representation in many kernel classifier-based models. In this study, we propose a universal BRE model, i.e. LBERT, which is a Lexically aware Transformer-based Bidirectional Encoder Representation model, and which explores both local and global contexts representations for sentence-level classification tasks. Results This article presents one of the most exhaustive BRE studies ever conducted over five different bio-entity relation types. Our model outperforms state-of-the-art deep learning models in protein–protein interaction (PPI), drug–drug interaction and protein–bio-entity relation classification tasks by 0.02%, 11.2% and 41.4%, respectively. LBERT representations show a statistically significant improvement over BioBERT in detecting true bio-entity relation for large corpora like PPI. Our ablation studies clearly indicate the contribution of the lexical features and distance-adjusted attention in improving prediction performance by learning additional local semantic context along with bi-directionally learned global context. Availability and implementation Github. https://github.com/warikoone/LBERT. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals How to Adapt Deep Learning Models to a New Domain: The Case of Biomedical Relation Extraction

TecnoLógicas ◽  
2019 ◽  
Vol 22 ◽  
pp. 49-62 ◽  
Author(s):  
Jefferson A. Peña-Torres ◽  
Raúl E. Gutiérrez ◽  
Víctor A. Bucheli ◽  
Fabio A. González
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Intelligent Systems ◽  
Domain Adaptation ◽  
Relation Extraction ◽  
Semantic Relations ◽  
Biomedical Domain ◽  
External Resources ◽  
Short Time ◽  
Biomedical Relation Extraction

In this article, we study the relation extraction problem from Natural Language Processing (NLP) implementing a domain adaptation setting without external resources. We trained a Deep Learning (DL) model for Relation Extraction (RE), which extracts semantic relations in the biomedical domain. However, can the model be applied to different domains? The model should be adaptable to automatically extract relationships across different domains using the DL network. Completely training DL models in a short time is impractical because the models should quickly adapt to different datasets in several domains without delay. Therefore, adaptation is crucial for intelligent systems, where changing factors and unanticipated perturbations are common. In this study, we present a detailed analysis of the problem, as well as preliminary experimentation, results, and their evaluation.

scholarly journals Deep-learning approach to identifying cancer subtypes using high-dimensional genomic data

2019 ◽  
Author(s):  
Runpu Chen ◽  
Le Yang ◽  
Steve Goodison ◽  
Yijun Sun
Keyword(s):  
Deep Learning ◽  
Cluster Structure ◽  
Data Representation ◽  
Supplementary Information ◽  
High Dimensional ◽  
Breast Cancer Dataset ◽  
Cancer Dataset ◽  
Cancer Subtypes ◽  
Novel Approach ◽  
Open Source Software Package

Abstract Motivation Cancer subtype classification has the potential to significantly improve disease prognosis and develop individualized patient management. Existing methods are limited by their ability to handle extremely high-dimensional data and by the influence of misleading, irrelevant factors, resulting in ambiguous and overlapping subtypes. Results To address the above issues, we proposed a novel approach to disentangling and eliminating irrelevant factors by leveraging the power of deep learning. Specifically, we designed a deep-learning framework, referred to as DeepType, that performs joint supervised classification, unsupervised clustering and dimensionality reduction to learn cancer-relevant data representation with cluster structure. We applied DeepType to the METABRIC breast cancer dataset and compared its performance to state-of-the-art methods. DeepType significantly outperformed the existing methods, identifying more robust subtypes while using fewer genes. The new approach provides a framework for the derivation of more accurate and robust molecular cancer subtypes by using increasingly complex, multi-source data. Availability and implementation An open-source software package for the proposed method is freely available at http://www.acsu.buffalo.edu/~yijunsun/lab/DeepType.html. Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals The Unreasonable Effectiveness of the Baseline: Discussing SVMs in Legal Text Classification

2021 ◽  
Author(s):  
Benjamin Clavié ◽  
Marc Alphonsus
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Text Classification ◽  
Traditional Approach ◽  
Error Reduction ◽  
Support Vector ◽  
Learning Models ◽  
Legal Text ◽  
Classification Tasks ◽  
Legal Domain

We aim to highlight an interesting trend to contribute to the ongoing debate around advances within legal Natural Language Processing. Recently, the focus for most legal text classification tasks has shifted towards large pre-trained deep learning models such as BERT. In this paper, we show that a more traditional approach based on Support Vector Machine classifiers reaches competitive performance with deep learning models. We also highlight that error reduction obtained by using specialised BERT-based models over baselines is noticeably smaller in the legal domain when compared to general language tasks. We discuss some hypotheses for these results to support future discussions.

scholarly journals A comparative review on deep learning models for text classification

2020 ◽  
Vol 19 (1) ◽  
pp. 325
Author(s):  
Muhammad Zulqarnain ◽  
Rozaida Ghazali ◽  
Yana Mazwin Mohmad Hassim ◽  
Muhammad Rehan
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Language Processing ◽  
Text Classification ◽  
Question Answering ◽  
Learning Models ◽  
Semantic Classification ◽  
Analysis Question ◽  
Comparative Review ◽  
Classification Tasks

<p>Text classification is a fundamental task in several areas of natural language processing (NLP), including words semantic classification, sentiment analysis, question answering, or dialog management. This paper investigates three basic architectures of deep learning models for the tasks of text classification: Deep Belief Neural (DBN), Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), these three main types of deep learning architectures, are largely explored to handled various classification tasks. DBN have excellent learning capabilities to extracts highly distinguishable features and good for general purpose. CNN have supposed to be better at extracting the position of various related features while RNN is modeling in sequential of long-term dependencies. This paper work shows the systematic comparison of DBN, CNN, and RNN on text classification tasks. Finally, we show the results of deep models by research experiment. The aim of this paper to provides basic guidance about the deep learning models that which models are best for the task of text classification.</p>

scholarly journals Using distant supervision to augment manually annotated data for relation extraction

2019 ◽  
Author(s):  
Peng Su ◽  
Gang Li ◽  
Cathy Wu ◽  
K. Vijay-Shanker
Keyword(s):  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Relation Extraction ◽  
Biomedical Literature ◽  
Training Data ◽  
Distant Supervision ◽  
Large Size ◽  
Domain Expertise

AbstractSignificant progress has been made in applying deep learning on natural language processing tasks recently. However, deep learning models typically require a large amount of annotated training data while often only small labeled datasets are available for many natural language processing tasks in biomedical literature. Building large-size datasets for deep learning is expensive since it involves considerable human effort and usually requires domain expertise in specialized fields. In this work, we consider augmenting manually annotated data with large amounts of data using distant supervision. However, data obtained by distant supervision is often noisy, we first apply some heuristics to remove some of the incorrect annotations. Then using methods inspired from transfer learning, we show that the resulting models outperform models trained on the original manually annotated sets.

scholarly journals Same Representation, Different Attentions: Shareable Sentence Representation Learning from Multiple Tasks

2018 ◽  
Author(s):  
Renjie Zheng ◽  
Junkun Chen ◽  
Xipeng Qiu
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Text Classification ◽  
Representation Learning ◽  
Training Data ◽  
Specific Information ◽  
Distributed Representation ◽  
Source Codes ◽  
Multiple Tasks ◽  
Classification Tasks

Distributed representation plays an important role in deep learning based natural language processing. However, the representation of a sentence often varies in different tasks, which is usually learned from scratch and suffers from the limited amounts of training data. In this paper, we claim that a good sentence representation should be invariant and can benefit the various subsequent tasks. To achieve this purpose, we propose a new scheme of information sharing for multi-task learning. More specifically, all tasks share the same sentence representation and each task can select the task-specific information from the shared sentence representation with attention mechanisms. The query vector of each task's attention could be either static parameters or generated dynamically. We conduct extensive experiments on 16 different text classification tasks, which demonstrate the benefits of our architecture. Source codes of this paper are available on Github.

Case Studies in Amalgamation of Deep Learning and Big Data

2018 ◽  
pp. 159-174 ◽  
Author(s):  
Balajee Jeyakumar ◽  
M.A. Saleem Durai ◽  
Daphne Lopez
Keyword(s):  
Big Data ◽  
Deep Learning ◽  
Language Processing ◽  
Predictive Analytics ◽  
Vital Role ◽  
Absolute Amount ◽  
Water Resource System ◽  
The World ◽  
Research Domain ◽  
Resource System

Deep learning is now more popular research domain in machine learning and pattern recognition in the world. It is widely success in the far-reaching area of applications such as Speech recognition, Computer vision, Natural language processing and Reinforcement learning. With the absolute amount of data accessible nowadays, big data brings chances and transformative possible for several sectors, on the other hand, it also performs on the unpredicted defies to connecting data and information. The size of the data is getting larger, and deep learning is imminent to play a vital role in big data predictive analytics solutions. In this paper, we make available a brief outline of deep learning and focus recent research efforts and the challenges in the fields of science, medical and water resource system.

Extraction of similar biomedical terms in biomedical literature mining: Examining the effect of the ratio of biomedical domain to general domain data (Preprint)

2021 ◽  
Author(s):  
Ziheng Zhang ◽  
Feng Han ◽  
Hongjian Zhang ◽  
Tomohiro Aoki ◽  
Katsuhiko Ogasawara
Keyword(s):  
Language Processing ◽  
Relation Extraction ◽  
Medical Data ◽  
Biomedical Literature ◽  
Literature Mining ◽  
Biomedical Domain ◽  
Pubmed Central ◽  
General Domain ◽  
Biomedical Information Retrieval ◽  
Science Community

BACKGROUND Biomedical terms extracted using Word2vec, the most popular word embedding model in recent years, serve as the foundation for various natural language processing (NLP) applications, such as biomedical information retrieval, relation extraction, and recommendation systems. OBJECTIVE The objective of this study is to examine how changes in the ratio of biomedical domain to general domain data in the corpus affect the extraction of similar biomedical terms using Word2vec. METHODS We downloaded abstracts of 214892 articles from PubMed Central (PMC) and the 3.9 GB Billion Word (BW) benchmark corpus from the computer science community. The datasets were preprocessed and grouped into 11 corpora based on the ratio of BW to PMC, ranging from 0:10 to 10:0, and then Word2vec models were trained on these corpora. The cosine similarities between the biomedical terms obtained from the Word2vec models were then compared in each model. RESULTS The results indicated that the models trained with both BW and PMC data outperformed the model trained only with medical data. The similarity between the biomedical terms extracted by the Word2vec model increased, when the ratio of biomedical domain to general domain data was 3: 7 to 5: 5. CONCLUSIONS This study allows NLP researchers to apply Word2vec based on more information and increase the similarity of extracted biomedical terms to improve their effectiveness in NLP applications, such as biomedical information extraction.

Case Studies in Amalgamation of Deep Learning and Big Data

2020 ◽  
pp. 981-993
Author(s):  
Balajee Jeyakumar ◽  
M.A. Saleem Durai ◽  
Daphne Lopez
Keyword(s):  
Big Data ◽  
Deep Learning ◽  
Language Processing ◽  
Predictive Analytics ◽  
Vital Role ◽  
Absolute Amount ◽  
Water Resource System ◽  
The World ◽  
Research Domain ◽  
Resource System

Deep learning is now more popular research domain in machine learning and pattern recognition in the world. It is widely success in the far-reaching area of applications such as Speech recognition, Computer vision, Natural language processing and Reinforcement learning. With the absolute amount of data accessible nowadays, big data brings chances and transformative possible for several sectors, on the other hand, it also performs on the unpredicted defies to connecting data and information. The size of the data is getting larger, and deep learning is imminent to play a vital role in big data predictive analytics solutions. In this paper, we make available a brief outline of deep learning and focus recent research efforts and the challenges in the fields of science, medical and water resource system.

scholarly journals Evaluation of Methods for Protein Representation Learning: A Quantitative Analysis

2020 ◽  
Author(s):  
Serbulent Unsal ◽  
Heval Ataş ◽  
Muammer Albayrak ◽  
Kemal Turhan ◽  
Aybar C. Acar ◽  
...  
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Complex Traits ◽  
Representation Learning ◽  
Data Representation ◽  
Language Models ◽  
Learning Methods ◽  
Advantages And Disadvantages ◽  
Data Representations ◽  
Novel Methods

AbstractData-centric approaches have been utilized to develop predictive methods for elucidating uncharacterized aspects of proteins such as their functions, biophysical properties, subcellular locations and interactions. However, studies indicate that the performance of these methods should be further improved to effectively solve complex problems in biomedicine and biotechnology. A data representation method can be defined as an algorithm that calculates numerical feature vectors for samples in a dataset, to be later used in quantitative modelling tasks. Data representation learning methods do this by training and using a model that employs statistical and machine/deep learning algorithms. These novel methods mostly take inspiration from the data-driven language models that have yielded ground-breaking improvements in the field of natural language processing. Lately, these learned data representations have been applied to the field of protein informatics and have displayed highly promising results in terms of extracting complex traits of proteins regarding sequence-structure-function relations. In this study, we conducted a detailed investigation over protein representation learning methods, by first categorizing and explaining each approach, and then conducting benchmark analyses on; (i) inferring semantic similarities between proteins, (ii) predicting ontology-based protein functions, and (iii) classifying drug target protein families. We examine the advantages and disadvantages of each representation approach over the benchmark results. Finally, we discuss current challenges and suggest future directions. We believe the conclusions of this study will help researchers in applying machine/deep learning-based representation techniques on protein data for various types of predictive tasks. Furthermore, we hope it will demonstrate the potential of machine learning-based data representations for protein science and inspire the development of novel methods/tools to be utilized in the fields of biomedicine and biotechnology.

Sign in / Sign up

Export Citation Format

Share Document