scholarly journals From Tokenization to Self-Supervision: Building a High-Performance Information Extraction System for Chemical Reactions in Patents

2021 ◽  
Vol 6 ◽  
Author(s):  
Jingqi Wang ◽  
Yuankai Ren ◽  
Zhi Zhang ◽  
Hua Xu ◽  
Yaoyun Zhang
Keyword(s):  
Information Extraction ◽  
Chemical Reactions ◽  
Chemical Reaction ◽  
High Performance ◽  
Event Extraction ◽  
Entity Recognition ◽  
Language Models ◽  
Accurate Information ◽  
Free Text ◽  
Semantic Roles

Chemical reactions and experimental conditions are fundamental information for chemical research and pharmaceutical applications. However, the latest information of chemical reactions is usually embedded in the free text of patents. The rapidly accumulating chemical patents urge automatic tools based on natural language processing (NLP) techniques for efficient and accurate information extraction. This work describes the participation of the Melax Tech team in the CLEF 2020—ChEMU Task of Chemical Reaction Extraction from Patent. The task consisted of two subtasks: (1) named entity recognition to identify compounds and different semantic roles in the chemical reaction and (2) event extraction to identify event triggers of chemical reaction and their relations with the semantic roles recognized in subtask 1. To build an end-to-end system with high performance, multiple strategies tailored to chemical patents were applied and evaluated, ranging from optimizing the tokenization, pre-training patent language models based on self-supervision, to domain knowledge-based rules. Our hybrid approaches combining different strategies achieved state-of-the-art results in both subtasks, with the top-ranked F1 of 0.957 for entity recognition and the top-ranked F1 of 0.9536 for event extraction, indicating that the proposed approaches are promising.

scholarly journals Identifying Chemical Reactions and Their Associated Attributes in Patents

2021 ◽  
Vol 6 ◽  
Author(s):  
Darshini Mahendran ◽  
Gabrielle Gurdin ◽  
Nastassja Lewinski ◽  
Christina Tang ◽  
Bridget T. McInnes
Keyword(s):  
Chemical Reactions ◽  
Chemical Reaction ◽  
Short Term Memory ◽  
Named Entity Recognition ◽  
Event Extraction ◽  
Entity Recognition ◽  
Automatic Identification ◽  
Named Entity ◽  
Long Short Term Memory ◽  
Source Of Information

Chemical patents are an essential source of information about novel chemicals and chemical reactions. However, with the increasing volume of such patents, mining information about these chemicals and chemical reactions has become a time-intensive and laborious endeavor. In this study, we present a system to extract chemical reaction events from patents automatically. Our approach consists of two steps: 1) named entity recognition (NER)—the automatic identification of chemical reaction parameters from the corresponding text, and 2) event extraction (EE)—the automatic classifying and linking of entities based on their relationships to each other. For our NER system, we evaluate bidirectional long short-term memory (BiLSTM)-based and bidirectional encoder representations from transformer (BERT)-based methods. For our EE system, we evaluate BERT-based, convolutional neural network (CNN)-based, and rule-based methods. We evaluate our NER and EE components independently and as an end-to-end system, reporting the precision, recall, and F1 score. Our results show that the BiLSTM-based method performed best at identifying the entities, and the CNN-based method performed best at extracting events.

Enriching contextualized language model from knowledge graph for biomedical information extraction

2020 ◽  
Author(s):  
Hao Fei ◽  
Yafeng Ren ◽  
Yue Zhang ◽  
Donghong Ji ◽  
Xiaohui Liang
Keyword(s):  
Information Extraction ◽  
Large Scale ◽  
Language Model ◽  
Relation Extraction ◽  
Event Extraction ◽  
Entity Recognition ◽  
Language Models ◽  
Training Procedure ◽  
Biomedical Knowledge ◽  
Biomedical Texts

Abstract Biomedical information extraction (BioIE) is an important task. The aim is to analyze biomedical texts and extract structured information such as named entities and semantic relations between them. In recent years, pre-trained language models have largely improved the performance of BioIE. However, they neglect to incorporate external structural knowledge, which can provide rich factual information to support the underlying understanding and reasoning for biomedical information extraction. In this paper, we first evaluate current extraction methods, including vanilla neural networks, general language models and pre-trained contextualized language models on biomedical information extraction tasks, including named entity recognition, relation extraction and event extraction. We then propose to enrich a contextualized language model by integrating a large scale of biomedical knowledge graphs (namely, BioKGLM). In order to effectively encode knowledge, we explore a three-stage training procedure and introduce different fusion strategies to facilitate knowledge injection. Experimental results on multiple tasks show that BioKGLM consistently outperforms state-of-the-art extraction models. A further analysis proves that BioKGLM can capture the underlying relations between biomedical knowledge concepts, which are crucial for BioIE.

scholarly journals ChEMU 2020: Natural Language Processing Methods Are Effective for Information Extraction From Chemical Patents

2021 ◽  
Vol 6 ◽  
Author(s):  
Jiayuan He ◽  
Dat Quoc Nguyen ◽  
Saber A. Akhondi ◽  
Christian Druckenbrodt ◽  
Camilo Thorne ◽  
...  
Keyword(s):  
Information Extraction ◽  
Chemical Reactions ◽  
Language Processing ◽  
Named Entity Recognition ◽  
Event Extraction ◽  
Entity Recognition ◽  
Named Entity ◽  
Fundamental Information ◽  
Essential Chemical ◽  
Reaction Processes

Chemical patents represent a valuable source of information about new chemical compounds, which is critical to the drug discovery process. Automated information extraction over chemical patents is, however, a challenging task due to the large volume of existing patents and the complex linguistic properties of chemical patents. The Cheminformatics Elsevier Melbourne University (ChEMU) evaluation lab 2020, part of the Conference and Labs of the Evaluation Forum 2020 (CLEF2020), was introduced to support the development of advanced text mining techniques for chemical patents. The ChEMU 2020 lab proposed two fundamental information extraction tasks focusing on chemical reaction processes described in chemical patents: (1) chemical named entity recognition, requiring identification of essential chemical entities and their roles in chemical reactions, as well as reaction conditions; and (2) event extraction, which aims at identification of event steps relating the entities involved in chemical reactions. The ChEMU 2020 lab received 37 team registrations and 46 runs. Overall, the performance of submissions for these tasks exceeded our expectations, with the top systems outperforming strong baselines. We further show the methods to be robust to variations in sampling of the test data. We provide a detailed overview of the ChEMU 2020 corpus and its annotation, showing that inter-annotator agreement is very strong. We also present the methods adopted by participants, provide a detailed analysis of their performance, and carefully consider the potential impact of data leakage on interpretation of the results. The ChEMU 2020 Lab has shown the viability of automated methods to support information extraction of key information in chemical patents.

scholarly journals TaxoNERD: deep neural models for the recognition of taxonomic entities in the ecological and evolutionary literature

2021 ◽  
Author(s):  
Nicolas Le Guillarme ◽  
Wilfried Thuiller
Keyword(s):  
Information Extraction ◽  
High Performance ◽  
Named Entity Recognition ◽  
Open Data ◽  
Entity Recognition ◽  
Data Repositories ◽  
Biodiversity Crisis ◽  
Domain Specific ◽  
Ecological Information ◽  
Gold Standard Corpus

1. Given the biodiversity crisis, we more than ever need to access information on multiple taxa (e.g. distribution, traits, diet) in the scientific literature to understand, map and predict all-inclusive biodiversity. Tools are needed to automatically extract useful information from the ever-growing corpus of ecological texts and feed this information to open data repositories. A prerequisite is the ability to recognise mentions of taxa in text, a special case of named entity recognition (NER). In recent years, deep learning-based NER systems have become ubiqutous, yielding state-of-the-art results in the general and biomedical domains. However, no such tool is available to ecologists wishing to extract information from the biodiversity literature. 2. We propose a new tool called TaxoNERD that provides two deep neural network (DNN) models to recognise taxon mentions in ecological documents. To achieve high performance, DNN-based NER models usually need to be trained on a large corpus of manually annotated text. Creating such a gold standard corpus (GSC) is a laborious and costly process, with the result that GSCs in the ecological domain tend to be too small to learn an accurate DNN model from scratch. To address this issue, we leverage existing DNN models pretrained on large biomedical corpora using transfer learning. The performance of our models is evaluated on four GSCs and compared to the most popular taxonomic NER tools. 3. Our experiments suggest that existing taxonomic NER tools are not suited to the extraction of ecological information from text as they performed poorly on ecologically-oriented corpora, either because they do not take account of the variability of taxon naming practices, or because they do not generalise well to the ecological domain. Conversely, a domain-specific DNN-based tool like TaxoNERD outperformed the other approaches on an ecological information extraction task. 4. Efforts are needed in order to raise ecological information extraction to the same level of performance as its biomedical counterpart. One promising direction is to leverage the huge corpus of unlabelled ecological texts to learn a language representation model that could benefit downstream tasks. These efforts could be highly beneficial to ecologists on the long term.

scholarly journals Overview of CCKS 2020 Task 3: Named Entity Recognition and Event Extraction in Chinese Electronic Medical Records

2021 ◽  
pp. 1-13
Author(s):  
Xia Li ◽  
Qinghua Wen ◽  
Zengtao Jiao ◽  
Jiangtao Zhang
Keyword(s):  
Electronic Medical Records ◽  
Medical Records ◽  
Named Entity Recognition ◽  
Event Extraction ◽  
Entity Recognition ◽  
Language Models ◽  
Data Sets ◽  
External Resources ◽  
Named Entity ◽  
Evaluation Task

Abstract The China Conference on Knowledge Graph and Semantic Computing (CCKS) 2020 Evaluation Task 3 presented clinical named entity recognition and event extraction for the Chinese electronic medical records. Two annotated data sets and some other additional resources for these two subtasks were provided for participators. This evaluation competition attracted 354 teams and 46 of them successfully submitted the valid results. The pre-trained language models are widely applied in this evaluation task. Data argumentation and external resources are also helpful.

Information Extraction

2012 ◽  
Author(s):  
Ralph Grishman
Keyword(s):  
Information Extraction ◽  
Conventional Treatment ◽  
Linguistic Analysis ◽  
Event Extraction ◽  
Automatic Identification ◽  
Free Text ◽  
Regular Expressions ◽  
Language Analysis ◽  
Finite State ◽  
Linguistic Units

Information extraction (IE) is the automatic identification of selected types of entities, relations, or events in free text. This article appraises two specific strands of IE — name identification and classification, and event extraction. Conventional treatment of languages pays little attention to proper names, addresses etc. Presentations of language analysis generally look up words in a dictionary and identify them as nouns etc. The incessant presence of names in a text, makes linguistic analysis of the same difficult, in the absence of the names being identified by their types and as linguistic units. Name tagging involves creating, several finite-state patterns, each corresponding to some noun subset. Elements of the patterns would match specific/classes of tokens with particular features. Event extraction typically works by creating a series of regular expressions, customized to capture the relevant events. Enhancement of each expression is corresponded by a relevant, suitable enhancement in the event patterns.

scholarly journals Ensemble of deep masked language models for effective named entity recognition in multi-domain corpora

2021 ◽  
Author(s):  
Nona Naderi ◽  
Julien Knafou ◽  
Jenny Copara ◽  
Patrick Ruch ◽  
Douglas Teodoro
Keyword(s):  
Named Entity Recognition ◽  
Entity Recognition ◽  
Language Models ◽  
Free Text ◽  
Percentage Point Increase ◽  
Named Entity ◽  
Secondary Use ◽  
Point Increase ◽  
Wet Lab ◽  
Multiple Domains

AbstractThe health and life science domains are well known for their wealth of entities. These entities are presented as free text in large corpora, such as biomedical scientific and electronic health records. To enable the secondary use of these corpora and unlock their value, named entity recognition (NER) methods are proposed. Inspired by the success of deep masked language models, we present an ensemble approach for NER using these models. Results show statistically significant improvement of the ensemble models over baselines based on individual models in multiple domains - chemical, clinical and wet lab - and languages - English and French. The ensemble model achieves an overall performance of 79.2% macro F1-score, a 4.6 percentage point increase upon the baseline in multiple domains and languages. These results suggests that ensembles are a more effective strategy for tackling NER. We further perform a detailed analysis of their performance based on a set of entity properties.

scholarly journals ChEMU shared task: chemical entity recognition and event extraction of chemical reactions from patents

2020 ◽  
Author(s):  
Christian Druckenbrodt ◽  
Saber Akhondi ◽  
Karin Verspoor ◽  
Zenan Zhai ◽  
Camilo Thorne ◽  
...  
Keyword(s):  
Chemical Reactions ◽  
Chemical Entity ◽  
Event Extraction ◽  
Entity Recognition ◽  
Shared Task

scholarly journals The Impact of Pretrained Language Models on Negation and Speculation Detection in Cross-Lingual Medical Text: Comparative Study

10.2196/18953 ◽  
2020 ◽  
Vol 8 (12) ◽  
pp. e18953
Author(s):  
Renzo Rivera Zavala ◽  
Paloma Martinez
Keyword(s):  
Machine Learning ◽  
Information Extraction ◽  
Fine Tuning ◽  
Entity Recognition ◽  
Language Models ◽  
Special Focus ◽  
Rule Based ◽  
Clinical Narrative ◽  
Cross Lingual ◽  
The Impact

Background Negation and speculation are critical elements in natural language processing (NLP)-related tasks, such as information extraction, as these phenomena change the truth value of a proposition. In the clinical narrative that is informal, these linguistic facts are used extensively with the objective of indicating hypotheses, impressions, or negative findings. Previous state-of-the-art approaches addressed negation and speculation detection tasks using rule-based methods, but in the last few years, models based on machine learning and deep learning exploiting morphological, syntactic, and semantic features represented as spare and dense vectors have emerged. However, although such methods of named entity recognition (NER) employ a broad set of features, they are limited to existing pretrained models for a specific domain or language. Objective As a fundamental subsystem of any information extraction pipeline, a system for cross-lingual and domain-independent negation and speculation detection was introduced with special focus on the biomedical scientific literature and clinical narrative. In this work, detection of negation and speculation was considered as a sequence-labeling task where cues and the scopes of both phenomena are recognized as a sequence of nested labels recognized in a single step. Methods We proposed the following two approaches for negation and speculation detection: (1) bidirectional long short-term memory (Bi-LSTM) and conditional random field using character, word, and sense embeddings to deal with the extraction of semantic, syntactic, and contextual patterns and (2) bidirectional encoder representations for transformers (BERT) with fine tuning for NER. Results The approach was evaluated for English and Spanish languages on biomedical and review text, particularly with the BioScope corpus, IULA corpus, and SFU Spanish Review corpus, with F-measures of 86.6%, 85.0%, and 88.1%, respectively, for NeuroNER and 86.4%, 80.8%, and 91.7%, respectively, for BERT. Conclusions These results show that these architectures perform considerably better than the previous rule-based and conventional machine learning–based systems. Moreover, our analysis results show that pretrained word embedding and particularly contextualized embedding for biomedical corpora help to understand complexities inherent to biomedical text.

scholarly journals A Pre-Training Technique to Localize Medical BERT and to Enhance Biomedical BERT

2020 ◽  
Author(s):  
Shoya Wada ◽  
Toshihiro Takeda ◽  
Shiro Manabe ◽  
Shozo Konishi ◽  
Jun Kamohara ◽  
...  
Keyword(s):  
Language Processing ◽  
High Performance ◽  
Large Scale ◽  
Language Models ◽  
Free Text ◽  
Medical Databases ◽  
Large Size ◽  
Training Technique ◽  
Medical Domain ◽  
Medical Document

Abstract Background: Pre-training large-scale neural language models on raw texts has been shown to make a significant contribution to a strategy for transfer learning in natural language processing (NLP). With the introduction of transformer-based language models, such as Bidirectional Encoder Representations from Transformers (BERT), the performance of information extraction from free text by NLP has significantly improved for both the general domain and the medical domain; however, it is difficult for languages in which there are few publicly available medical databases with a high quality and a large size to train medical BERT models that perform well.Method: We introduce a method to train a BERT model using a small medical corpus both in English and in Japanese. Our proposed method consists of two interventions: simultaneous pre-training, which is intended to encourage masked language modeling and next-sentence prediction on the small medical corpus, and amplified vocabulary, which helps with suiting the small corpus when building the customized corpus by byte-pair encoding. Moreover, we used whole PubMed abstracts and developed a high-performance BERT model, Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University (ouBioBERT), in English via our method. We then evaluated the performance of our BERT models and publicly available baselines and compared them.Results: We confirmed that our Japanese medical BERT outperforms conventional baselines and the other BERT models in terms of the medical-document classification task and that our English BERT pre-trained using both the general and medical domain corpora performs sufficiently for practical use in terms of the biomedical language understanding evaluation (BLUE) benchmark. Moreover, ouBioBERT shows that the total score of the BLUE benchmark is 1.1 points above that of BioBERT and 0.3 points above that of the ablation model trained without our proposed method.Conclusions: Our proposed method makes it feasible to construct a practical medical BERT model in both Japanese and English, and it has a potential to produce higher performing models for biomedical shared tasks.

Sign in / Sign up

Export Citation Format

Share Document