scholarly journals Hierarchical Contextualized Representation for Named Entity Recognition

2020 ◽  
Vol 34 (05) ◽  
pp. 8441-8448
Author(s):  
Ying Luo ◽  
Fengshun Xiao ◽  
Hai Zhao
Keyword(s):  
Named Entity Recognition ◽  
Entity Recognition ◽  
Named Entity ◽  
Sentence Level ◽  
Unique Word ◽  
Entire Sentence ◽  
Single Input ◽  
Memory Network ◽  
Full Utilization ◽  
Document Level

Named entity recognition (NER) models are typically based on the architecture of Bi-directional LSTM (BiLSTM). The constraints of sequential nature and the modeling of single input prevent the full utilization of global information from larger scope, not only in the entire sentence, but also in the entire document (dataset). In this paper, we address these two deficiencies and propose a model augmented with hierarchical contextualized representation: sentence-level representation and document-level representation. In sentence-level, we take different contributions of words in a single sentence into consideration to enhance the sentence representation learned from an independent BiLSTM via label embedding attention mechanism. In document-level, the key-value memory network is adopted to record the document-aware information for each unique word which is sensitive to similarity of context information. Our two-level hierarchical contextualized representations are fused with each input token embedding and corresponding hidden state of BiLSTM, respectively. The experimental results on three benchmark NER datasets (CoNLL-2003 and Ontonotes 5.0 English datasets, CoNLL-2002 Spanish dataset) show that we establish new state-of-the-art results.

scholarly journals Leveraging Document-Level Label Consistency for Named Entity Recognition

2020 ◽  
Author(s):  
Tao Gui ◽  
Jiacheng Ye ◽  
Qi Zhang ◽  
Yaqian Zhou ◽  
Yeyun Gong ◽  
...  
Keyword(s):  
Neural Networks ◽  
Side Effects ◽  
High Probability ◽  
State Of The Art ◽  
Named Entity Recognition ◽  
Entity Recognition ◽  
Named Entity ◽  
Memory Network ◽  
Document Level ◽  
Token Sequence

Document-level label consistency is an effective indicator that different occurrences of a particular token sequence are very likely to have the same entity types. Previous work focused on better context representations and used the CRF for label decoding. However, CRF-based methods are inadequate for modeling document-level label consistency. This work introduces a novel two-stage label refinement approach to handle document-level label consistency, where a key-value memory network is first used to record draft labels predicted by the base model, and then a multi-channel Transformer makes refinements on these draft predictions based on the explicit co-occurrence relationship derived from the memory network. In addition, in order to mitigate the side effects of incorrect draft labels, Bayesian neural networks are used to indicate the labels with a high probability of being wrong, which can greatly assist in preventing the incorrect refinement of correct draft labels. The experimental results on three named entity recognition benchmarks demonstrated that the proposed method significantly outperformed the state-of-the-art methods.

scholarly journals Leveraging Multi-Token Entities in Document-Level Named Entity Recognition

2020 ◽  
Vol 34 (05) ◽  
pp. 7961-7968
Author(s):  
Anwen Hu ◽  
Zhicheng Dou ◽  
Jian-Yun Nie ◽  
Ji-Rong Wen
Keyword(s):  
Neural Networks ◽  
State Of The Art ◽  
Named Entity Recognition ◽  
Entity Recognition ◽  
Context Information ◽  
Named Entity ◽  
Sentence Level ◽  
Semantic Relevance ◽  
Recognition Systems ◽  
Document Level

Most state-of-the-art named entity recognition systems are designed to process each sentence within a document independently. These systems are easy to confuse entity types when the context information in a sentence is not sufficient enough. To utilize the context information within the whole document, most document-level work let neural networks on their own to learn the relation across sentences, which is not intuitive enough for us humans. In this paper, we divide entities to multi-token entities that contain multiple tokens and single-token entities that are composed of a single token. We propose that the context information of multi-token entities should be more reliable in document-level NER for news articles. We design a fusion attention mechanism which not only learns the semantic relevance between occurrences of the same token, but also focuses more on occurrences belonging to multi-tokens entities. To identify multi-token entities, we design an auxiliary task namely ‘Multi-token Entity Classification’ and perform this task simultaneously with document-level NER. This auxiliary task is simplified from NER and doesn't require extra annotation. Experimental results on the CoNLL-2003 dataset and OntoNotesnbm dataset show that our model outperforms state-of-the-art sentence-level and document-level NER methods.

scholarly journals InferNER: an attentive model leveraging the sentence-level information for Named Entity Recognition in Microblogs

2021 ◽  
Vol 34 (1) ◽  
Author(s):  
Moemmur Shahzad ◽  
Ayesha Amin ◽  
Diego Esteves ◽  
Axel-Cyrille Ngonga Ngomo
Keyword(s):  
Social Media ◽  
Visual Information ◽  
State Of The Art ◽  
Named Entity Recognition ◽  
Entity Recognition ◽  
Named Entity ◽  
Current State ◽  
Sentence Level ◽  
Level Information ◽  
External Sources

We investigate the problem of named entity recognition in the user-generated text such as social media posts. This task is rendered particularly difficult by the restricted length and limited grammatical coherence of this data type. Current state-of-the-art approaches rely on external sources such as gazetteers to alleviate some of these restrictions. We present a neural model able to outperform state of the art on this task without recurring to gazetteers or similar external sources of information. Our approach relies on word-, character-, and sentence-level information for NER in short-text. Social media posts like tweets often have associated images that may provide auxiliary context relevant to understand these texts. Hence, we also incorporate visual information and introduce an attention component which computes attention weight probabilities over textual and text-relevant visual contexts separately. Our model outperforms the current state of the art on various NER datasets. On WNUT 2016 and 2017, our model achieved 53.48\% and 50.52\% F1 score, respectively. With Multimodal model, our system also outperforms the current SOTA with an F1 score of 74\% on the multimodal dataset. Our evaluation further suggests that our model also goes beyond the current state-of-the-art on newswire data, hence corroborating its suitability for various NER tasks.

scholarly journals An ERNIE-Based Joint Model for Chinese Named Entity Recognition

2020 ◽  
Vol 10 (16) ◽  
pp. 5711
Author(s):  
Yu Wang ◽  
Yining Sun ◽  
Zuchang Ma ◽  
Lisheng Gao ◽  
Yang Xu
Keyword(s):  
Language Processing ◽  
Text Classification ◽  
Knowledge Integration ◽  
Named Entity Recognition ◽  
Training Model ◽  
Joint Model ◽  
Entity Recognition ◽  
Named Entity ◽  
Sentence Level ◽  
Joint Training

Named Entity Recognition (NER) is the fundamental task for Natural Language Processing (NLP) and the initial step in building a Knowledge Graph (KG). Recently, BERT (Bidirectional Encoder Representations from Transformers), which is a pre-training model, has achieved state-of-the-art (SOTA) results in various NLP tasks, including the NER. However, Chinese NER is still a more challenging task for BERT because there are no physical separations between Chinese words, and BERT can only obtain the representations of Chinese characters. Nevertheless, the Chinese NER cannot be well handled with character-level representations, because the meaning of a Chinese word is quite different from that of the characters, which make up the word. ERNIE (Enhanced Representation through kNowledge IntEgration), which is an improved pre-training model of BERT, is more suitable for Chinese NER because it is designed to learn language representations enhanced by the knowledge masking strategy. However, the potential of ERNIE has not been fully explored. ERNIE only utilizes the token-level features and ignores the sentence-level feature when performing the NER task. In this paper, we propose the ERNIE-Joint, which is a joint model based on ERNIE. The ERNIE-Joint can utilize both the sentence-level and token-level features by joint training the NER and text classification tasks. In order to use the raw NER datasets for joint training and avoid additional annotations, we perform the text classification task according to the number of entities in the sentences. The experiments are conducted on two datasets: MSRA-NER and Weibo. These datasets contain Chinese news data and Chinese social media data, respectively. The results demonstrate that the ERNIE-Joint not only outperforms BERT and ERNIE but also achieves the SOTA results on both datasets.

scholarly journals A hybrid deep learning framework for bacterial named entity recognition with domain features

2019 ◽  
Vol 20 (S16) ◽  
Author(s):  
Xusheng Li ◽  
Chengcheng Fu ◽  
Ran Zhong ◽  
Duo Zhong ◽  
Tingting He ◽  
...  
Keyword(s):  
Deep Learning ◽  
Short Term Memory ◽  
Named Entity Recognition ◽  
Biomedical Literature ◽  
Entity Recognition ◽  
Named Entity ◽  
Learning Framework ◽  
Part Of Speech ◽  
Interaction Extraction ◽  
Memory Network

Abstract Background Microbes have been shown to play a crucial role in various ecosystems. Many human diseases have been proved to be associated with bacteria, so it is essential to extract the interaction between bacteria for medical research and application. At the same time, many bacterial interactions with certain experimental evidences have been reported in biomedical literature. Integrating this knowledge into a database or knowledge graph could accelerate the progress of biomedical research. A crucial and necessary step in interaction extraction (IE) is named entity recognition (NER). However, due to the specificity of bacterial naming, there are still challenges in bacterial named entity recognition. Results In this paper, we propose a novel method for bacterial named entity recognition, which integrates domain features into a deep learning framework combining bidirectional long short-term memory network and convolutional neural network. When domain features are not added, F1-measure of the model achieves 89.14%. After part-of-speech (POS) features and dictionary features are added, F1-measure of the model achieves 89.7%. Hence, our model achieves an advanced performance in bacterial NER with the domain features. Conclusions We propose an efficient method for bacterial named entity recognition which combines domain features and deep learning models. Compared with the previous methods, the effect of our model has been improved. At the same time, the process of complex manual extraction and feature design are significantly reduced.

scholarly journals Chinese named entity recognition model based on BERT

2021 ◽  
Vol 336 ◽  
pp. 06021
Author(s):  
Hongshuai Liu ◽  
Ge Jun ◽  
Yuanyuan Zheng
Keyword(s):  
Deep Learning ◽  
Language Model ◽  
Named Entity Recognition ◽  
Experimental Results ◽  
Entity Recognition ◽  
Global Information ◽  
Learning Models ◽  
Named Entity ◽  
Whole Word ◽  
Document Level

Nowadays, most deep learning models ignore Chinese habits and global information when processing Chinese tasks. To solve this problem, we constructed the BERT-BiLSTM-Attention-CRF model. In the model, we embeded the BERT pre-training language model that adopts the Whole Word Mask strategy, and added a document-level attention. Experimental results show that our method achieves good results in the MSRA corpus, and F1 reaches 95.00%.

Sign in / Sign up

Export Citation Format

Share Document