Target-oriented Fine-tuning for Zero-Resource Named Entity Recognition

Background. Clinical named entity recognition is the basic task of mining electronic medical records text, which are with some challenges containing the language features of Chinese electronic medical records text with many compound entities, serious missing sentence components, and unclear entity boundary. Moreover, the corpus of Chinese electronic medical records is difficult to obtain. Methods. Aiming at these characteristics of Chinese electronic medical records, this study proposed a Chinese clinical entity recognition model based on deep learning pretraining. The model used word embedding from domain corpus and fine-tuning of entity recognition model pretrained by relevant corpus. Then BiLSTM and Transformer are, respectively, used as feature extractors to identify four types of clinical entities including diseases, symptoms, drugs, and operations from the text of Chinese electronic medical records. Results. 75.06% Macro-P, 76.40% Macro-R, and 75.72% Macro-F1 aiming at test dataset could be achieved. These experiments show that the Chinese clinical entity recognition model based on deep learning pretraining can effectively improve the recognition effect. Conclusions. These experiments show that the proposed Chinese clinical entity recognition model based on deep learning pretraining can effectively improve the recognition performance.

Download Full-text

MenuNER: Domain-Adapted BERT Based NER Approach for a Domain with Limited Dataset and Its Application to Food Menu Domain

Applied Sciences ◽

10.3390/app11136007 ◽

2021 ◽

Vol 11 (13) ◽

pp. 6007

Author(s):

Muzamil Hussain Syed ◽

Sun-Tae Chung

Keyword(s):

Domain Adaptation ◽

Language Model ◽

Named Entity Recognition ◽

Word Embedding ◽

Fine Tuning ◽

Entity Recognition ◽

Language Models ◽

Feature Vectors ◽

Named Entity ◽

Domain Specific

Entity-based information extraction is one of the main applications of Natural Language Processing (NLP). Recently, deep transfer-learning utilizing contextualized word embedding from pre-trained language models has shown remarkable results for many NLP tasks, including Named-entity recognition (NER). BERT (Bidirectional Encoder Representations from Transformers) is gaining prominent attention among various contextualized word embedding models as a state-of-the-art pre-trained language model. It is quite expensive to train a BERT model from scratch for a new application domain since it needs a huge dataset and enormous computing time. In this paper, we focus on menu entity extraction from online user reviews for the restaurant and propose a simple but effective approach for NER task on a new domain where a large dataset is rarely available or difficult to prepare, such as food menu domain, based on domain adaptation technique for word embedding and fine-tuning the popular NER task network model ‘Bi-LSTM+CRF’ with extended feature vectors. The proposed NER approach (named as ‘MenuNER’) consists of two step-processes: (1) Domain adaptation for target domain; further pre-training of the off-the-shelf BERT language model (BERT-base) in semi-supervised fashion on a domain-specific dataset, and (2) Supervised fine-tuning the popular Bi-LSTM+CRF network for downstream task with extended feature vectors obtained by concatenating word embedding from the domain-adapted pre-trained BERT model from the first step, character embedding and POS tag feature information. Experimental results on handcrafted food menu corpus from customers’ review dataset show that our proposed approach for domain-specific NER task, that is: food menu named-entity recognition, performs significantly better than the one based on the baseline off-the-shelf BERT-base model. The proposed approach achieves 92.5% F1 score on the YELP dataset for the MenuNER task.

Download Full-text

Fine-Tuning BERT Model for Materials Named Entity Recognition

10.1109/bigdata52589.2021.9671697 ◽

2021 ◽

Author(s):

Xintong Zhao ◽

Jane Greenberg ◽

Yuan An ◽

Xiaohua Tony Hu

Keyword(s):

Named Entity Recognition ◽

Fine Tuning ◽

Entity Recognition ◽

Named Entity

Download Full-text

Fine Tuning Features and Post-processing Rules to Improve Named Entity Recognition

Natural Language Processing and Information Systems - Lecture Notes in Computer Science ◽

10.1007/11765448_16 ◽

2006 ◽

pp. 176-185

Author(s):

Óscar Ferrández ◽

Antonio Toral ◽

Rafael Muñoz

Keyword(s):

Named Entity Recognition ◽

Fine Tuning ◽

Entity Recognition ◽

Post Processing ◽

Named Entity

Download Full-text

Few-Sample Named Entity Recognition for Security Vulnerability Reports by Fine-Tuning Pre-trained Language Models

10.1007/978-3-030-87839-9_3 ◽

2021 ◽

pp. 55-78

Author(s):

Guanqun Yang ◽

Shay Dineen ◽

Zhipeng Lin ◽

Xueqing Liu

Keyword(s):

Named Entity Recognition ◽

Fine Tuning ◽

Entity Recognition ◽

Language Models ◽

Security Vulnerability ◽

Named Entity

Download Full-text

A Fine-Tuned Bidirectional Encoder Representations From Transformers Model for Food Named-Entity Recognition: Algorithm Development and Validation (Preprint)

10.2196/preprints.28229 ◽

2021 ◽

Author(s):

Riste Stojanov ◽

Gorjan Popovski ◽

Gjorgjina Cenikj ◽

Barbara Koroušić Seljak ◽

Tome Eftimov

Keyword(s):

Machine Learning ◽

Information Extraction ◽

State Of The Art ◽

Named Entity Recognition ◽

Fine Tuning ◽

Entity Recognition ◽

Food Science ◽

Named Entity ◽

Semantic Resources ◽

Systematized Nomenclature Of Medicine

BACKGROUND Recently, food science has been garnering a lot of attention. There are many open research questions on food interactions, as one of the main environmental factors, with other health-related entities such as diseases, treatments, and drugs. In the last 2 decades, a large amount of work has been done in natural language processing and machine learning to enable biomedical information extraction. However, machine learning in food science domains remains inadequately resourced, which brings to attention the problem of developing methods for food information extraction. There are only few food semantic resources and few rule-based methods for food information extraction, which often depend on some external resources. However, an annotated corpus with food entities along with their normalization was published in 2019 by using several food semantic resources. OBJECTIVE In this study, we investigated how the recently published bidirectional encoder representations from transformers (BERT) model, which provides state-of-the-art results in information extraction, can be fine-tuned for food information extraction. METHODS We introduce FoodNER, which is a collection of corpus-based food named-entity recognition methods. It consists of 15 different models obtained by fine-tuning 3 pretrained BERT models on 5 groups of semantic resources: food versus nonfood entity, 2 subsets of Hansard food semantic tags, FoodOn semantic tags, and Systematized Nomenclature of Medicine Clinical Terms food semantic tags. RESULTS All BERT models provided very promising results with 93.30% to 94.31% macro F1 scores in the task of distinguishing food versus nonfood entity, which represents the new state-of-the-art technology in food information extraction. Considering the tasks where semantic tags are predicted, all BERT models obtained very promising results once again, with their macro F1 scores ranging from 73.39% to 78.96%. CONCLUSIONS FoodNER can be used to extract and annotate food entities in 5 different tasks: food versus nonfood entities and distinguishing food entities on the level of food groups by using the closest Hansard semantic tags, the parent Hansard semantic tags, the FoodOn semantic tags, or the Systematized Nomenclature of Medicine Clinical Terms semantic tags.

Download Full-text

Zero-Resource Cross-Lingual Named Entity Recognition

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i05.6237 ◽

2020 ◽

Vol 34 (05) ◽

pp. 7415-7423

Author(s):

M Saiful Bari ◽

Shafiq Joty ◽

Prathyusha Jwalapuram

Keyword(s):

Named Entity Recognition ◽

Training Data ◽

Fine Tuning ◽

Entity Recognition ◽

Adversarial Learning ◽

Named Entity ◽

Word Level ◽

Parallel Data ◽

Feature Augmentation ◽

Cross Lingual

Recently, neural methods have achieved state-of-the-art (SOTA) results in Named Entity Recognition (NER) tasks for many languages without the need for manually crafted features. However, these models still require manually annotated training data, which is not available for many languages. In this paper, we propose an unsupervised cross-lingual NER model that can transfer NER knowledge from one language to another in a completely unsupervised way without relying on any bilingual dictionary or parallel data. Our model achieves this through word-level adversarial learning and augmented fine-tuning with parameter sharing and feature augmentation. Experiments on five different languages demonstrate the effectiveness of our approach, outperforming existing models by a good margin and setting a new SOTA for each language pair.

Download Full-text

HUNER: improving biomedical NER with pretraining

Bioinformatics ◽

10.1093/bioinformatics/btz528 ◽

2019 ◽

Vol 36 (1) ◽

pp. 295-302 ◽

Cited By ~ 2

Author(s):

Leon Weber ◽

Jannes Münchmeyer ◽

Tim Rocktäschel ◽

Maryam Habibi ◽

Ulf Leser

Keyword(s):

Deep Neural Networks ◽

State Of The Art ◽

Named Entity Recognition ◽

The State ◽

Fine Tuning ◽

Entity Recognition ◽

Supplementary Information ◽

Named Entity ◽

Training Development ◽

The Impact

Abstract Motivation Several recent studies showed that the application of deep neural networks advanced the state-of-the-art in named entity recognition (NER), including biomedical NER. However, the impact on performance and the robustness of improvements crucially depends on the availability of sufficiently large training corpora, which is a problem in the biomedical domain with its often rather small gold standard corpora. Results We evaluate different methods for alleviating the data sparsity problem by pretraining a deep neural network (LSTM-CRF), followed by a rather short fine-tuning phase focusing on a particular corpus. Experiments were performed using 34 different corpora covering five different biomedical entity types, yielding an average increase in F1-score of ∼2 pp compared to learning without pretraining. We experimented both with supervised and semi-supervised pretraining, leading to interesting insights into the precision/recall trade-off. Based on our results, we created the stand-alone NER tool HUNER incorporating fully trained models for five entity types. On the independent CRAFT corpus, which was not used for creating HUNER, it outperforms the state-of-the-art tools GNormPlus and tmChem by 5–13 pp on the entity types chemicals, species and genes. Availability and implementation HUNER is freely available at https://hu-ner.github.io. HUNER comes in containers, making it easy to install and use, and it can be applied off-the-shelf to arbitrary texts. We also provide an integrated tool for obtaining and converting all 34 corpora used in our evaluation, including fixed training, development and test splits to enable fair comparisons in the future. Supplementary information Supplementary data are available at Bioinformatics online.

Download Full-text

Low-Resource Named Entity Recognition via the Pre-Training Model

Symmetry ◽

10.3390/sym13050786 ◽

2021 ◽

Vol 13 (5) ◽

pp. 786

Author(s):

Siqi Chen ◽

Yijie Pei ◽

Zunwang Ke ◽

Wushour Silamu

Keyword(s):

Data Augmentation ◽

Language Model ◽

Named Entity Recognition ◽

Name Entity Recognition ◽

Fine Tuning ◽

Entity Recognition ◽

Language Models ◽

Low Resource ◽

Named Entity ◽

High Resource

Named entity recognition (NER) is an important task in the processing of natural language, which needs to determine entity boundaries and classify them into pre-defined categories. For low-resource languages, most state-of-the-art systems require tens of thousands of annotated sentences to obtain high performance. However, there is minimal annotated data available about Uyghur and Hungarian (UH languages) NER tasks. There are also specificities in each task—differences in words and word order across languages make it a challenging problem. In this paper, we present an effective solution to providing a meaningful and easy-to-use feature extractor for named entity recognition tasks: fine-tuning the pre-trained language model. Therefore, we propose a fine-tuning method for a low-resource language model, which constructs a fine-tuning dataset through data augmentation; then the dataset of a high-resource language is added; and finally the cross-language pre-trained model is fine-tuned on this dataset. In addition, we propose an attention-based fine-tuning strategy that uses symmetry to better select relevant semantic and syntactic information from pre-trained language models and apply these symmetry features to name entity recognition tasks. We evaluated our approach on Uyghur and Hungarian datasets, which showed wonderful performance compared to some strong baselines. We close with an overview of the available resources for named entity recognition and some of the open research questions.

Download Full-text

Emerging trends: Deep nets for poets

Natural Language Engineering ◽

10.1017/s1351324921000231 ◽

2021 ◽

Vol 27 (5) ◽

pp. 631-645

Author(s):

Kenneth Ward Church ◽

Xiaopeng Yuan ◽

Sheng Guo ◽

Zewu Wu ◽

Yehua Yang ◽

...

Keyword(s):

Character Recognition ◽

Optical Character Recognition ◽

Question Answering ◽

Named Entity Recognition ◽

Fine Tuning ◽

Entity Recognition ◽

Training Models ◽

Named Entity ◽

Emerging Trends ◽

Programming Skills

AbstractDeep nets have done well with early adopters, but the future will soon depend on crossing the chasm. The goal of this paper is to make deep nets more accessible to a broader audience including people with little or no programming skills, and people with little interest in training new models. A github is provided with simple implementations of image classification, optical character recognition, sentiment analysis, named entity recognition, question answering (QA/SQuAD), machine translation, speech to text (SST), and speech recognition (STT). The emphasis is on instant gratification. Non-programmers should be able to install these programs and use them in 15 minutes or less (per program). Programs are short (10–100 lines each) and readable by users with modest programming skills. Much of the complexity is hidden behind abstractions such as pipelines and auto classes, and pretrained models and datasets provided by hubs: PaddleHub, PaddleNLP, HuggingFaceHub, and Fairseq. Hubs have different priorities than research. Research is training models from corpora and fine-tuning them for tasks. Users are already overwhelmed with an embarrassment of riches (13k models and 1k datasets). Do they want more? We believe the broader market is more interested in inference (how to run pretrained models on novel inputs) and less interested in training (how to create even more models).

Download Full-text