Cost-aware active learning for named entity recognition in clinical text

Abstract Objective Active Learning (AL) attempts to reduce annotation cost (ie, time) by selecting the most informative examples for annotation. Most approaches tacitly (and unrealistically) assume that the cost for annotating each sample is identical. This study introduces a cost-aware AL method, which simultaneously models both the annotation cost and the informativeness of the samples and evaluates both via simulation and user studies. Materials and Methods We designed a novel, cost-aware AL algorithm (Cost-CAUSE) for annotating clinical named entities; we first utilized lexical and syntactic features to estimate annotation cost, then we incorporated this cost measure into an existing AL algorithm. Using the 2010 i2b2/VA data set, we then conducted a simulation study comparing Cost-CAUSE with noncost-aware AL methods, and a user study comparing Cost-CAUSE with passive learning. Results Our cost model fit empirical annotation data well, and Cost-CAUSE increased the simulation area under the learning curve (ALC) scores by up to 5.6% and 4.9%, compared with random sampling and alternate AL methods. Moreover, in a user annotation task, Cost-CAUSE outperformed passive learning on the ALC score and reduced annotation time by 20.5%–30.2%. Discussion Although AL has proven effective in simulations, our user study shows that a real-world environment is far more complex. Other factors have a noticeable effect on the AL method, such as the annotation accuracy of users, the tiredness of users, and even the physical and mental condition of users. Conclusion Cost-CAUSE saves significant annotation cost compared to random sampling.

Download Full-text

Text Mining of Hazard and Operability Analysis Reports Based on Active Learning

Processes ◽

10.3390/pr9071178 ◽

2021 ◽

Vol 9 (7) ◽

pp. 1178

Author(s):

Zhenhua Wang ◽

Beike Zhang ◽

Dong Gao

Keyword(s):

Active Learning ◽

Named Entity Recognition ◽

Entity Recognition ◽

Chemical System ◽

Chemical Safety ◽

High Quality ◽

Data Set ◽

Final Model ◽

Named Entity ◽

Sampling Algorithms

In the field of chemical safety, a named entity recognition (NER) model based on deep learning can mine valuable information from hazard and operability analysis (HAZOP) text, which can guide experts to carry out a new round of HAZOP analysis, help practitioners optimize the hidden dangers in the system, and be of great significance to improve the safety of the whole chemical system. However, due to the standardization and professionalism of chemical safety analysis text, it is difficult to improve the performance of traditional models. To solve this problem, in this study, an improved method based on active learning is proposed, and three novel sampling algorithms are designed, Variation of Token Entropy (VTE), HAZOP Confusion Entropy (HCE) and Amplification of Least Confidence (ALC), which improve the ability of the model to understand HAZOP text. In this method, a part of data is used to establish the initial model. The sampling algorithm is then used to select high-quality samples from the data set. Finally, these high-quality samples are used to retrain the whole model to obtain the final model. The experimental results show that the performance of the VTE, HCE, and ALC algorithms are better than that of random sampling algorithms. In addition, compared with other methods, the performance of the traditional model is improved effectively by the method proposed in this paper, which proves that the method is reliable and advanced.

Download Full-text

Active learning for ontological event extraction incorporating named entity recognition and unknown word handling

Journal of Biomedical Semantics ◽

10.1186/s13326-016-0059-z ◽

2016 ◽

Vol 7 (1) ◽

Cited By ~ 2

Author(s):

Xu Han ◽

Jung-jae Kim ◽

Chee Keong Kwoh

Keyword(s):

Active Learning ◽

Named Entity Recognition ◽

Event Extraction ◽

Entity Recognition ◽

Unknown Word ◽

Named Entity

Download Full-text

LTP: A New Active Learning Strategy for CRF-Based Named Entity Recognition

Neural Processing Letters ◽

10.1007/s11063-021-10737-x ◽

2022 ◽

Author(s):

Mingyi Liu ◽

Zhiying Tu ◽

Tong Zhang ◽

Tonghua Su ◽

Xiaofei Xu ◽

...

Keyword(s):

Active Learning ◽

Learning Strategy ◽

Named Entity Recognition ◽

Entity Recognition ◽

Named Entity ◽

Active Learning Strategy

Download Full-text

ChemTok: A New Rule Based Tokenizer for Chemical Named Entity Recognition

BioMed Research International ◽

10.1155/2016/4248026 ◽

2016 ◽

Vol 2016 ◽

pp. 1-9 ◽

Cited By ~ 5

Author(s):

Abbas Akkasi ◽

Ekrem Varoğlu ◽

Nazife Dimililer

Keyword(s):

Conditional Random Fields ◽

Named Entity Recognition ◽

Classification Performance ◽

Entity Recognition ◽

Support Vector ◽

Learning Approaches ◽

Data Set ◽

Rule Based ◽

Named Entity ◽

Vector Machines

Named Entity Recognition (NER) from text constitutes the first step in many text mining applications. The most important preliminary step for NER systems using machine learning approaches is tokenization where raw text is segmented into tokens. This study proposes an enhanced rule based tokenizer, ChemTok, which utilizes rules extracted mainly from the train data set. The main novelty of ChemTok is the use of the extracted rules in order to merge the tokens split in the previous steps, thus producing longer and more discriminative tokens. ChemTok is compared to the tokenization methods utilized by ChemSpot and tmChem. Support Vector Machines and Conditional Random Fields are employed as the learning algorithms. The experimental results show that the classifiers trained on the output of ChemTok outperforms all classifiers trained on the output of the other two tokenizers in terms of classification performance, and the number of incorrectly segmented entities.

Download Full-text

Fine-Grained Mechanical Chinese Named Entity Recognition Based on ALBERT-AttBiLSTM-CRF and Transfer Learning

Symmetry ◽

10.3390/sym12121986 ◽

2020 ◽

Vol 12 (12) ◽

pp. 1986

Author(s):

Liguo Yao ◽

Haisong Huang ◽

Kuan-Wei Wang ◽

Shih-Huan Chen ◽

Qiaoqiao Xiong

Keyword(s):

Active Learning ◽

Manufacturing Industry ◽

Learning Strategy ◽

Named Entity Recognition ◽

Entity Recognition ◽

Utilization Rate ◽

Data Types ◽

Fine Grained ◽

Named Entity ◽

Model Transfer

Manufacturing text often exists as unlabeled data; the entity is fine-grained and the extraction is difficult. The above problems mean that the manufacturing industry knowledge utilization rate is low. This paper proposes a novel Chinese fine-grained NER (named entity recognition) method based on symmetry lightweight deep multinetwork collaboration (ALBERT-AttBiLSTM-CRF) and model transfer considering active learning (MTAL) to research fine-grained named entity recognition of a few labeled Chinese textual data types. The method is divided into two stages. In the first stage, the ALBERT-AttBiLSTM-CRF was applied for verification in the CLUENER2020 dataset (Public dataset) to get a pretrained model; the experiments show that the model obtains an F1 score of 0.8962, which is better than the best baseline algorithm, an improvement of 9.2%. In the second stage, the pretrained model was transferred into the Manufacturing-NER dataset (our dataset), and we used the active learning strategy to optimize the model effect. The final F1 result of Manufacturing-NER was 0.8931 after the model transfer (it was higher than 0.8576 before the model transfer); so, this method represents an improvement of 3.55%. Our method effectively transfers the existing knowledge from public source data to scientific target data, solving the problem of named entity recognition with scarce labeled domain data, and proves its effectiveness.

Download Full-text

Active Learning Approaches for Labeling Text: Review and Assessment of the Performance of Active Learning Approaches

Political Analysis ◽

10.1017/pan.2020.4 ◽

2020 ◽

Vol 28 (4) ◽

pp. 532-551

Author(s):

Blake Miller ◽

Fridolin Linder ◽

Walter R. Mebane

Keyword(s):

Machine Learning ◽

Active Learning ◽

Random Sampling ◽

Supervised Machine Learning ◽

Learning Approaches ◽

Simulation Studies ◽

Text Data ◽

Passive Learning ◽

Machine Learning Model ◽

The Cost

Supervised machine learning methods are increasingly employed in political science. Such models require costly manual labeling of documents. In this paper, we introduce active learning, a framework in which data to be labeled by human coders are not chosen at random but rather targeted in such a way that the required amount of data to train a machine learning model can be minimized. We study the benefits of active learning using text data examples. We perform simulation studies that illustrate conditions where active learning can reduce the cost of labeling text data. We perform these simulations on three corpora that vary in size, document length, and domain. We find that in cases where the document class of interest is not balanced, researchers can label a fraction of the documents one would need using random sampling (or “passive” learning) to achieve equally performing classifiers. We further investigate how varying levels of intercoder reliability affect the active learning procedures and find that even with low reliability, active learning performs more efficiently than does random sampling.

Download Full-text

Using error decay prediction to overcome practical issues of deep active learning for named entity recognition

Machine Learning ◽

10.1007/s10994-020-05897-1 ◽

2020 ◽

Vol 109 (9-10) ◽

pp. 1749-1778

Author(s):

Haw-Shiuan Chang ◽

Shankar Vembu ◽

Sunil Mohan ◽

Rheeya Uppaal ◽

Andrew McCallum

Keyword(s):

Active Learning ◽

Named Entity Recognition ◽

Entity Recognition ◽

Named Entity

Download Full-text

A Low-Cost Named Entity Recognition Research Based on Active Learning

Scientific Programming ◽

10.1155/2018/1890683 ◽

2018 ◽

Vol 2018 ◽

pp. 1-10 ◽

Cited By ~ 1

Author(s):

Han Huang ◽

Hongyu Wang ◽

Dawei Jin

Keyword(s):

Active Learning ◽

Language Processing ◽

Selection Process ◽

Conditional Random Field ◽

Low Cost ◽

Named Entity Recognition ◽

Entity Recognition ◽

Training Set ◽

Processing Technologies ◽

Named Entity

Named entity recognition (NER) is an indispensable and very important part of many natural language processing technologies, such as information extraction, information retrieval, and intelligent Q & A. This paper describes the development of the AL-CRF model, which is a NER approach based on active learning (AL). The algorithmic sequence of the processes performed by the AL-CRF model is the following: first, the samples are clustered using the k-means approach. Then, stratified sampling is performed on the produced clusters in order to obtain initial samples, which are used to train the basic conditional random field (CRF) classifier. The next step includes the initiation of the selection process which uses the criterion of entropy. More specifically, samples having the highest entropy values are added to the training set. Afterwards, the learning process is repeated, and the CRF classifier is retrained based on the obtained training set. The learning and the selection process of the AL is running iteratively until the harmonic mean F stabilizes and the final NER model is obtained. Several NER experiments are performed on legislative and medical cases in order to validate the AL-CRF performance. The testing data include Chinese judicial documents and Chinese electronic medical records (EMRs). Testing indicates that our proposed algorithm has better recognition accuracy and recall rate compared to the conventional CRF model. Moreover, the main advantage of our approach is that it requires fewer manually labelled training samples, and at the same time, it is more effective. This can result in a more cost effective and more reliable process.

Download Full-text