A New Modeling Method Base on Candidate Window for Clinical Concept Extraction

Abstract Background Recently, how to structuralize electronic medical records (EMRs) has attracted considerable attention from researchers. Extracting clinical concepts from EMRs is a critical part of EMR structuralization. The performance of clinical concept extraction will directly affect the performance of the downstream tasks related to EMR structuralization. We propose a new modeling method based on candidate window classification, which is different from mainstream sequence labeling models, to improves the performance of clinical concept extraction tasks under strict standards by considering the overall semantics of the token sequence instead of the semantics of each token. We call this model as slide window model. Method In this paper, we comprehensively study the performance of the slide window model in clinical concept extraction tasks. We model the clinical concept extraction task as the task of classifying each candidate window, which was extracted by the slide window. The proposed model mainly consists of four parts. First, the pre-trained language model is used to generate the context-sensitive token representation. Second, a convolutional neural network (CNN) is used to generate all representation vector of the candidate windows in the sentence. Third, every candidate window is classified by a Softmax classifier to obtain concept type probability distribution. Finally, the knapsack algorithm is used as a post-process to maximize the sum of disjoint clinical concepts scores and filter the clinical concepts. Results Experiments show that the slide window model achieves the best micro-average F1 score(81.22%) on the corpora of the 2012 i2b2 NLP challenges and achieves 89.25% F1 score on the 2010 i2b2 NLP challenges under the strict standard. Furthermore, the performance of our approach is always better than the BiLSTM-CRF model and softmax classifier with the same pre-trained language model.Conclusions The slide window model shows a new modeling method for solving clinical concept extraction tasks. It models clinical concept extraction as a problem for classifying candidate windows and extracts clinical concepts by considering the semantics of the entire candidate window. Experiments show that this method of considering the overall semantics of the candidate window can improve the performance of clinical concept extraction tasks under strict standards.

Download Full-text

A New Modeling Method Base on Candidate Window for Clinical Concept Extraction

10.21203/rs.2.22462/v1 ◽

2020 ◽

Author(s):

Yongtao Tang ◽

Shasha Li ◽

Bin Ji ◽

Jie Yu ◽

Yusong Tan ◽

...

Keyword(s):

Language Model ◽

Modeling Method ◽

Softmax Classifier ◽

Concept Extraction ◽

Context Sensitive ◽

Post Process ◽

Proposed Model ◽

Knapsack Algorithm ◽

Clinical Concept ◽

Candidate Window

Abstract Background Recently, how to structuralize electronic medical records (EMRs) has attracted considerable attention from researchers. Extracting clinical concepts from EMRs is a critical part of EMR structuralization. The performance of clinical concept extraction will directly affect the performance of the downstream tasks related to EMR structuralization. We propose a new modeling method based on candidate window classification, which is different from mainstream sequence labeling models, to improves the performance of clinical concept extraction tasks under strict standards by considering the overall semantics of the token sequence instead of the semantics of each token. We call this model as slide window model. MethodIn this paper, we comprehensively study the performance of the slide window model in clinical concept extraction tasks. We model the clinical concept extraction task as the task of classifying each candidate window, which was extracted by the slide window. The proposed model mainly consists of four parts. First, the pre-trained language model is used to generate the context-sensitive token representation. Second, a convolutional neural network (CNN) is used to generate all representation vector of the candidate windows in the sentence. Third, every candidate window is classified by a Softmax classifier to obtain concept type probability distribution. Finally, the knapsack algorithm is used as a post-process to maximize the sum of disjoint clinical concepts scores and filter the clinical concepts. Results Experiments show that the slide window model achieves the best micro-average F1 score(81.22%) on the corpora of the 2012 i2b2 NLP challenges and achieves 89.25% F1 score on the 2010 i2b2 NLP challenges under the strict standard. Furthermore, the performance of our approach is always better than the BiLSTM-CRF model and softmax classifier with the same pre-trained language model. ConclusionsThe slide window model shows a new modeling method for solving clinical concept extraction tasks. It models clinical concept extraction as a problem for classifying candidate windows and extracts clinical concepts by considering the semantics of the entire candidate window. Experiments show that this method of considering the overall semantics of the candidate window can improve the performance of clinical concept extraction tasks under strict standards.

Download Full-text

Enhancing clinical concept extraction with contextual embeddings

Journal of the American Medical Informatics Association ◽

10.1093/jamia/ocz096 ◽

2019 ◽

Vol 26 (11) ◽

pp. 1297-1304 ◽

Cited By ~ 29

Author(s):

Yuqi Si ◽

Jingqi Wang ◽

Hua Xu ◽

Kirk Roberts

Keyword(s):

Language Processing ◽

Semantic Information ◽

Medical Information ◽

State Of The Art ◽

Language Model ◽

The State ◽

Concept Extraction ◽

The Impact ◽

Clinical Concept ◽

Embedding Methods

Abstract Objective Neural network–based representations (“embeddings”) have dramatically advanced natural language processing (NLP) tasks, including clinical NLP tasks such as concept extraction. Recently, however, more advanced embedding methods and representations (eg, ELMo, BERT) have further pushed the state of the art in NLP, yet there are no common best practices for how to integrate these representations into clinical tasks. The purpose of this study, then, is to explore the space of possible options in utilizing these new models for clinical concept extraction, including comparing these to traditional word embedding methods (word2vec, GloVe, fastText). Materials and Methods Both off-the-shelf, open-domain embeddings and pretrained clinical embeddings from MIMIC-III (Medical Information Mart for Intensive Care III) are evaluated. We explore a battery of embedding methods consisting of traditional word embeddings and contextual embeddings and compare these on 4 concept extraction corpora: i2b2 2010, i2b2 2012, SemEval 2014, and SemEval 2015. We also analyze the impact of the pretraining time of a large language model like ELMo or BERT on the extraction performance. Last, we present an intuitive way to understand the semantic information encoded by contextual embeddings. Results Contextual embeddings pretrained on a large clinical corpus achieves new state-of-the-art performances across all concept extraction tasks. The best-performing model outperforms all state-of-the-art methods with respective F1-measures of 90.25, 93.18 (partial), 80.74, and 81.65. Conclusions We demonstrate the potential of contextual embeddings through the state-of-the-art performance these methods achieve on clinical concept extraction. Additionally, we demonstrate that contextual embeddings encode valuable semantic information not accounted for in traditional word representations.

Download Full-text

Hierarchical Concept-Driven Language Model

ACM Transactions on Knowledge Discovery from Data ◽

10.1145/3451167 ◽

2021 ◽

Vol 15 (6) ◽

pp. 1-22

Author(s):

Yashen Wang ◽

Huanhuan Zhang ◽

Zhirun Liu ◽

Qiang Zhou

Keyword(s):

Language Model ◽

Generation Process ◽

Data Generation ◽

Modeling Framework ◽

Long Distance ◽

Short Text ◽

Proposed Model ◽

Scalable Inference ◽

End To End ◽

Hidden Layer

For guiding natural language generation, many semantic-driven methods have been proposed. While clearly improving the performance of the end-to-end training task, these existing semantic-driven methods still have clear limitations: for example, (i) they only utilize shallow semantic signals (e.g., from topic models) with only a single stochastic hidden layer in their data generation process, which suffer easily from noise (especially adapted for short-text etc.) and lack of interpretation; (ii) they ignore the sentence order and document context, as they treat each document as a bag of sentences, and fail to capture the long-distance dependencies and global semantic meaning of a document. To overcome these problems, we propose a novel semantic-driven language modeling framework, which is a method to learn a Hierarchical Language Model and a Recurrent Conceptualization-enhanced Gamma Belief Network, simultaneously. For scalable inference, we develop the auto-encoding Variational Recurrent Inference, allowing efficient end-to-end training and simultaneously capturing global semantics from a text corpus. Especially, this article introduces concept information derived from high-quality lexical knowledge graph Probase, which leverages strong interpretability and anti-nose capability for the proposed model. Moreover, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence concept dependence. Experiments conducted on several NLP tasks validate the superiority of the proposed approach, which could effectively infer meaningful hierarchical concept structure of document and hierarchical multi-scale structures of sequences, even compared with latest state-of-the-art Transformer-based models.

Download Full-text

Factors related to sense of competence in family caregivers of people living with dementia in the community: a narrative synthesis

International Psychogeriatrics ◽

10.1017/s1041610218001394 ◽

2018 ◽

Vol 31 (06) ◽

pp. 799-813 ◽

Cited By ~ 1

Author(s):

Jacki Stansfeld ◽

Nadia Crellin ◽

Martin Orrell ◽

Jennifer Wenborn ◽

Georgina Charlesworth ◽

...

Keyword(s):

Psychological Symptoms ◽

Future Research ◽

Narrative Synthesis ◽

Review Of The Literature ◽

Factors Associated ◽

Sense Of Competence ◽

Proposed Model ◽

Quality Rating ◽

Clinical Concept ◽

Behavioral And Psychological Symptoms

ABSTRACTObjectives:Sense of competence defines a caregiver’s feeling of being capable to manage the caregiving task and is an important clinical concept in the caregiving literature. The aim of this review was to identify the factors, both positive and negative, associated with a caregiver’s perception of their sense of competence.Design:A systematic review of the literature was conducted, retrieving both quantitative and qualitative papers from databases PsycINFO, CINAHL, EMBASE, and Medline. A quality assessment was conducted using the STROBE and CASP checklists, and the quality rating informed the inclusion of papers ensuring the evidence was robust. Narrative synthesis was employed to synthesize the findings and to generate an updated conceptual model of sense of competence.Results:Seventeen papers were included in the review, all of which were moderate to high quality. These included 13 quantitative, three mixed-methods and one qualitative study. Factors associated with sense of competence included: behavioral and psychological symptoms of dementia (BPSD), caregiver depression, gratitude, and the ability to find meaning in caregiving.Conclusions:The results of this review demonstrate that both positive and negative aspects of caring are associated with caregiver sense of competence. Positive and negative aspects of caregiving act in tandem to influence caregiver perception of their competence. The proposed model of sense of competence aims to guide future research and clinical interventions aimed at improving this domain but requires further testing, as due to the observational nature of the include papers, the direction of causality could not be inferred.

Download Full-text

ACCURATE ANALYSIS OF GLOBAL INTERCONNECTS IN NANO-FPGAs

NANO ◽

10.1142/s1793292009001629 ◽

2009 ◽

Vol 04 (03) ◽

pp. 171-176 ◽

Cited By ~ 2

Author(s):

DAVOOD FATHI ◽

BEHJAT FOROUZANDEH

Keyword(s):

Propagation Delay ◽

Modeling Method ◽

Accurate Analysis ◽

Proposed Model ◽

Simulation Results ◽

The Difference ◽

A New Technique ◽

Global Interconnects ◽

Technology Nodes ◽

Model Technique

This paper introduces a new technique for analyzing the behavior of global interconnects in FPGAs, for nanoscale technologies. Using this new enhanced modeling method, new enhanced accurate expressions for calculating the propagation delay of global interconnects in nano-FPGAs have been derived. In order to verify the proposed model, we have performed the delay simulations in 45 nm, 65 nm, 90 nm, and 130 nm technology nodes, with our modeling method and the conventional Pi-model technique. Then, the results obtained from these two methods have been compared with HSPICE simulation results. The obtained results show a better match in the propagation delay computations for global interconnects between our proposed model and HSPICE simulations, with respect to the conventional techniques such as Pi-model. According to the obtained results, the difference between our model and HSPICE simulations in the mentioned technology nodes is (0.29–22.92)%, whereas this difference is (11.13–38.29)% for another model.

Download Full-text

Clinical Concept Extraction for Document-Level Coding

10.18653/v1/w19-5028 ◽

2019 ◽

Author(s):

Sarah Wiegreffe ◽

Edward Choi ◽

Sherry Yan ◽

Jimeng Sun ◽

Jacob Eisenstein

Keyword(s):

Concept Extraction ◽

Clinical Concept ◽

Document Level

Download Full-text

Automatic Title Generation for Learning Resources and Pathways with Pre-trained Transformer Models

International Journal of Semantic Computing ◽

10.1142/s1793351x21400134 ◽

2021 ◽

Vol 15 (04) ◽

pp. 487-510

Author(s):

Prakhar Mishra ◽

Chaitali Diwan ◽

Srinath Srinivasa ◽

G. Srinivasaraghavan

Keyword(s):

Online Learning ◽

Language Model ◽

Research Paper ◽

Learning Resources ◽

Research Papers ◽

Medium Length ◽

Proposed Model ◽

Learning Pathways ◽

Test Sets ◽

Transformer Model

To create curiosity and interest for a topic in online learning is a challenging task. A good preview that outlines the contents of a learning pathway could help learners know the topic and get interested in it. Towards this end, we propose a hierarchical title generation approach to generate semantically relevant titles for the learning resources in a learning pathway and a title for the pathway itself. Our approach to Automatic Title Generation for a given text is based on pre-trained Transformer Language Model GPT-2. A pool of candidate titles are generated and an appropriate title is selected among them which is then refined or de-noised to get the final title. The model is trained on research paper abstracts from arXiv and evaluated on three different test sets. We show that it generates semantically and syntactically relevant titles as reflected in ROUGE, BLEU scores and human evaluations. We propose an optional abstractive Summarizer module based on pre-trained Transformer model T5 to shorten medium length documents. This module is also trained and evaluated on research papers from arXiv dataset. Finally, we show that the proposed model of hierarchical title generation for learning pathways has promising results.

Download Full-text

Span Classification Based Model for Clinical Concept Extraction

Advances in Natural Computation, Fuzzy Systems and Knowledge Discovery - Advances in Intelligent Systems and Computing ◽

10.1007/978-3-030-70665-4_203 ◽

2021 ◽

pp. 1880-1889

Author(s):

Yongtao Tang ◽

Jie Yu ◽

Shasha Li ◽

Bin Ji ◽

Yusong Tan ◽

...

Keyword(s):

Concept Extraction ◽

Clinical Concept

Download Full-text

A Quantum Expectation Value Based Language Model with Application to Question Answering

Entropy ◽

10.3390/e22050533 ◽

2020 ◽

Vol 22 (5) ◽

pp. 533

Author(s):

Qin Zhao ◽

Chenguang Hou ◽

Changjian Liu ◽

Peng Zhang ◽

Ruifeng Xu

Keyword(s):

Hilbert Space ◽

Density Matrix ◽

Question Answering ◽

Language Model ◽

Language Models ◽

Quantum Model ◽

Expectation Value ◽

Proposed Model ◽

Matching Score ◽

The Relationship

Quantum-inspired language models have been introduced to Information Retrieval due to their transparency and interpretability. While exciting progresses have been made, current studies mainly investigate the relationship between density matrices of difference sentence subspaces of a semantic Hilbert space. The Hilbert space as a whole which has a unique density matrix is lack of exploration. In this paper, we propose a novel Quantum Expectation Value based Language Model (QEV-LM). A unique shared density matrix is constructed for the Semantic Hilbert Space. Words and sentences are viewed as different observables in this quantum model. Under this background, a matching score describing the similarity between a question-answer pair is naturally explained as the quantum expectation value of a joint question-answer observable. In addition to the theoretical soundness, experiment results on the TREC-QA and WIKIQA datasets demonstrate the computational efficiency of our proposed model with excellent performance and low time consumption.

Download Full-text