scholarly journals Incorporating Domain Knowledge Into Language Models Using Graph Convolutional Networks for Clinical Semantic Textual Similarity (Preprint)

2020 ◽  
Author(s):  
David Chang ◽  
Eric Lin ◽  
Cynthia Brandt ◽  
Richard Andrew Taylor
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Domain Knowledge ◽  
Data Augmentation ◽  
Pearson Correlation ◽  
Clinical Documentation ◽  
Convolutional Networks ◽  
Knowledge Distillation ◽  
Semantic Textual Similarity

BACKGROUND While electronic health record systems have facilitated clinical documentation in healthcare, they also introduce new challenges such as the proliferation of redundant information through copy-and-paste commands or templates. One approach to trim down bloated clinical documentation and improve clinical summarization is to identify highly similar text snippets for the goal of removing such text. OBJECTIVE We develop a natural language processing system for the task of clinical semantic textual similarity that assigns scores to pairs of clinical text snippets based on their clinical semantic similarity. METHODS We leverage recent advances in natural language processing and graph representation learning to create a model that combines linguistic and domain knowledge information from the MedSTS dataset to assess clinical semantic textual similarity. We use Bidirectional Encoder Representation from Transformers (BERT)¬–based models as text encoders for the sentence pairs in the dataset and graph convolutional networks (GCNs) as graph encoders for corresponding concept graphs constructed based on the sentences. We also explore techniques including data augmentation, ensembling, and knowledge distillation to improve the performance as measured by Pearson correlation. RESULTS Fine–tuning BERT-base and ClinicalBERT on the MedSTS dataset provided a strong baseline (0.842 and 0.848 Pearson correlation, respectively) compared to the previous year’s submissions. Our data augmentation techniques yielded moderate gains in performance, and adding a GCN–based graph encoder to incorporate the concept graphs also boosted performance, especially when the node features were initialized with pretrained knowledge graph embeddings of the concepts (0.868). As expected, ensembling improved performance, and multi–source ensembling using different language model variants, conducting knowledge distillation on the multi–source ensemble model, and taking a final ensemble of the distilled models further improved the system’s performance (0.875, 0.878, and 0.882, respectively). CONCLUSIONS We develop a system for the MedSTS clinical semantic textual similarity benchmark task by combining BERT–based text encoders and GCN–based graph encoders in order to incorporate domain knowledge into the natural language processing pipeline. We also experiment with other techniques involving data augmentation, pretrained concept embeddings, ensembling, and knowledge distillation to further increase our performance.

scholarly journals Natural language processing methods for knowledge management—Applying document clustering for fast search and grouping of engineering documents

2021 ◽  
pp. 1063293X2098297
Author(s):  
Ivar Örn Arnarsson ◽  
Otto Frost ◽  
Emil Gustavsson ◽  
Mats Jirstrand ◽  
Johan Malmqvist
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Domain Knowledge ◽  
Clustering Algorithms ◽  
Document Clustering ◽  
Unstructured Data ◽  
Free Text ◽  
Engineering Change ◽  
Engineering Documents

Product development companies collect data in form of Engineering Change Requests for logged design issues, tests, and product iterations. These documents are rich in unstructured data (e.g. free text). Previous research affirms that product developers find that current IT systems lack capabilities to accurately retrieve relevant documents with unstructured data. In this research, we demonstrate a method using Natural Language Processing and document clustering algorithms to find structurally or contextually related documents from databases containing Engineering Change Request documents. The aim is to radically decrease the time needed to effectively search for related engineering documents, organize search results, and create labeled clusters from these documents by utilizing Natural Language Processing algorithms. A domain knowledge expert at the case company evaluated the results and confirmed that the algorithms we applied managed to find relevant document clusters given the queries tested.

scholarly journals 1358. Using natural language processing to optimize case ascertainment of acute otitis media in a large, state-wide pediatric practice network

2020 ◽  
Vol 7 (Supplement_1) ◽  
pp. S690-S691
Author(s):  
Joshua C Herigon ◽  
Amir Kimia ◽  
Marvin Harper
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Otitis Media ◽  
Language Processing ◽  
Validation Cohort ◽  
Recursive Partitioning ◽  
Acute Otitis Media ◽  
Clinical Documentation ◽  
Training Cohort ◽  
Icd Codes

Abstract Background Antibiotics are the most commonly prescribed drugs for children and frequently inappropriately prescribed. Outpatient antimicrobial stewardship interventions aim to reduce inappropriate antibiotic use. Previous work has relied on diagnosis coding for case identification which may be inaccurate. In this study, we sought to develop automated methods for analyzing note text to identify cases of acute otitis media (AOM) based on clinical documentation. Methods We conducted a cross-sectional retrospective chart review and sampled encounters from 7/1/2018 – 6/30/2019 for patients < 5 years old presenting for a problem-focused visit. Complete note text and limited structured data were extracted for 12 randomly selected weekdays (one from each month during the study period). An additional weekday was randomly selected for validation. The primary outcome was correctly identifying encounters where AOM was present. Human review was considered the “gold standard” and was compared to ICD codes, a natural language processing (NLP) model, and a recursive partitioning (RP) model. Results A total of 2,724 encounters were included in the training cohort and 793 in the validation cohort. ICD codes and NLP had good performance overall with sensitivity 91.2% and 93.1% respectively in the training cohort. However, NLP had a significant drop-off in performance in the validation cohort (sensitivity: 83.4%). The RP model had the highest sensitivity (97.2% training cohort; 94.1% validation cohort) out of the 3 methods. Figure 1. Details of encounters included in the training and validation cohorts. Table 1. Performance of ICD coding, a natural language processing (NLP) model, and a recursive partitioning (RP) model for identifying cases of acute otitis media (AOM) Conclusion Natural language processing of outpatient pediatric visit documentation can be used successfully to create models accurately identifying cases of AOM based on clinical documentation. Combining NLP and structured data can improve automated case detection, leading to more accurate assessment of antibiotic prescribing practices. These techniques may be valuable in optimizing outpatient antimicrobial stewardship efforts. Disclosures All Authors: No reported disclosures

scholarly journals Linking Free Text Documentation of Functioning and Disability to the ICF With Natural Language Processing

2021 ◽  
Vol 2 ◽  
Author(s):  
Denis Newman-Griffis ◽  
Jonathan Camacho Maldonado ◽  
Pei-Shu Ho ◽  
Maryanne Sacco ◽  
Rafael Jimenez Silva ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Functional Status ◽  
Language Processing ◽  
Free Text ◽  
Clinical Documentation ◽  
Disability Benefits ◽  
Domestic Life ◽  
Patient Functioning ◽  
Status Information

Background: Invaluable information on patient functioning and the complex interactions that define it is recorded in free text portions of the Electronic Health Record (EHR). Leveraging this information to improve clinical decision-making and conduct research requires natural language processing (NLP) technologies to identify and organize the information recorded in clinical documentation.Methods: We used natural language processing methods to analyze information about patient functioning recorded in two collections of clinical documents pertaining to claims for federal disability benefits from the U.S. Social Security Administration (SSA). We grounded our analysis in the International Classification of Functioning, Disability, and Health (ICF), and used the Activities and Participation domain of the ICF to classify information about functioning in three key areas: mobility, self-care, and domestic life. After annotating functional status information in our datasets through expert clinical review, we trained machine learning-based NLP models to automatically assign ICF categories to mentions of functional activity.Results: We found that rich and diverse information on patient functioning was documented in the free text records. Annotation of 289 documents for Mobility information yielded 2,455 mentions of Mobility activities and 3,176 specific actions corresponding to 13 ICF-based categories. Annotation of 329 documents for Self-Care and Domestic Life information yielded 3,990 activity mentions and 4,665 specific actions corresponding to 16 ICF-based categories. NLP systems for automated ICF coding achieved over 80% macro-averaged F-measure on both datasets, indicating strong performance across all ICF categories used.Conclusions: Natural language processing can help to navigate the tradeoff between flexible and expressive clinical documentation of functioning and standardizable data for comparability and learning. The ICF has practical limitations for classifying functional status information in clinical documentation but presents a valuable framework for organizing the information recorded in health records about patient functioning. This study advances the development of robust, ICF-based NLP technologies to analyze information on patient functioning and has significant implications for NLP-powered analysis of functional status information in disability benefits management, clinical care, and research.

scholarly journals Benchmarking Effectiveness and Efficiency of Deep Learning Models for Semantic Textual Similarity in the Clinical Domain: Validation Study

10.2196/27386 ◽  
2021 ◽  
Vol 9 (12) ◽  
pp. e27386
Author(s):  
Qingyu Chen ◽  
Alex Rankine ◽  
Yifan Peng ◽  
Elaheh Aghaarabi ◽  
Zhiyong Lu
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Mean Squared Error ◽  
Pearson Correlation ◽  
Data Set ◽  
Squared Error ◽  
Real Time Applications ◽  
Effectiveness And Efficiency ◽  
Pearson Correlations

Background Semantic textual similarity (STS) measures the degree of relatedness between sentence pairs. The Open Health Natural Language Processing (OHNLP) Consortium released an expertly annotated STS data set and called for the National Natural Language Processing Clinical Challenges. This work describes our entry, an ensemble model that leverages a range of deep learning (DL) models. Our team from the National Library of Medicine obtained a Pearson correlation of 0.8967 in an official test set during 2019 National Natural Language Processing Clinical Challenges/Open Health Natural Language Processing shared task and achieved a second rank. Objective Although our models strongly correlate with manual annotations, annotator-level correlation was only moderate (weighted Cohen κ=0.60). We are cautious of the potential use of DL models in production systems and argue that it is more critical to evaluate the models in-depth, especially those with extremely high correlations. In this study, we benchmark the effectiveness and efficiency of top-ranked DL models. We quantify their robustness and inference times to validate their usefulness in real-time applications. Methods We benchmarked five DL models, which are the top-ranked systems for STS tasks: Convolutional Neural Network, BioSentVec, BioBERT, BlueBERT, and ClinicalBERT. We evaluated a random forest model as an additional baseline. For each model, we repeated the experiment 10 times, using the official training and testing sets. We reported 95% CI of the Wilcoxon rank-sum test on the average Pearson correlation (official evaluation metric) and running time. We further evaluated Spearman correlation, R², and mean squared error as additional measures. Results Using only the official training set, all models obtained highly effective results. BioSentVec and BioBERT achieved the highest average Pearson correlations (0.8497 and 0.8481, respectively). BioSentVec also had the highest results in 3 of 4 effectiveness measures, followed by BioBERT. However, their robustness to sentence pairs of different similarity levels varies significantly. A particular observation is that BERT models made the most errors (a mean squared error of over 2.5) on highly similar sentence pairs. They cannot capture highly similar sentence pairs effectively when they have different negation terms or word orders. In addition, time efficiency is dramatically different from the effectiveness results. On average, the BERT models were approximately 20 times and 50 times slower than the Convolutional Neural Network and BioSentVec models, respectively. This results in challenges for real-time applications. Conclusions Despite the excitement of further improving Pearson correlations in this data set, our results highlight that evaluations of the effectiveness and efficiency of STS models are critical. In future, we suggest more evaluations on the generalization capability and user-level testing of the models. We call for community efforts to create more biomedical and clinical STS data sets from different perspectives to reflect the multifaceted notion of sentence-relatedness.

scholarly journals Using Natural Language Processing and Artificial Intelligence to Explore the Nutrition and Sustainability of Recipes and Food

2021 ◽  
Vol 3 ◽  
Author(s):  
Marieke van Erp ◽  
Christian Reynolds ◽  
Diana Maynard ◽  
Alain Starke ◽  
Rebeca Ibáñez Martín ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Domain Knowledge ◽  
State Of The Art ◽  
Interdisciplinary Approach ◽  
Comprehensive Analysis ◽  
The State ◽  
Use Cases

In this paper, we discuss the use of natural language processing and artificial intelligence to analyze nutritional and sustainability aspects of recipes and food. We present the state-of-the-art and some use cases, followed by a discussion of challenges. Our perspective on addressing these is that while they typically have a technical nature, they nevertheless require an interdisciplinary approach combining natural language processing and artificial intelligence with expert domain knowledge to create practical tools and comprehensive analysis for the food domain.

Natural Language Processing

1996 ◽  
Vol 16 ◽  
pp. 70-85 ◽  
Author(s):  
Thomas C. Rindflesch
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Computational Linguistics ◽  
Language Processing ◽  
Domain Knowledge ◽  
The State ◽  
Point Of View ◽  
Computer Applications ◽  
Significant Progress ◽  
Future Directions

Work in computational linguistics began very soon after the development of the first computers (Booth, Brandwood and Cleave 1958), yet in the intervening four decades there has been a pervasive feeling that progress in computer understanding of natural language has not been commensurate with progress in other computer applications. Recently, a number of prominent researchers in natural language processing met to assess the state of the discipline and discuss future directions (Bates and Weischedel 1993). The consensus of this meeting was that increased attention to large amounts of lexical and domain knowledge was essential for significant progress, and current research efforts in the field reflect this point of view.

The Applications of Description Logics in Natural Language Processing

2011 ◽  
Vol 181-182 ◽  
pp. 236-241
Author(s):  
Xian Yi Cheng ◽  
Chen Cheng ◽  
Qian Zhu
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Domain Knowledge ◽  
Description Logic ◽  
Description Logics ◽  
Semantic Interpretation ◽  
Natural Language Semantic ◽  
Ontology Language ◽  
Basic Ideas

As a sort of formalizing tool of knowledge representation, Description Logics have been successfully applied in Information System, Software Engineering and Natural Language processing and so on. Description Logics also play a key role in text representation, Natural Language semantic interpretation and language ontology description. Description Logics have been logical basis of OWL which is an ontology language that is recommended by W3C. This paper discusses the description logic basic ideas under vocabulary semantic, context meaning, domain knowledge and background knowledge.

scholarly journals Linking Free Text Documentation of Functioning and Disability to the ICF with Natural Language Processing

2021 ◽  
Author(s):  
Denis R Newman-Griffis ◽  
Jonathan Camacho Maldonado ◽  
Pei-Shu Ho ◽  
Maryanne Sacco ◽  
Rafael Jimenez Silva ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Functional Status ◽  
Language Processing ◽  
Free Text ◽  
Clinical Documentation ◽  
Disability Benefits ◽  
Domestic Life ◽  
Patient Functioning ◽  
Status Information

Background: Invaluable information on patient functioning and the complex interactions that define it is recorded in free text portions of the Electronic Health Record (EHR). Leveraging this information to improve clinical decision-making and conduct research requires natural language processing (NLP) technologies to identify and organize the information recorded in clinical documentation. Methods: We used NLP methods to analyze information about patient functioning recorded in two collections of clinical documents pertaining to claims for federal disability benefits from the U.S. Social Security Administration (SSA). We grounded our analysis in the International Classification of Functioning, Disability and Health (ICF), and used the ICF's Activities and Participation domain to classify information about functioning in three key areas: Mobility, Self-Care, and Domestic Life. After annotating functional status information in our datasets through expert clinical review, we trained machine learning-based NLP models to automatically assign ICF codes to mentions of functional activity. Results: We found that rich and diverse information on patient functioning was documented in the free text records. Annotation of 289 documents for Mobility information yielded 2,455 mentions of Mobility activities and 3,176 specific actions corresponding to 13 ICF-based codes. Annotation of 329 documents for Self-Care and Domestic Life information yielded 3,990 activity mentions and 4,665 specific actions corresponding to 16 ICF-based codes. NLP systems for automated ICF coding achieved over 80% macro-averaged F-measure on both datasets, indicating strong performance across all ICF codes used. Conclusions: NLP can help to navigate the tradeoff between flexible and expressive clinical documentation of functioning and standardizable data for comparability and learning. The ICF has practical limitations for classifying functional status information in clinical documentation, but presents a valuable framework for organizing the information recorded in health records about patient functioning. This study advances the development of robust, ICF-based NLP technologies to analyze information on patient functioning, and has significant implications for NLP-powered analysis of functional status information in disability benefits management, clinical care, and research.

scholarly journals The 2019 n2c2/OHNLP Track on Clinical Semantic Textual Similarity: Overview (Preprint)

2020 ◽  
Author(s):  
Yanshan Wang ◽  
Sunyang Fu ◽  
Feichen Shen ◽  
Sam Henry ◽  
Ozlem Uzuner ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Shared Task ◽  
Data Set ◽  
Clinical Text ◽  
Clinical Notes ◽  
Clinical Domain ◽  
Semantic Textual Similarity

BACKGROUND Semantic textual similarity is a common task in the general English domain to assess the degree to which the underlying semantics of 2 text segments are equivalent to each other. Clinical Semantic Textual Similarity (ClinicalSTS) is the semantic textual similarity task in the clinical domain that attempts to measure the degree of semantic equivalence between 2 snippets of clinical text. Due to the frequent use of templates in the Electronic Health Record system, a large amount of redundant text exists in clinical notes, making ClinicalSTS crucial for the secondary use of clinical text in downstream clinical natural language processing applications, such as clinical text summarization, clinical semantics extraction, and clinical information retrieval. OBJECTIVE Our objective was to release ClinicalSTS data sets and to motivate natural language processing and biomedical informatics communities to tackle semantic text similarity tasks in the clinical domain. METHODS We organized the first BioCreative/OHNLP ClinicalSTS shared task in 2018 by making available a real-world ClinicalSTS data set. We continued the shared task in 2019 in collaboration with National NLP Clinical Challenges (n2c2) and the Open Health Natural Language Processing (OHNLP) consortium and organized the 2019 n2c2/OHNLP ClinicalSTS track. We released a larger ClinicalSTS data set comprising 1642 clinical sentence pairs, including 1068 pairs from the 2018 shared task and 1006 new pairs from 2 electronic health record systems, GE and Epic. We released 80% (1642/2054) of the data to participating teams to develop and fine-tune the semantic textual similarity systems and used the remaining 20% (412/2054) as blind testing to evaluate their systems. The workshop was held in conjunction with the American Medical Informatics Association 2019 Annual Symposium. RESULTS Of the 78 international teams that signed on to the n2c2/OHNLP ClinicalSTS shared task, 33 produced a total of 87 valid system submissions. The top 3 systems were generated by IBM Research, the National Center for Biotechnology Information, and the University of Florida, with Pearson correlations of <i>r</i>=.9010, <i>r</i>=.8967, and <i>r</i>=.8864, respectively. Most top-performing systems used state-of-the-art neural language models, such as BERT and XLNet, and state-of-the-art training schemas in deep learning, such as pretraining and fine-tuning schema, and multitask learning. Overall, the participating systems performed better on the Epic sentence pairs than on the GE sentence pairs, despite a much larger portion of the training data being GE sentence pairs. CONCLUSIONS The 2019 n2c2/OHNLP ClinicalSTS shared task focused on computing semantic similarity for clinical text sentences generated from clinical notes in the real world. It attracted a large number of international teams. The ClinicalSTS shared task could continue to serve as a venue for researchers in natural language processing and medical informatics communities to develop and improve semantic textual similarity techniques for clinical text.

Sign in / Sign up

Export Citation Format

Share Document