scholarly journals Interactive NLP in Clinical Care: Identifying Incidental Findings in Radiology Reports

2019 ◽  
Vol 10 (04) ◽  
pp. 655-669
Author(s):  
Gaurav Trivedi ◽  
Esmaeel R. Dadashzadeh ◽  
Robert M. Handzel ◽  
Wendy W. Chapman ◽  
Shyam Visweswaran ◽  
...  
Keyword(s):  
Language Processing ◽  
Gold Standard ◽  
Incidental Findings ◽  
User Study ◽  
Clinical Care ◽  
Model Performance ◽  
Data Set ◽  
Radiology Reports ◽  
System Usability Scale ◽  
And Control

Abstract Background Despite advances in natural language processing (NLP), extracting information from clinical text is expensive. Interactive tools that are capable of easing the construction, review, and revision of NLP models can reduce this cost and improve the utility of clinical reports for clinical and secondary use. Objectives We present the design and implementation of an interactive NLP tool for identifying incidental findings in radiology reports, along with a user study evaluating the performance and usability of the tool. Methods Expert reviewers provided gold standard annotations for 130 patient encounters (694 reports) at sentence, section, and report levels. We performed a user study with 15 physicians to evaluate the accuracy and usability of our tool. Participants reviewed encounters split into intervention (with predictions) and control conditions (no predictions). We measured changes in model performance, the time spent, and the number of user actions needed. The System Usability Scale (SUS) and an open-ended questionnaire were used to assess usability. Results Starting from bootstrapped models trained on 6 patient encounters, we observed an average increase in F1 score from 0.31 to 0.75 for reports, from 0.32 to 0.68 for sections, and from 0.22 to 0.60 for sentences on a held-out test data set, over an hour-long study session. We found that tool helped significantly reduce the time spent in reviewing encounters (134.30 vs. 148.44 seconds in intervention and control, respectively), while maintaining overall quality of labels as measured against the gold standard. The tool was well received by the study participants with a very good overall SUS score of 78.67. Conclusion The user study demonstrated successful use of the tool by physicians for identifying incidental findings. These results support the viability of adopting interactive NLP tools in clinical care settings for a wider range of clinical applications.

scholarly journals Summarisation with Majority Opinion

2020 ◽  
Author(s):  
Oliver Ray ◽  
Amy Conroy ◽  
Rozano Imansyah
Keyword(s):  
Language Processing ◽  
Gold Standard ◽  
User Study ◽  
Conditional Random Fields ◽  
Data Set ◽  
Majority Opinion ◽  
Score Improvement ◽  
House Of Lords

This paper introduces a method called SUmmarisation with Majority Opinion (SUMO) that integrates and extends two prior approaches for abstractively and extractively summarising UK House of Lords cases. We show how combining two previously distinct lines of work allows us to better address the challenges resulting from this court’s unusual tradition of publishing the opinions of multiple judges with no formal statement of the reasoning (if any) agreed by a majority. We do this by applying natural language processing and machine learning, Conditional Random Fields (CRFs), to a data set we created by fusing together expert-annotated sentence labels from the HOLJ corpus of rhetorical role summary relevance with the ASMO corpus of agreement statement and majority opinion. By using CRFs and a bespoke summary generator on our enriched data set, we show a significant quantitative F1-score improvement in rhetorical role and relevance classification of 10–15% over the state-of-the-art SUM system; and we show a significant qualitative improvement in the quality of our summaries, which closely resemble gold-standard multi-judge abstracts according to a proof-of-principle user study.

scholarly journals Preparing Clinical Text for Use in Biomedical Research

2011 ◽  
pp. 2085-2095
Author(s):  
John P. Pestian ◽  
Lukasz Itert ◽  
Charlotte Andersen
Keyword(s):  
Language Processing ◽  
Medical Center ◽  
Clinical Care ◽  
Academic Medical Center ◽  
Clinical Text ◽  
Delivery Of Care ◽  
Radiology Reports ◽  
Academic Medical ◽  
Textual Data ◽  
Regulatory Actions

Approximately 57 different types of clinical annotations construct a patient’s medical record. These annotations include radiology reports, discharge summaries, and surgical and nursing notes. Hospitals typically produce millions of text-based medical records over the course of a year. These records are essential for the delivery of care, but many are underutilized or not utilized at all for clinical research. The textual data found in these annotations is a rich source of insights into aspects of clinical care and the clinical delivery system. Recent regulatory actions, however, require that, in many cases, data not obtained through informed consent or data not related to the delivery of care must be made anonymous (as referred to by regulators as harmless), before they can be used. This article describes a practical approach with which Cincinnati Children’s Hospital Medical Center (CCHMC), a large pediatric academic medical center with more than 761,000 annual patient encounters, developed open source software for making pediatric clinical text harmless without losing its rich meaning. Development of the software dealt with many of the issues that often arise in natural language processing, such as data collection, disambiguation, and data scrubbing.

scholarly journals NLPReViz: an interactive tool for natural language processing on clinical text

2017 ◽  
Vol 25 (1) ◽  
pp. 81-87 ◽  
Author(s):  
Gaurav Trivedi ◽  
Phuong Pham ◽  
Wendy W Chapman ◽  
Rebecca Hwa ◽  
Janyce Wiebe ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
User Study ◽  
Clinical Text ◽  
Domain Experts ◽  
Expert Review ◽  
System Usability Scale ◽  
Average System ◽  
Colonoscopy Quality

Abstract The gap between domain experts and natural language processing expertise is a barrier to extracting understanding from clinical text. We describe a prototype tool for interactive review and revision of natural language processing models of binary concepts extracted from clinical notes. We evaluated our prototype in a user study involving 9 physicians, who used our tool to build and revise models for 2 colonoscopy quality variables. We report changes in performance relative to the quantity of feedback. Using initial training sets as small as 10 documents, expert review led to final F1scores for the “appendiceal-orifice” variable between 0.78 and 0.91 (with improvements ranging from 13.26% to 29.90%). F1for “biopsy” ranged between 0.88 and 0.94 (−1.52% to 11.74% improvements). The average System Usability Scale score was 70.56. Subjective feedback also suggests possible design improvements.

Automating incidental findings in radiology reports using natural language processing and machine learning to identify and classify pulmonary nodules.

2019 ◽  
Vol 37 (15_suppl) ◽  
pp. e18093-e18093
Author(s):  
Christi French ◽  
Maciek Makowski ◽  
Samantha Terker ◽  
Paul Alexander Clark
Keyword(s):  
Machine Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Pulmonary Nodule ◽  
Incidental Findings ◽  
Lung Nodule ◽  
Radiology Reports

e18093 Background: Pulmonary nodule incidental findings challenge providers to balance resource efficiency and high clinical quality. Incidental findings tend to be undertreated with studies reporting appropriate follow-up rates as low as 29%. Ensuring appropriate follow-up on all incidental findings is labor-intensive; requires the clinical reading and classification of radiology reports to identify high-risk lung nodules. We tested the feasibility of automating this process with natural language processing (NLP) and machine learning (ML). Methods: In cooperation with Sarah Cannon Research Institute (SCRI), we conducted a series of data science experiments utilizing NLP and ML computing techniques on 8,879 free-text, narrative CT (computerized tomography) radiology reports. Reports used were dated from Dec 8, 2015 - April 23, 2017, came from SCRI-affiliated Emergency Department, Inpatient, and Outpatient facilities and were a representative, random sample of the patient populations. Reports were divided into a development set for model training and validation, and a test set to evaluate model performance. Two models were developed - a “Nodule Model” was trained to detect the reported presence of a pulmonary nodule and a rules-based “Sizing Model” was developed to extract the size of the nodule in millimeters. Reports were bucketed into three prediction groups: > = 6 mm, < 6 mm, and no size indicated. Nodules were considered positives and placed in a queue for follow-up if the nodule was predicted > = 6 mm, or if the nodule had no size indicated and the radiology report contained the word “mass.” The Fleischner Society Guidelines and clinical review informed these definitions. Results: Precision and recall metrics were calculated for multiple model thresholds. A threshold was selected based on the validation set calculations and a success criterion of 90% queue precision was selected to minimize false positives. On the test dataset, the F1 measure of the entire pipeline (lung nodule classification model and size extraction model) was 72.9%, recall was 60.3%, and queue precision was 90.2%, exceeding success criteria. Conclusions: The experiments demonstrate the feasibility of NLP and ML technology to automate the detection and classification of pulmonary nodule incidental findings in radiology reports. This approach promises to improve healthcare quality by increasing the rate of appropriate lung nodule incidental finding follow-up and treatment without excessive labor or risking overutilization.

scholarly journals Efficient and Accurate Extracting of Unstructured EHRs on Cancer Therapy Responses for the Development of RECIST Natural Language Processing Tools: Part I, the Corpus

2020 ◽  
pp. 383-391 ◽  
Author(s):  
Yalun Li ◽  
Yung-Hung Luo ◽  
Jason A. Wampfler ◽  
Samuel M. Rubinstein ◽  
Firat Tiryaki ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Gold Standard ◽  
Data Extraction ◽  
Complete Response ◽  
Time Interval ◽  
Data Set ◽  
Clinical Notes ◽  
Standard Data

PURPOSE Electronic health records (EHRs) are created primarily for nonresearch purposes; thus, the amounts of data are enormous, and the data are crude, heterogeneous, incomplete, and largely unstructured, presenting challenges to effective analyses for timely, reliable results. Particularly, research dealing with clinical notes relevant to patient care and outcome is seldom conducted, due to the complexity of data extraction and accurate annotation in the past. RECIST is a set of widely accepted research criteria to evaluate tumor response in patients undergoing antineoplastic therapy. The aim for this study was to identify textual sources for RECIST information in EHRs and to develop a corpus of pharmacotherapy and response entities for development of natural language processing tools. METHODS We focused on pharmacotherapies and patient responses, using 55,120 medical notes (n = 72 types) in Mayo Clinic’s EHRs from 622 randomly selected patients who signed authorization for research. Using the Multidocument Annotation Environment tool, we applied and evaluated predefined keywords, and time interval and note-type filters for identifying RECIST information and established a gold standard data set for patient outcome research. RESULTS Key words reduced clinical notes to 37,406, and using four note types within 12 months postdiagnosis further reduced the number of notes to 5,005 that were manually annotated, which covered 97.9% of all cases (n = 609 of 622). The resulting data set of 609 cases (n = 503 for training and n = 106 for validation purpose), contains 736 fully annotated, deidentified clinical notes, with pharmacotherapies and four response end points: complete response, partial response, stable disease, and progressive disease. This resource is readily expandable to specific drugs, regimens, and most solid tumors. CONCLUSION We have established a gold standard data set to accommodate development of biomedical informatics tools in accelerating research into antineoplastic therapeutic response.

Dual-cell differentiable architecture search for language modeling

2021 ◽  
pp. 1-8
Author(s):  
Quan Wan ◽  
Lin Wu ◽  
Zhengtao Yu
Keyword(s):  
Language Processing ◽  
Network Architecture ◽  
Language Model ◽  
Model Performance ◽  
Input Sequence ◽  
Search Space ◽  
Language Modeling ◽  
Information Storage ◽  
Data Set ◽  
Initial Results

Initial results of neural architecture search (NAS) in natural language processing (NLP) have been achieved, but the search space of most NAS methods is based on the simplest recurrent cell and thus does not consider the modeling of long sequences. The remote information tends to disappear gradually when the input sequence is long, resulting in poor model performance. In this paper, we present an approach based on dual cells to search for a better-performing network architecture. We construct a search space that is more compatible with language modeling tasks by adding an information storage cell inside the search cell, so that we can make better use of the remote information of the sequence and improve the performance of the model. The language model searched by our method achieves better results than those of the baseline method on the Penn Treebank data set and WikiText-2 data set.

Automating Clinical Chart Review: An Open-Source Natural Language Processing Pipeline Developed on Free-Text Radiology Reports From Patients With Glioblastoma

2020 ◽  
pp. 25-34
Author(s):  
Joeky T. Senders ◽  
Logan D. Cho ◽  
Paola Calvachi ◽  
John J. McNulty ◽  
Joanna L. Ashby ◽  
...  
Keyword(s):  
Text Mining ◽  
Open Source ◽  
Language Processing ◽  
Model Performance ◽  
Area Under The Curve ◽  
Free Text ◽  
Rater Agreement ◽  
Excellent Performance ◽  
Radiology Reports ◽  
Radiologic Characteristics

PURPOSE The aim of this study was to develop an open-source natural language processing (NLP) pipeline for text mining of medical information from clinical reports. We also aimed to provide insight into why certain variables or reports are more suitable for clinical text mining than others. MATERIALS AND METHODS Various NLP models were developed to extract 15 radiologic characteristics from free-text radiology reports for patients with glioblastoma. Ten-fold cross-validation was used to optimize the hyperparameter settings and estimate model performance. We examined how model performance was associated with quantitative attributes of the radiologic characteristics and reports. RESULTS In total, 562 unique brain magnetic resonance imaging reports were retrieved. NLP extracted 15 radiologic characteristics with high to excellent discrimination (area under the curve, 0.82 to 0.98) and accuracy (78.6% to 96.6%). Model performance was correlated with the inter-rater agreement of the manually provided labels (ρ = 0.904; P < .001) but not with the frequency distribution of the variables of interest (ρ = 0.179; P = .52). All variables labeled with a near perfect inter-rater agreement were classified with excellent performance (area under the curve > 0.95). Excellent performance could be achieved for variables with only 50 to 100 observations in the minority group and class imbalances up to a 9:1 ratio. Report-level classification accuracy was not associated with the number of words or the vocabulary size in the distinct text documents. CONCLUSION This study provides an open-source NLP pipeline that allows for text mining of narratively written clinical reports. Small sample sizes and class imbalance should not be considered as absolute contraindications for text mining in clinical research. However, future studies should report measures of inter-rater agreement whenever ground truth is based on a consensus label and use this measure to identify clinical variables eligible for text mining.

scholarly journals Surrogate-guided sampling designs for classification of rare outcomes from electronic medical records data

Biostatistics ◽  
2020 ◽  
Author(s):  
W Katherine Tan ◽  
Patrick J Heagerty
Keyword(s):  
Machine Learning ◽  
Clinical Outcomes ◽  
Language Processing ◽  
Large Scale ◽  
Model Performance ◽  
Learning Performance ◽  
Accurate Identification ◽  
Text Data ◽  
Data Set ◽  
The Impact

Summary Scalable and accurate identification of specific clinical outcomes has been enabled by machine-learning applied to electronic medical record systems. The development of classification models requires the collection of a complete labeled data set, where true clinical outcomes are obtained by human expert manual review. For example, the development of natural language processing algorithms requires the abstraction of clinical text data to obtain outcome information necessary for training models. However, if the outcome is rare then simple random sampling results in very few cases and insufficient information to develop accurate classifiers. Since large scale detailed abstraction is often expensive, time-consuming, and not feasible, more efficient strategies are needed. Under such resource constrained settings, we propose a class of enrichment sampling designs, where selection for abstraction is stratified by auxiliary variables related to the true outcome of interest. Stratified sampling on highly specific variables results in targeted samples that are more enriched with cases, which we show translates to increased model discrimination and better statistical learning performance. We provide mathematical details and simulation evidence that links sampling designs to their resulting prediction model performance. We discuss the impact of our proposed sampling on both model training and validation. Finally, we illustrate the proposed designs for outcome label collection and subsequent machine-learning, using radiology report text data from the Lumbar Imaging with Reporting of Epidemiology study.

scholarly journals Natural language processing of radiology reports for the detection of thromboembolic diseases and clinically relevant incidental findings

2014 ◽  
Vol 15 (1) ◽  
Author(s):  
Anne-Dominique Pham ◽  
Aurélie Névéol ◽  
Thomas Lavergne ◽  
Daisuke Yasunaga ◽  
Olivier Clément ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Incidental Findings ◽  
Radiology Reports

Parsing Noun Phrases in the Penn Treebank

2011 ◽  
Vol 37 (4) ◽  
pp. 753-809 ◽  
Author(s):  
David Vadas ◽  
James R. Curran
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Gold Standard ◽  
High Performance ◽  
Large Scale ◽  
Complex Structure ◽  
Noun Phrases ◽  
Simple Task ◽  
Data Set ◽  
Lexical Information

Noun phrases (nps) are a crucial part of natural language, and can have a very complex structure. However, this np structure is largely ignored by the statistical parsing field, as the most widely used corpus is not annotated with it. This lack of gold-standard data has restricted previous efforts to parse nps, making it impossible to perform the supervised experiments that have achieved high performance in so many Natural Language Processing (nlp) tasks. We comprehensively solve this problem by manually annotating np structure for the entire Wall Street Journal section of the Penn Treebank. The inter-annotator agreement scores that we attain dispel the belief that the task is too difficult, and demonstrate that consistent np annotation is possible. Our gold-standard np data is now available for use in all parsers. We experiment with this new data, applying the Collins (2003) parsing model, and find that its recovery of np structure is significantly worse than its overall performance. The parser's F-score is up to 5.69% lower than a baseline that uses deterministic rules. Through much experimentation, we determine that this result is primarily caused by a lack of lexical information. To solve this problem we construct a wide-coverage, large-scale np Bracketing system. With our Penn Treebank data set, which is orders of magnitude larger than those used previously, we build a supervised model that achieves excellent results. Our model performs at 93.8% F-score on the simple task that most previous work has undertaken, and extends to bracket longer, more complex nps that are rarely dealt with in the literature. We attain 89.14% F-score on this much more difficult task. Finally, we implement a post-processing module that brackets nps identified by the Bikel (2004) parser. Our np Bracketing model includes a wide variety of features that provide the lexical information that was missing during the parser experiments, and as a result, we outperform the parser's F-score by 9.04%. These experiments demonstrate the utility of the corpus, and show that many nlp applications can now make use of np structure.

Sign in / Sign up

Export Citation Format

Share Document