scholarly journals Markup: A Web-Based Clinical Annotation Tool with Enhanced Ontology Mapping

2020 ◽  
Vol 5 (5) ◽  
Author(s):  
Samuel Dobbie ◽  
Huw Strafford ◽  
W Owen Pickrell ◽  
Beata Fonferko-Shadrach ◽  
Ashley Akbari ◽  
...  
Keyword(s):  
Language Processing ◽  
Gold Standard ◽  
Clinical Information ◽  
Healthcare Research ◽  
Learning Technologies ◽  
Free Text ◽  
Specific Information ◽  
Annotation Tool ◽  
Snomed Ct ◽  
Web Based

IntroductionUnstructured free-text clinical notes often contain valuable information relating to patient symptoms, prescriptions and diagnoses. These can assist with better care for patients and novel healthcare research if transformed into accessible, structured clinical text. In particular, Natural Language Processing (NLP) algorithms can produce such structured outputs, but require gold standard data to train and validate their accuracy. While existing tools such as Brat and Webanno provide interfaces to manually annotate text, there is a lack of capability to efficiently annotate complex clinical information. Objectives and ApproachWe present Markup, an open-source, web-based annotation tool developed for use within clinical contexts by domain experts to produce gold standard annotations for NLP development. Markup incorporates NLP and Active Learning technologies to enable rapid and accurate annotation of unstructured documents. Markup supports custom user configurations, automated annotation suggestions, and automated mapping to existing clinical ontologies such as the Unified Medical Language System (UMLS), the Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT), or custom, user-defined ontologies. ResultsMarkup has been tested on Epilepsy clinic letters, where captured annotations were used to build and test NLP applications. Markup allowed for inter-annotator statistics to be calculated in the case of multiple annotators. Re-annotation, following iterations of annotation definitions, was incorporated for flexibility. UMLS codes, certainty context, and multiple components from complex phrases were all able to be captured and exported in a structured format. Conclusions / ImplicationsMarkup allows gold standard annotations to be collected efficiently across unstructured text and is optimized to capture health-specific information. These annotations are important to develop and validate NLP algorithms that automate the capture of important information from clinic letters at scale.

scholarly journals Using natural language processing to extract structured epilepsy data from unstructured clinic letters

2018 ◽  
Vol 3 (4) ◽  
Author(s):  
Beata Fonferko-Shadrach ◽  
Arron Lacey ◽  
Ashley Akbari ◽  
Simon Thompson ◽  
David Ford ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Large Scale ◽  
Clinical Information ◽  
Training Sample ◽  
Healthcare Research ◽  
Free Text ◽  
Specific Information ◽  
Data Types

IntroductionElectronic health records (EHR) are a powerful resource in enabling large-scale healthcare research. EHRs often lack detailed disease-specific information that is collected in free text within clinical settings. This challenge can be addressed by using Natural Language Processing (NLP) to derive and extract detailed clinical information from free text. Objectives and ApproachUsing a training sample of 40 letters, we used the General Architecture for Text Engineering (GATE) framework to build custom rule sets for nine categories of epilepsy information as well as clinic date and date of birth. We used a validation set of 200 clinic letters to compare the results of our algorithm to a separate manual review by a clinician, where we evaluated a “per item” and a “per letter” approach for each category. ResultsThe “per letter” approach identified 1,939 items of information with overall precision, recall and F1-score of 92.7%, 77.7% and 85.6%. Precision and recall for epilepsy specific categories were: diagnosis (85.3%,92.4%),  type (93.7%,83.2%), focal seizure (99.0%,68.3%), generalised seizure (92.5%,57.0%), seizure frequency (92.0%,52.3%), medication (96.1%,94.0%), CT (66.7%,47.1%), MRI (96.6%,51.4%) and EEG (95.8%,40.6%). By combining all items per category, per letter we were able to achieve higher precision, recall and F1-scores of 94.6%, 84.2% and 89.0% across all categories. Conclusion/ImplicationsOur results demonstrate that NLP techniques can be used to accurately extract rich phenotypic details from clinic letters that is often missing from routinely-collected data. Capturing these new data types provides a platform for conducting novel precision neurology research, in addition to potential applicability to other disease areas.

scholarly journals Markup: A Web-Based Annotation Tool Powered by Active Learning

2021 ◽  
Vol 3 ◽  
Author(s):  
Samuel Dobbie ◽  
Huw Strafford ◽  
W. Owen Pickrell ◽  
Beata Fonferko-Shadrach ◽  
Carys Jones ◽  
...  
Keyword(s):  
Active Learning ◽  
Language Processing ◽  
Healthcare Setting ◽  
Free Text ◽  
Annotation Tool ◽  
Web Based ◽  
Automated Mapping ◽  
Unified Medical Language System ◽  
Domain Specific ◽  
Health And Social Care

Across various domains, such as health and social care, law, news, and social media, there are increasing quantities of unstructured texts being produced. These potential data sources often contain rich information that could be used for domain-specific and research purposes. However, the unstructured nature of free-text data poses a significant challenge for its utilisation due to the necessity of substantial manual intervention from domain-experts to label embedded information. Annotation tools can assist with this process by providing functionality that enables the accurate capture and transformation of unstructured texts into structured annotations, which can be used individually, or as part of larger Natural Language Processing (NLP) pipelines. We present Markup (https://www.getmarkup.com/) an open-source, web-based annotation tool that is undergoing continued development for use across all domains. Markup incorporates NLP and Active Learning (AL) technologies to enable rapid and accurate annotation using custom user configurations, predictive annotation suggestions, and automated mapping suggestions to both domain-specific ontologies, such as the Unified Medical Language System (UMLS), and custom, user-defined ontologies. We demonstrate a real-world use case of how Markup has been used in a healthcare setting to annotate structured information from unstructured clinic letters, where captured annotations were used to build and test NLP applications.

scholarly journals Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology

10.2196/20492 ◽  
2021 ◽  
Vol 9 (7) ◽  
pp. e20492
Author(s):  
Lea Canales ◽  
Sebastian Menke ◽  
Stephanie Marchesseau ◽  
Ariel D’Agostino ◽  
Carlos del Rio-Bermudez ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Gold Standard ◽  
Performance Metrics ◽  
Evaluation Methodology ◽  
Free Text ◽  
Use Case ◽  
Five Phases ◽  
Clinical Natural Language Processing

Background Clinical natural language processing (cNLP) systems are of crucial importance due to their increasing capability in extracting clinically important information from free text contained in electronic health records (EHRs). The conversion of a nonstructured representation of a patient’s clinical history into a structured format enables medical doctors to generate clinical knowledge at a level that was not possible before. Finally, the interpretation of the insights gained provided by cNLP systems has a great potential in driving decisions about clinical practice. However, carrying out robust evaluations of those cNLP systems is a complex task that is hindered by a lack of standard guidance on how to systematically approach them. Objective Our objective was to offer natural language processing (NLP) experts a methodology for the evaluation of cNLP systems to assist them in carrying out this task. By following the proposed phases, the robustness and representativeness of the performance metrics of their own cNLP systems can be assured. Methods The proposed evaluation methodology comprised five phases: (1) the definition of the target population, (2) the statistical document collection, (3) the design of the annotation guidelines and annotation project, (4) the external annotations, and (5) the cNLP system performance evaluation. We presented the application of all phases to evaluate the performance of a cNLP system called “EHRead Technology” (developed by Savana, an international medical company), applied in a study on patients with asthma. As part of the evaluation methodology, we introduced the Sample Size Calculator for Evaluations (SLiCE), a software tool that calculates the number of documents needed to achieve a statistically useful and resourceful gold standard. Results The application of the proposed evaluation methodology on a real use-case study of patients with asthma revealed the benefit of the different phases for cNLP system evaluations. By using SLiCE to adjust the number of documents needed, a meaningful and resourceful gold standard was created. In the presented use-case, using as little as 519 EHRs, it was possible to evaluate the performance of the cNLP system and obtain performance metrics for the primary variable within the expected CIs. Conclusions We showed that our evaluation methodology can offer guidance to NLP experts on how to approach the evaluation of their cNLP systems. By following the five phases, NLP experts can assure the robustness of their evaluation and avoid unnecessary investment of human and financial resources. Besides the theoretical guidance, we offer SLiCE as an easy-to-use, open-source Python library.

FindThatQuote: A Question-Answering Web-based System to Locate Quotes using Deep Learning and Natural-Language Processing

2021 ◽  
Author(s):  
Nathan Ji ◽  
Yu Sun
Keyword(s):  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Success Rate ◽  
Language Processing ◽  
Question Answering ◽  
Specific Information ◽  
Web Based ◽  
User Friendly ◽  
Web Based System

The digital age gives us access to a multitude of both information and mediums in which we can interpret information. A majority of the time, many people find interpreting such information difficult as the medium may not be as user friendly as possible. This project has examined the inquiry of how one can identify specific information in a given text based on a question. This inquiry is intended to streamline one's ability to determine the relevance of a given text relative to his objective. The project has an overall 80% success rate given 10 articles with three questions asked per article. This success rate indicates that this project is likely applicable to those who are asking for content level questions within an article.

scholarly journals Using natural language processing to extract structured epilepsy data from unstructured clinic letters: development and validation of the ExECT (extraction of epilepsy clinical text) system

BMJ Open ◽  
2019 ◽  
Vol 9 (4) ◽  
pp. e023232 ◽  
Author(s):  
Beata Fonferko-Shadrach ◽  
Arron S Lacey ◽  
Angus Roberts ◽  
Ashley Akbari ◽  
Simon Thompson ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Seizure Frequency ◽  
Extraction System ◽  
Health Board ◽  
Free Text ◽  
Specific Information ◽  
Clinical Text ◽  
Routinely Collected Data

ObjectiveRoutinely collected healthcare data are a powerful research resource but often lack detailed disease-specific information that is collected in clinical free text, for example, clinic letters. We aim to use natural language processing techniques to extract detailed clinical information from epilepsy clinic letters to enrich routinely collected data.DesignWe used the general architecture for text engineering (GATE) framework to build an information extraction system, ExECT (extraction of epilepsy clinical text), combining rule-based and statistical techniques. We extracted nine categories of epilepsy information in addition to clinic date and date of birth across 200 clinic letters. We compared the results of our algorithm with a manual review of the letters by an epilepsy clinician.SettingDe-identified and pseudonymised epilepsy clinic letters from a Health Board serving half a million residents in Wales, UK.ResultsWe identified 1925 items of information with overall precision, recall and F1 score of 91.4%, 81.4% and 86.1%, respectively. Precision and recall for epilepsy-specific categories were: epilepsy diagnosis (88.1%, 89.0%), epilepsy type (89.8%, 79.8%), focal seizures (96.2%, 69.7%), generalised seizures (88.8%, 52.3%), seizure frequency (86.3%–53.6%), medication (96.1%, 94.0%), CT (55.6%, 58.8%), MRI (82.4%, 68.8%) and electroencephalogram (81.5%, 75.3%).ConclusionsWe have built an automated clinical text extraction system that can accurately extract epilepsy information from free text in clinic letters. This can enhance routinely collected data for research in the UK. The information extracted with ExECT such as epilepsy type, seizure frequency and neurological investigations are often missing from routinely collected data. We propose that our algorithm can bridge this data gap enabling further epilepsy research opportunities. While many of the rules in our pipeline were tailored to extract epilepsy specific information, our methods can be applied to other diseases and also can be used in clinical practice to record patient information in a structured manner.

scholarly journals Automated SNOMED CT concept and attribute relationship detection through a web-based implementation of cTAKES

2019 ◽  
Vol 10 (1) ◽  
Author(s):  
Martijn G. Kersloot ◽  
Francis Lau ◽  
Ameen Abu-Hanna ◽  
Derk L. Arts ◽  
Ronald Cornet
Keyword(s):  
Lung Cancer ◽  
Small Cell Lung Cancer ◽  
Language Processing ◽  
Cell Lung Cancer ◽  
Permutation Test ◽  
Small Cell ◽  
Entity Recognition ◽  
Free Text ◽  
Snomed Ct ◽  
Small Cell Lung

Abstract Background Information in Electronic Health Records is largely stored as unstructured free text. Natural language processing (NLP), or Medical Language Processing (MLP) in medicine, aims at extracting structured information from free text, and is less expensive and time-consuming than manual extraction. However, most algorithms in MLP are institution-specific or address only one clinical need, and thus cannot be broadly applied. In addition, most MLP systems do not detect concepts in misspelled text and cannot detect attribute relationships between concepts. The objective of this study was to develop and evaluate an MLP application that includes generic algorithms for the detection of (misspelled) concepts and of attribute relationships between them. Methods An implementation of the MLP system cTAKES, called DIRECT, was developed with generic SNOMED CT concept filter, concept relationship detection, and attribute relationship detection algorithms and a custom dictionary. Four implementations of cTAKES were evaluated by comparing 98 manually annotated oncology charts with the output of DIRECT. The F1-score was determined for named-entity recognition and attribute relationship detection for the concepts ‘lung cancer’, ‘non-small cell lung cancer’, and ‘recurrence’. The performance of the four implementations was compared with a two-tailed permutation test. Results DIRECT detected lung cancer and non-small cell lung cancer concepts with F1-scores between 0.828 and 0.947 and between 0.862 and 0.933, respectively. The concept recurrence was detected with a significantly higher F1-score of 0.921, compared to the other implementations, and the relationship between recurrence and lung cancer with an F1-score of 0.857. The precision of the detection of lung cancer, non-small cell lung cancer, and recurrence concepts were 1.000, 0.966, and 0.879, compared to precisions of 0.943, 0.967, and 0.000 in the original implementation, respectively. Conclusion DIRECT can detect oncology concepts and attribute relationships with high precision and can detect recurrence with significant increase in F1-score, compared to the original implementation of cTAKES, due to the usage of a custom dictionary and a generic concept relationship detection algorithm. These concepts and relationships can be used to encode clinical narratives, and can thus substantially reduce manual chart abstraction efforts, saving time for clinicians and researchers.

scholarly journals Food entries in a large allergy data repository

2015 ◽  
Vol 23 (e1) ◽  
pp. e79-e87 ◽  
Author(s):  
Joseph M. Plasek ◽  
Foster R. Goss ◽  
Kenneth H. Lai ◽  
Jason J. Lau ◽  
Diane L. Seger, ◽  
...  
Keyword(s):  
Language Processing ◽  
Food Allergen ◽  
External Validation ◽  
Data Repository ◽  
Free Text ◽  
Snomed Ct ◽  
Health Records ◽  
Mapping System ◽  
Systematized Nomenclature Of Medicine ◽  
New Strategies

Abstract Objective Accurate food adverse sensitivity documentation in electronic health records (EHRs) is crucial to patient safety. This study examined, encoded, and grouped foods that caused any adverse sensitivity in a large allergy repository using natural language processing and standard terminologies. Methods Using the Medical Text Extraction, Reasoning, and Mapping System (MTERMS), we processed both structured and free-text entries stored in an enterprise-wide allergy repository (Partners’ Enterprise-wide Allergy Repository), normalized diverse food allergen terms into concepts, and encoded these concepts using the Systematized Nomenclature of Medicine – Clinical Terms (SNOMED-CT) and Unique Ingredient Identifiers (UNII) terminologies. Concept coverage also was assessed for these two terminologies. We further categorized allergen concepts into groups and calculated the frequencies of these concepts by group. Finally, we conducted an external validation of MTERMS’s performance when identifying food allergen terms, using a randomized sample from a different institution. Results We identified 158 552 food allergen records (2140 unique terms) in the Partners repository, corresponding to 672 food allergen concepts. High-frequency groups included shellfish (19.3%), fruits or vegetables (18.4%), dairy (9.0%), peanuts (8.5%), tree nuts (8.5%), eggs (6.0%), grains (5.1%), and additives (4.7%). Ambiguous, generic concepts such as “nuts” and “seafood” accounted for 8.8% of the records. SNOMED-CT covered more concepts than UNII in terms of exact (81.7% vs 68.0%) and partial (14.3% vs 9.7%) matches. Discussion Adverse sensitivities to food are diverse, and existing standard terminologies have gaps in their coverage of the breadth of allergy concepts. Conclusion New strategies are needed to represent and standardize food adverse sensitivity concepts, to improve documentation in EHRs.

scholarly journals Natural Language Processing and Technical Challenges of Influenza-Like Illness Surveillance

2016 ◽  
Vol 8 (1) ◽  
Author(s):  
Dino P. Rumoro ◽  
Gillian S. Gibbs ◽  
Shital C. Shah ◽  
Marilyn M. Hallock ◽  
Gordon M. Trenholme ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Clinical Information ◽  
Free Text ◽  
Surveillance Systems ◽  
Clinical Text ◽  
Influenza Like Illness ◽  
Technical Issues ◽  
Original Information

Processing free-text clinical information in an electronic medical record may enhance surveillance systems for early identification of influenza-like illness outbreaks. However, processing clinical text using natural language processing (NLP) poses a challenge in preserving the semantics of the original information recorded. In this study, we discuss several NLP and technical issues as well as potential solutions for implementation in syndromic surveillance systems.

scholarly journals Obtaining structured clinical data from unstructured data using natural language processing software

2017 ◽  
Vol 1 (1) ◽  
Author(s):  
Arron S Lacey ◽  
Beata Fonferko-Shadrach ◽  
Ronan A Lyons ◽  
Mike P Kerr ◽  
David V Ford ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Age Of Onset ◽  
Focal Epilepsy ◽  
Clinical Information ◽  
Unstructured Data ◽  
Free Text ◽  
Epilepsy Diagnosis ◽  
Generalized Epilepsy

ABSTRACT BackgroundFree text documents in healthcare settings contain a wealth of information not captured in electronic healthcare records (EHRs). Epilepsy clinic letters are an example of an unstructured data source containing a large amount of intricate disease information. Extracting meaningful and contextually correct clinical information from free text sources, to enhance EHRs, remains a significant challenge. SCANR (Swansea University Collaborative in the Analysis of NLP Research) was set up to use natural language processing (NLP) technology to extract structured data from unstructured sources. IBM Watson Content Analytics software (ICA) uses NLP technology. It enables users to define annotations based on dictionaries and language characteristics to create parsing rules that highlight relevant items. These include clinical details such as symptoms and diagnoses, medication and test results, as well as personal identifiers.   ApproachTo use ICA to build a pipeline to accurately extract detailed epilepsy information from clinic letters. MethodsWe used ICA to retrieve important epilepsy information from 41 pseudo-anonymized unstructured epilepsy clinic letters. The 41 letters consisted of 13 ‘new’ and 28 ‘follow-up’ letters (for 15 different patients) written by 12 different doctors in different styles. We designed dictionaries and annotators to enable ICA to extract epilepsy type (focal, generalized or unclassified), epilepsy cause, age of onset, investigation results (EEG, CT and MRI), medication, and clinic date. Epilepsy clinicians assessed the accuracy of the pipeline. ResultsThe accuracy (sensitivity, specificity) of each concept was: epilepsy diagnosis 98% (97%, 100%), focal epilepsy 100%, generalized epilepsy 98% (93%, 100%), medication 95% (93%, 100%), age of onset 100% and clinic date 95% (95%, 100%). Precision and recall for each concept were respectively, 98% and 97% for epilepsy diagnosis, 100% each for focal epilepsy, 100% and 93% for generalized epilepsy, 100% each for age of onset, 100% and 93% for medication, 100% and 96% for EEG results, 100% and 83% for MRI scan results, and 100% and 95% for clinic date. Conclusions ICA is capable of extracting detailed, structured epilepsy information from unstructured clinic letters to a high degree of accuracy. This data can be used to populate relational databases and be linked to EHRs. Researchers can build in custom rules to identify concepts of interest from letters and produce structured information. We plan to extend our work to hundreds and then thousands of clinic letters, to provide phenotypically rich epilepsy data to link with other anonymised, routinely collected data.

Sign in / Sign up

Export Citation Format

Share Document