clinical natural language processing
Recently Published Documents


TOTAL DOCUMENTS

39
(FIVE YEARS 19)

H-INDEX

10
(FIVE YEARS 2)

2021 ◽  
Vol 30 (01) ◽  
pp. 257-263
Author(s):  
Natalia Grabar ◽  
Cyril Grouin ◽  

Summary Objectives: To analyze the content of publications within the medical NLP domain in 2020. Methods: Automatic and manual preselection of publications to be reviewed, and selection of the best NLP papers of the year. Analysis of the important issues. Results: Three best papers have been selected in 2020. We also propose an analysis of the content of the NLP publications in 2020, all topics included. Conclusion: The two main issues addressed in 2020 are related to the investigation of COVID-related questions and to the further adaptation and use of transformer models. Besides, the trends from the past years continue, such as diversification of languages processed and use of information from social networks


10.2196/20492 ◽  
2021 ◽  
Vol 9 (7) ◽  
pp. e20492
Author(s):  
Lea Canales ◽  
Sebastian Menke ◽  
Stephanie Marchesseau ◽  
Ariel D’Agostino ◽  
Carlos del Rio-Bermudez ◽  
...  

Background Clinical natural language processing (cNLP) systems are of crucial importance due to their increasing capability in extracting clinically important information from free text contained in electronic health records (EHRs). The conversion of a nonstructured representation of a patient’s clinical history into a structured format enables medical doctors to generate clinical knowledge at a level that was not possible before. Finally, the interpretation of the insights gained provided by cNLP systems has a great potential in driving decisions about clinical practice. However, carrying out robust evaluations of those cNLP systems is a complex task that is hindered by a lack of standard guidance on how to systematically approach them. Objective Our objective was to offer natural language processing (NLP) experts a methodology for the evaluation of cNLP systems to assist them in carrying out this task. By following the proposed phases, the robustness and representativeness of the performance metrics of their own cNLP systems can be assured. Methods The proposed evaluation methodology comprised five phases: (1) the definition of the target population, (2) the statistical document collection, (3) the design of the annotation guidelines and annotation project, (4) the external annotations, and (5) the cNLP system performance evaluation. We presented the application of all phases to evaluate the performance of a cNLP system called “EHRead Technology” (developed by Savana, an international medical company), applied in a study on patients with asthma. As part of the evaluation methodology, we introduced the Sample Size Calculator for Evaluations (SLiCE), a software tool that calculates the number of documents needed to achieve a statistically useful and resourceful gold standard. Results The application of the proposed evaluation methodology on a real use-case study of patients with asthma revealed the benefit of the different phases for cNLP system evaluations. By using SLiCE to adjust the number of documents needed, a meaningful and resourceful gold standard was created. In the presented use-case, using as little as 519 EHRs, it was possible to evaluate the performance of the cNLP system and obtain performance metrics for the primary variable within the expected CIs. Conclusions We showed that our evaluation methodology can offer guidance to NLP experts on how to approach the evaluation of their cNLP systems. By following the five phases, NLP experts can assure the robustness of their evaluation and avoid unnecessary investment of human and financial resources. Besides the theoretical guidance, we offer SLiCE as an easy-to-use, open-source Python library.


Author(s):  
Naga Lalitha Valli ALLA ◽  
Aipeng CHEN ◽  
Sean BATONGBACAL ◽  
Chandini NEKKANTTI ◽  
Hong-Jie Dai ◽  
...  

Author(s):  
William Digan ◽  
Aurélie Névéol ◽  
Antoine Neuraz ◽  
Maxime Wack ◽  
David Baudoin ◽  
...  

Abstract Background The increasing complexity of data streams and computational processes in modern clinical health information systems makes reproducibility challenging. Clinical natural language processing (NLP) pipelines are routinely leveraged for the secondary use of data. Workflow management systems (WMS) have been widely used in bioinformatics to handle the reproducibility bottleneck. Objective To evaluate if WMS and other bioinformatics practices could impact the reproducibility of clinical NLP frameworks. Materials and Methods Based on the literature across multiple research fields (NLP, bioinformatics and clinical informatics) we selected articles which (1) review reproducibility practices and (2) highlight a set of rules or guidelines to ensure tool or pipeline reproducibility. We aggregate insight from the literature to define reproducibility recommendations. Finally, we assess the compliance of 7 NLP frameworks to the recommendations. Results We identified 40 reproducibility features from 8 selected articles. Frameworks based on WMS match more than 50% of features (26 features for LAPPS Grid, 22 features for OpenMinted) compared to 18 features for current clinical NLP framework (cTakes, CLAMP) and 17 features for GATE, ScispaCy, and Textflows. Discussion 34 recommendations are endorsed by at least 2 articles from our selection. Overall, 15 features were adopted by every NLP Framework. Nevertheless, frameworks based on WMS had a better compliance with the features. Conclusion NLP frameworks could benefit from lessons learned from the bioinformatics field (eg, public repositories of curated tools and workflows or use of containers for shareability) to enhance the reproducibility in a clinical setting.


2020 ◽  
Author(s):  
Lea Canales ◽  
Ariel D’Agostino ◽  
Sebastian Menke

BACKGROUND Clinical Natural Language Processing (NLP) systems are of crucial importance, because of their increasing relevance in driving decisions about clinical practice. However, carrying out a sound evaluation of NLP systems is complex and hindered by a lack of guidance on how to approach it. OBJECTIVE This research aims to provide a state-of-the-art methodology for the evaluation of a clinical NLP system, thereby guiding NLP researchers in this process with the final goal to ensure the robustness and representativeness of the performance metrics. METHODS We developed a methodology that guides through the process of developing an evaluation of a clinical NLP system using Savana’s ‘EHRead technology’ applied on a real use-case on chronic obstructive pulmonary disease (COPD). In addition, we further introduce SLiCE, a software tool that assists NLP specialists to create a statistically useful gold standard. RESULTS The gold standard contained 49.6% positive and 50.4% negative examples for COPD. For the COPD study, the confidence interval (CI) of the primary variable COPD, calculated using SLiCE, demonstrated its usefulness with CI widths of 0.074 for Precision, 0.046 for Recall, and 0.061 for F1, respectively. CONCLUSIONS Our proposed methodology aims to assist the process of creating an evaluation of a clinical NLP system. Researchers can follow our suggestions step-by-step and use SLiCE to statistically back up their gold standard. We successfully evaluated Savana’s ‘EHRead technology’ using our proposed methodology on a real use-case. We share here the outcome of our experiences working in developing NLP solutions for the clinical domain, hoping that it might help others to establish sound protocols for the evaluation of their NLP system.


JAMIA Open ◽  
2020 ◽  
Vol 3 (2) ◽  
pp. 146-150
Author(s):  
Egoitz Laparra ◽  
Steven Bethard ◽  
Timothy A Miller

Abstract Building clinical natural language processing (NLP) systems that work on widely varying data is an absolute necessity because of the expense of obtaining new training data. While domain adaptation research can have a positive impact on this problem, the most widely studied paradigms do not take into account the realities of clinical data sharing. To address this issue, we lay out a taxonomy of domain adaptation, parameterizing by what data is shareable. We show that the most realistic settings for clinical use cases are seriously under-studied. To support research in these important directions, we make a series of recommendations, not just for domain adaptation but for clinical NLP in general, that ensure that data, shared tasks, and released models are broadly useful, and that initiate research directions where the clinical NLP community can lead the broader NLP and machine learning fields.


Sign in / Sign up

Export Citation Format

Share Document