The Worth of Words: Decision Support via Natural Language Processing of Trainee Data (Preprint)

2021 ◽  
Author(s):  
Yusuf Yilmaz ◽  
Alma Jurado Nunez ◽  
Ali Ariaeinejad ◽  
Mark Lee ◽  
Jonathan Sherbino ◽  
...  
Keyword(s):  
Machine Learning ◽  
Natural Language Processing ◽  
Decision Support ◽  
Natural Language ◽  
Language Processing ◽  
Qualitative Data ◽  
Supervised Machine Learning ◽  
Predicting Performance ◽  
Sensitivity Specificity ◽  
Narrative Comments

BACKGROUND Residents receive a numeric performance rating (e.g., 1-7 scoring scale) along with a narrative (i.e., qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance. OBJECTIVE This study evaluates the accuracy of a decision support system for raters using natural language processing (NLP) and machine learning (ML). METHODS NLP was performed retrospectively on a complete dataset of narrative comments (i.e., text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using bag-of-n-grams technique with three input types: unigram, bigrams, and trigrams. Supervised machine learning models using linear regression were trained for two outputs using the original ratings and dichotomized ratings (at risk or not). Sensitivity, specificity, and accuracy metrics are reported. RESULTS The database consisted of 7,199 unique direct observation assessments, containing both narrative comments and a 3 to 7 rating in imbalanced distribution (3-5: 726, and 6-7: 4,871 ratings). Total of 141 unique raters from five different hospitals and 45 unique residents participated over the course of five academic years. When comparing the three different input types for diagnosing if a trainee would be rated low (i.e., 1-5) or high (i.e., 6 or 7), our accuracy for trigrams was (87%), bigrams (86%), and unigrams (82%). We also found that all three input types had better prediction accuracy when using a bimodal cut (e.g., lower or higher) compared to predicting performance along the full 7-scale (50-52%). CONCLUSIONS The ML models can accurately identify underperforming residents via narrative comments provided for work-based assessments. The words generated in WBAs can be a worthy dataset to augment human decisions for educators tasked with processing large volumes of narrative assessments. CLINICALTRIAL N/A

scholarly journals Obtaining Knowledge in Pathology Reports Through a Natural Language Processing Approach With Classification, Named-Entity Recognition, and Relation-Extraction Heuristics

2019 ◽  
pp. 1-8 ◽  
Author(s):  
Tomasz Oliwa ◽  
Steven B. Maron ◽  
Leah M. Chase ◽  
Samantha Lomnicki ◽  
Daniel V.T. Catenacci ◽  
...  
Keyword(s):  
Machine Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Named Entity Recognition ◽  
Entity Recognition ◽  
Classification Model ◽  
Supervised Machine Learning ◽  
Named Entity ◽  
Pathology Reports

PURPOSE Robust institutional tumor banks depend on continuous sample curation or else subsequent biopsy or resection specimens are overlooked after initial enrollment. Curation automation is hindered by semistructured free-text clinical pathology notes, which complicate data abstraction. Our motivation is to develop a natural language processing method that dynamically identifies existing pathology specimen elements necessary for locating specimens for future use in a manner that can be re-implemented by other institutions. PATIENTS AND METHODS Pathology reports from patients with gastroesophageal cancer enrolled in The University of Chicago GI oncology tumor bank were used to train and validate a novel composite natural language processing-based pipeline with a supervised machine learning classification step to separate notes into internal (primary review) and external (consultation) reports; a named-entity recognition step to obtain label (accession number), location, date, and sublabels (block identifiers); and a results proofreading step. RESULTS We analyzed 188 pathology reports, including 82 internal reports and 106 external consult reports, and successfully extracted named entities grouped as sample information (label, date, location). Our approach identified up to 24 additional unique samples in external consult notes that could have been overlooked. Our classification model obtained 100% accuracy on the basis of 10-fold cross-validation. Precision, recall, and F1 for class-specific named-entity recognition models show strong performance. CONCLUSION Through a combination of natural language processing and machine learning, we devised a re-implementable and automated approach that can accurately extract specimen attributes from semistructured pathology notes to dynamically populate a tumor registry.

scholarly journals Information retrieval with text mining for Decision Support System

2012 ◽  
Vol 24 (2) ◽  
pp. 117-126 ◽  
Author(s):  
Mahmuda Rahman
Keyword(s):  
Machine Learning ◽  
Information Retrieval ◽  
Natural Language Processing ◽  
Decision Support ◽  
Text Mining ◽  
Decision Support System ◽  
Natural Language ◽  
Key Words ◽  
Language Processing ◽  
Support System

Key words: Natural language processing; C4.5 classification; DSS, machine learning; KNN clustering; SVMDOI: http://dx.doi.org/10.3329/bjsr.v24i2.10768 Bangladesh J. Sci. Res. 24(2):117-126, 2011 (December) 

scholarly journals Efficient identification of nationally mandated reportable cancer cases using natural language processing and machine learning

2016 ◽  
Vol 23 (6) ◽  
pp. 1077-1084 ◽  
Author(s):  
John D Osborne ◽  
Matthew Wyatt ◽  
Andrew O Westfall ◽  
James Willig ◽  
Steven Bethard ◽  
...  
Keyword(s):  
Machine Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Supervised Machine Learning ◽  
Inspection System ◽  
Control Panel ◽  
Pathology Report ◽  
Source Information ◽  
Clinical Notes

Abstract Objective To help cancer registrars efficiently and accurately identify reportable cancer cases. Material and Methods The Cancer Registry Control Panel (CRCP) was developed to detect mentions of reportable cancer cases using a pipeline built on the Unstructured Information Management Architecture – Asynchronous Scaleout (UIMA-AS) architecture containing the National Library of Medicine’s UIMA MetaMap annotator as well as a variety of rule-based UIMA annotators that primarily act to filter out concepts referring to nonreportable cancers. CRCP inspects pathology reports nightly to identify pathology records containing relevant cancer concepts and combines this with diagnosis codes from the Clinical Electronic Data Warehouse to identify candidate cancer patients using supervised machine learning. Cancer mentions are highlighted in all candidate clinical notes and then sorted in CRCP’s web interface for faster validation by cancer registrars. Results CRCP achieved an accuracy of 0.872 and detected reportable cancer cases with a precision of 0.843 and a recall of 0.848. CRCP increases throughput by 22.6% over a baseline (manual review) pathology report inspection system while achieving a higher precision and recall. Depending on registrar time constraints, CRCP can increase recall to 0.939 at the expense of precision by incorporating a data source information feature. Conclusion CRCP demonstrates accurate results when applying natural language processing features to the problem of detecting patients with cases of reportable cancer from clinical notes. We show that implementing only a portion of cancer reporting rules in the form of regular expressions is sufficient to increase the precision, recall, and speed of the detection of reportable cancer cases when combined with off-the-shelf information extraction software and machine learning.

scholarly journals Machine learning in medicine: a practical introduction to natural language processing

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Conrad J. Harrison ◽  
Chris J. Sidey-Gibbons
Keyword(s):  
Machine Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Latent Dirichlet Allocation ◽  
Mental Health Problems ◽  
Characteristic Curve ◽  
Machine Learning Algorithms ◽  
Supervised Machine Learning ◽  
Support Vector

Abstract Background Unstructured text, including medical records, patient feedback, and social media comments, can be a rich source of data for clinical research. Natural language processing (NLP) describes a set of techniques used to convert passages of written text into interpretable datasets that can be analysed by statistical and machine learning (ML) models. The purpose of this paper is to provide a practical introduction to contemporary techniques for the analysis of text-data, using freely-available software. Methods We performed three NLP experiments using publicly-available data obtained from medicine review websites. First, we conducted lexicon-based sentiment analysis on open-text patient reviews of four drugs: Levothyroxine, Viagra, Oseltamivir and Apixaban. Next, we used unsupervised ML (latent Dirichlet allocation, LDA) to identify similar drugs in the dataset, based solely on their reviews. Finally, we developed three supervised ML algorithms to predict whether a drug review was associated with a positive or negative rating. These algorithms were: a regularised logistic regression, a support vector machine (SVM), and an artificial neural network (ANN). We compared the performance of these algorithms in terms of classification accuracy, area under the receiver operating characteristic curve (AUC), sensitivity and specificity. Results Levothyroxine and Viagra were reviewed with a higher proportion of positive sentiments than Oseltamivir and Apixaban. One of the three LDA clusters clearly represented drugs used to treat mental health problems. A common theme suggested by this cluster was drugs taking weeks or months to work. Another cluster clearly represented drugs used as contraceptives. Supervised machine learning algorithms predicted positive or negative drug ratings with classification accuracies ranging from 0.664, 95% CI [0.608, 0.716] for the regularised regression to 0.720, 95% CI [0.664,0.776] for the SVM. Conclusions In this paper, we present a conceptual overview of common techniques used to analyse large volumes of text, and provide reproducible code that can be readily applied to other research studies using open-source software.

scholarly journals Natural Language Processing for the Identification of Silent Brain Infarcts From Neuroimaging Reports (Preprint)

2018 ◽  
Author(s):  
Sunyang Fu ◽  
Lester Y Leung ◽  
Yanshan Wang ◽  
Anne-Olivia Raulli ◽  
David F Kallmes ◽  
...  
Keyword(s):  
Machine Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Mayo Clinic ◽  
Predictive Value ◽  
Support Vector ◽  
Rule Based ◽  
Rule Based System ◽  
Sensitivity Specificity

BACKGROUND Silent brain infarction (SBI) is defined as the presence of 1 or more brain lesions, presumed to be because of vascular occlusion, found by neuroimaging (magnetic resonance imaging or computed tomography) in patients without clinical manifestations of stroke. It is more common than stroke and can be detected in 20% of healthy elderly people. Early detection of SBI may mitigate the risk of stroke by offering preventative treatment plans. Natural language processing (NLP) techniques offer an opportunity to systematically identify SBI cases from electronic health records (EHRs) by extracting, normalizing, and classifying SBI-related incidental findings interpreted by radiologists from neuroimaging reports. OBJECTIVE This study aimed to develop NLP systems to determine individuals with incidentally discovered SBIs from neuroimaging reports at 2 sites: Mayo Clinic and Tufts Medical Center. METHODS Both rule-based and machine learning approaches were adopted in developing the NLP system. The rule-based system was implemented using the open source NLP pipeline MedTagger, developed by Mayo Clinic. Features for rule-based systems, including significant words and patterns related to SBI, were generated using pointwise mutual information. The machine learning models adopted convolutional neural network (CNN), random forest, support vector machine, and logistic regression. The performance of the NLP algorithm was compared with a manually created gold standard. RESULTS A total of 5 reports were removed due to invalid scan types. The interannotator agreements across Mayo and Tufts neuroimaging reports were 0.87 and 0.91, respectively. The rule-based system yielded the best performance of predicting SBI with an accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of 0.991, 0.925, 1.000, 1.000, and 0.990, respectively. The CNN achieved the best score on predicting white matter disease (WMD) with an accuracy, sensitivity, specificity, PPV, and NPV of 0.994, 0.994, 0.994, 0.994, and 0.994, respectively. CONCLUSIONS We adopted a standardized data abstraction and modeling process to developed NLP techniques (rule-based and machine learning) to detect incidental SBIs and WMDs from annotated neuroimaging reports. Validation statistics suggested a high feasibility of detecting SBIs and WMDs from EHRs using NLP.

A Machine Learning Application for Raising WASH Awareness in the Times of COVID-19 Pandemic (Preprint)

2020 ◽  
Author(s):  
Rohan Pandey ◽  
Vaibhav Gautam ◽  
Ridam Pal ◽  
Harsh Bandhey ◽  
Lovedeep Singh Dhingra ◽  
...  
Keyword(s):  
Machine Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Machine Translation ◽  
Language Processing ◽  
User Feedback ◽  
Who Guidelines ◽  
The Times ◽  
The Right ◽  
Local Languages

BACKGROUND The COVID-19 pandemic has uncovered the potential of digital misinformation in shaping the health of nations. The deluge of unverified information that spreads faster than the epidemic itself is an unprecedented phenomenon that has put millions of lives in danger. Mitigating this ‘Infodemic’ requires strong health messaging systems that are engaging, vernacular, scalable, effective and continuously learn the new patterns of misinformation. OBJECTIVE We created WashKaro, a multi-pronged intervention for mitigating misinformation through conversational AI, machine translation and natural language processing. WashKaro provides the right information matched against WHO guidelines through AI, and delivers it in the right format in local languages. METHODS We theorize (i) an NLP based AI engine that could continuously incorporate user feedback to improve relevance of information, (ii) bite sized audio in the local language to improve penetrance in a country with skewed gender literacy ratios, and (iii) conversational but interactive AI engagement with users towards an increased health awareness in the community. RESULTS A total of 5026 people who downloaded the app during the study window, among those 1545 were active users. Our study shows that 3.4 times more females engaged with the App in Hindi as compared to males, the relevance of AI-filtered news content doubled within 45 days of continuous machine learning, and the prudence of integrated AI chatbot “Satya” increased thus proving the usefulness of an mHealth platform to mitigate health misinformation. CONCLUSIONS We conclude that a multi-pronged machine learning application delivering vernacular bite-sized audios and conversational AI is an effective approach to mitigate health misinformation. CLINICALTRIAL Not Applicable

Sign in / Sign up

Export Citation Format

Share Document