scholarly journals A Pre-Training Technique to Localize Medical BERT and to Enhance Biomedical BERT

2020 ◽  
Author(s):  
Shoya Wada ◽  
Toshihiro Takeda ◽  
Shiro Manabe ◽  
Shozo Konishi ◽  
Jun Kamohara ◽  
...  
Keyword(s):  
Language Processing ◽  
High Performance ◽  
Large Scale ◽  
Language Models ◽  
Free Text ◽  
Medical Databases ◽  
Large Size ◽  
Training Technique ◽  
Medical Domain ◽  
Medical Document

Abstract Background: Pre-training large-scale neural language models on raw texts has been shown to make a significant contribution to a strategy for transfer learning in natural language processing (NLP). With the introduction of transformer-based language models, such as Bidirectional Encoder Representations from Transformers (BERT), the performance of information extraction from free text by NLP has significantly improved for both the general domain and the medical domain; however, it is difficult for languages in which there are few publicly available medical databases with a high quality and a large size to train medical BERT models that perform well.Method: We introduce a method to train a BERT model using a small medical corpus both in English and in Japanese. Our proposed method consists of two interventions: simultaneous pre-training, which is intended to encourage masked language modeling and next-sentence prediction on the small medical corpus, and amplified vocabulary, which helps with suiting the small corpus when building the customized corpus by byte-pair encoding. Moreover, we used whole PubMed abstracts and developed a high-performance BERT model, Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University (ouBioBERT), in English via our method. We then evaluated the performance of our BERT models and publicly available baselines and compared them.Results: We confirmed that our Japanese medical BERT outperforms conventional baselines and the other BERT models in terms of the medical-document classification task and that our English BERT pre-trained using both the general and medical domain corpora performs sufficiently for practical use in terms of the biomedical language understanding evaluation (BLUE) benchmark. Moreover, ouBioBERT shows that the total score of the BLUE benchmark is 1.1 points above that of BioBERT and 0.3 points above that of the ablation model trained without our proposed method.Conclusions: Our proposed method makes it feasible to construct a practical medical BERT model in both Japanese and English, and it has a potential to produce higher performing models for biomedical shared tasks.

scholarly journals DICE: A Drug Indication Classification and Encyclopedia for AI-Based Indication Extraction

2021 ◽  
Vol 4 ◽  
Author(s):  
Arjun Bhatt ◽  
Ruth Roberts ◽  
Xi Chen ◽  
Ting Li ◽  
Skylar Connor ◽  
...  
Keyword(s):  
Language Processing ◽  
Clinical Decision Making ◽  
Short Term Memory ◽  
Drug Repositioning ◽  
External Validation ◽  
Clinical Decision ◽  
Language Models ◽  
Free Text ◽  
Drug Labeling ◽  
Real World Evidence

Drug labeling contains an ‘INDICATIONS AND USAGE’ that provides vital information to support clinical decision making and regulatory management. Effective extraction of drug indication information from free-text based resources could facilitate drug repositioning projects and help collect real-world evidence in support of secondary use of approved medicines. To enable AI-powered language models for the extraction of drug indication information, we used manual reading and curation to develop a Drug Indication Classification and Encyclopedia (DICE) based on FDA approved human prescription drug labeling. A DICE scheme with 7,231 sentences categorized into five classes (indications, contradictions, side effects, usage instructions, and clinical observations) was developed. To further elucidate the utility of the DICE, we developed nine different AI-based classifiers for the prediction of indications based on the developed DICE to comprehensively assess their performance. We found that the transformer-based language models yielded an average MCC of 0.887, outperforming the word embedding-based Bidirectional long short-term memory (BiLSTM) models (0.862) with a 2.82% improvement on the test set. The best classifiers were also used to extract drug indication information in DrugBank and achieved a high enrichment rate (>0.930) for this task. We found that domain-specific training could provide more explainable models without performance sacrifices and better generalization for external validation datasets. Altogether, the proposed DICE could be a standard resource for the development and evaluation of task-specific AI-powered, natural language processing (NLP) models.

scholarly journals Comparison of pretraining models and strategies for health-related social media text classification

2021 ◽  
Author(s):  
Yuting Guo ◽  
Yao Ge ◽  
Yuan-Chi Yang ◽  
Mohammed Ali Al-Garadi ◽  
Abeed Sarker
Keyword(s):  
Social Media ◽  
Language Processing ◽  
Text Classification ◽  
High Performance ◽  
Language Models ◽  
Learning Performance ◽  
Health Related ◽  
Social Media Text ◽  
Performance Results ◽  
Better Than

Motivation Pretrained contextual language models proposed in the recent past have been reported to achieve state-of-the-art performances in many natural language processing (NLP) tasks. There is a need to benchmark such models for targeted NLP tasks, and to explore effective pretraining strategies to improve machine learning performance. Results In this work, we addressed the task of health-related social media text classification. We benchmarked five models-RoBERTa, BERTweet, TwitterBERT, BioClinical_BERT, and BioBERT on 22 tasks. We attempted to boost performance for the best models by comparing distinct pretraining strategies-domain-adaptive pretraining (DAPT), source-adaptive pretraining (SAPT), and topic-specific pretraining (TSPT). RoBERTa and BERTweet performed comparably in most tasks, and better than others. For pretraining strategies, SAPT performed better or comparable to the off-the-shelf models, and significantly outperformed DAPT. SAPT+TSPT showed consistently high performance, with statistically significant improvement in one task. Our findings demonstrate that RoBERTa and BERTweet are excellent off-the-shelf models for health-related social media text classification, and extended pretraining using SAPT and TSPT can further improve performance.

P581Electronic health records (EHRs) data validation in atherosclerotic/cardiovascular clinical phenotypes

2019 ◽  
Vol 40 (Supplement_1) ◽  
Author(s):  
N Cruz ◽  
M Serrano ◽  
A Lopez ◽  
I H Medrano ◽  
J Lozano ◽  
...  
Keyword(s):  
Language Processing ◽  
High Performance ◽  
Medical Information ◽  
Research Quality ◽  
Free Text ◽  
Case Detection ◽  
Real World Data ◽  
Health Records ◽  
Clinical Phenotypes ◽  
Prevalent Disease

Abstract Background Research efforts to develop strategies to effectively identify patients and reduce the burden of cardiovascular diseases is essential for the future of the health system. Most research studies have used only coded parts of electronic health records (EHRs) for case-detection obtaining missed data cases, reducing study quality and in some case bias findings. Incorporating information from free-text into case-detection through Big Data and Artificial Intelligence techniques improves research quality. Savana has developed EHRead, a powerful technology that applies Natural Language Processing, Machine Learning and Deep Learning, to analyse and automatically extracts highly valuable medical information from unstructured free text contained in the EHR to support research and practice. Purpose We aimed to validate the linguistic accuracy performance of Savana, in terms of Precision (P), Recall (R) and overall performance (F-Score) in the cardiovascular domain since this is one of the most prevalent disease in the general population. This means validating the extent to which the Savana system identifies mentions to atherosclerotic/cardiovascular clinical phenotypes in EHRs. Methods The project was conducted in 3 Spanish sites and the system was validated using a corpus that consisted of 739 EHRs, including the emergency, medical and discharge records, written in free text. These EHRs were randomly selected from the total number of clinical documents generated during the period of 2012–2017 and were fully anonymized to comply with legal and ethical requirements. Two physicians per site reviewed records (randomly selected) and annotated all direct references to atherosclerotic/cardiovascular clinical phenotypes, following the annotation guidelines previously developed. A third physician adjudicated discordant annotations. Savana's performance was automatically calculated using as validation resource the gold standard created by the experts. Results We found good levels of performance achieved by Savana in the identification of mentions to atherosclerotic/cardiovascular clinical phenotypes, yielding an overall P, R, and F-score of 0.97, 0.92, and 0.94, respectively. We also found that going through all the EHRs and identifying the mentions to atherosclerotic/cardiovascular clinical phenotypes, the expert spent ∼ 60h while Savana ∼ 36 min. Conclusion(s) Innovative techniques to identify atherosclerotic/cardiovascular clinical phenotypes could be used to support real world data research and clinical practice. Overall Savana showed a high performance, comparable with those obtained by an expert physician annotator doing the same task. Additionally, a significant reduction of time in using automatic information extraction system was achieved.

scholarly journals Towards a symbiotic relationship between big data, artificial intelligence, and hospital pharmacy

2020 ◽  
Vol 13 (1) ◽  
Author(s):  
Carlos Del Rio-Bermudez ◽  
Ignacio H. Medrano ◽  
Laura Yebes ◽  
Jose Luis Poveda
Keyword(s):  
Artificial Intelligence ◽  
Big Data ◽  
Language Processing ◽  
Big Data Analytics ◽  
Improve Patient Care ◽  
Free Text ◽  
Medication History ◽  
Medical Databases ◽  
Therapeutic Outcomes ◽  
Text Information

Abstract The digitalization of health and medicine and the growing availability of electronic health records (EHRs) has encouraged healthcare professionals and clinical researchers to adopt cutting-edge methodologies in the realms of artificial intelligence (AI) and big data analytics to exploit existing large medical databases. In Hospital and Health System pharmacies, the application of natural language processing (NLP) and machine learning to access and analyze the unstructured, free-text information captured in millions of EHRs (e.g., medication safety, patients’ medication history, adverse drug reactions, interactions, medication errors, therapeutic outcomes, and pharmacokinetic consultations) may become an essential tool to improve patient care and perform real-time evaluations of the efficacy, safety, and comparative effectiveness of available drugs. This approach has an enormous potential to support share-risk agreements and guide decision-making in pharmacy and therapeutics (P&T) Committees.

scholarly journals Lexical predictability during natural reading: Effects of surprisal and entropy reduction

2017 ◽  
Author(s):  
Matthew Lowder ◽  
Wonil Choi ◽  
Fernanda Ferreira ◽  
John Henderson
Keyword(s):  
Language Processing ◽  
Sentence Processing ◽  
Large Scale ◽  
Language Models ◽  
Information Complexity ◽  
Complexity Metrics ◽  
Processing Times ◽  
Entropy Reduction ◽  
Theory Of Language ◽  
Word Predictability

What are the effects of word-by-word predictability on sentence processing times during the natural reading of a text? Although information-complexity metrics such as surprisal and entropy reduction have been useful in addressing this question, these metrics tend to be estimated using computational language models, which require some degree of commitment to a particular theory of language processing. Taking a different approach, the current study implemented a large-scale cumulative cloze task to collect word-by-word predictability data for 40 passages and compute surprisal and entropy reduction values in a theory-neutral manner. A separate group of participants read the same texts while their eye movements were recorded. Results showed that increases in surprisal and entropy reduction were both associated with increases in reading times. Further, these effects did not depend on the global difficulty of the text. The findings suggest that surprisal and entropy reduction independently contribute to variation in reading times, as these metrics seem to capture different aspects of lexical predictability.

scholarly journals Phenoscape: Semantic analysis of organismal traits and genes yields insights in evolutionary biology

2018 ◽  
Author(s):  
Paula M Mabee ◽  
Wasila M Dahdul ◽  
James P Balhoff ◽  
Hilmar Lapp ◽  
Prashanti Manda ◽  
...  
Keyword(s):  
Language Processing ◽  
Evolutionary Biology ◽  
Domain Knowledge ◽  
Large Scale ◽  
Semantic Analysis ◽  
Comparative Anatomy ◽  
Semantic Annotation ◽  
Model Organisms ◽  
Free Text ◽  
Mutant Phenotypes

The study of how the observable features of organisms, i.e., their phenotypes, result from the complex interplay between genetics, development, and the environment, is central to much research in biology. The varied language used in the description of phenotypes, however, impedes the large scale and interdisciplinary analysis of phenotypes by computational methods. The Phenoscape project (www.phenoscape.org) has developed semantic annotation tools and a gene–phenotype knowledgebase, the Phenoscape KB, that uses machine reasoning to connect evolutionary phenotypes from the comparative literature to mutant phenotypes from model organisms. The semantically annotated data enables the linking of novel species phenotypes with candidate genes that may underlie them. Semantic annotation of evolutionary phenotypes further enables previously difficult or novel analyses of comparative anatomy and evolution. These include generating large, synthetic character matrices of presence/absence phenotypes based on inference, and searching for taxa and genes with similar variation profiles using semantic similarity. Phenoscape is further extending these tools to enable users to automatically generate synthetic supermatrices for diverse character types, and use the domain knowledge encoded in ontologies for evolutionary trait analysis. Curating the annotated phenotypes necessary for this research requires significant human curator effort, although semi-automated natural language processing tools promise to expedite the curation of free text. As semantic tools and methods are developed for the biodiversity sciences, new insights from the increasingly connected stores of interoperable phenotypic and genetic data are anticipated.

scholarly journals What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations

2021 ◽  
Vol 4 ◽  
Author(s):  
Nikolai Ilinykh ◽  
Simon Dobnik
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Large Scale ◽  
Visual Representations ◽  
Language Models ◽  
Visual Stream ◽  
Visual Knowledge ◽  
Language And Vision ◽  
The Impact ◽  
Image Descriptions

Neural networks have proven to be very successful in automatically capturing the composition of language and different structures across a range of multi-modal tasks. Thus, an important question to investigate is how neural networks learn and organise such structures. Numerous studies have examined the knowledge captured by language models (LSTMs, transformers) and vision architectures (CNNs, vision transformers) for respective uni-modal tasks. However, very few have explored what structures are acquired by multi-modal transformers where linguistic and visual features are combined. It is critical to understand the representations learned by each modality, their respective interplay, and the task’s effect on these representations in large-scale architectures. In this paper, we take a multi-modal transformer trained for image captioning and examine the structure of the self-attention patterns extracted from the visual stream. Our results indicate that the information about different relations between objects in the visual stream is hierarchical and varies from local to a global object-level understanding of the image. In particular, while visual representations in the first layers encode the knowledge of relations between semantically similar object detections, often constituting neighbouring objects, deeper layers expand their attention across more distant objects and learn global relations between them. We also show that globally attended objects in deeper layers can be linked with entities described in image descriptions, indicating a critical finding - the indirect effect of language on visual representations. In addition, we highlight how object-based input representations affect the structure of learned visual knowledge and guide the model towards more accurate image descriptions. A parallel question that we investigate is whether the insights from cognitive science echo the structure of representations that the current neural architecture learns. The proposed analysis of the inner workings of multi-modal transformers can be used to better understand and improve on such problems as pre-training of large-scale multi-modal architectures, multi-modal information fusion and probing of attention weights. In general, we contribute to the explainable multi-modal natural language processing and currently shallow understanding of how the input representations and the structure of the multi-modal transformer affect visual representations.

scholarly journals WinoGrande

2021 ◽  
Vol 64 (9) ◽  
pp. 99-106
Author(s):  
Keisuke Sakaguchi ◽  
Ronan Le Bras ◽  
Chandra Bhagavatula ◽  
Yejin Choi
Keyword(s):  
High Performance ◽  
Large Scale ◽  
State Of The Art ◽  
Bias Reduction ◽  
Language Models ◽  
Systematic Bias ◽  
Commonsense Reasoning ◽  
Word Associations ◽  
Lower Accuracy ◽  
Key Steps

Commonsense reasoning remains a major challenge in AI, and yet, recent progresses on benchmarks may seem to suggest otherwise. In particular, the recent neural language models have reported above 90% accuracy on the Winograd Schema Challenge (WSC), a commonsense benchmark originally designed to be unsolvable for statistical models that rely simply on word associations. This raises an important question---whether these models have truly acquired robust commonsense capabilities or they rely on spurious biases in the dataset that lead to an overestimation of the true capabilities of machine commonsense. To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) large-scale crowdsourcing, followed by (2) systematic bias reduction using a novel AFLITE algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. Our experiments demonstrate that state-of-the-art models achieve considerably lower accuracy (59.4%-79.1%) on WINOGRANDE compared to humans (94%), confirming that the high performance on the original WSC was inflated by spurious biases in the dataset. Furthermore, we report new state-of-the-art results on five related benchmarks with emphasis on their dual implications. On the one hand, they demonstrate the effectiveness of WINOGRANDE when used as a resource for transfer learning. On the other hand, the high performance on all these benchmarks suggests the extent to which spurious biases are prevalent in all such datasets, which motivates further research on algorithmic bias reduction.

scholarly journals Machine learning and big scientific data

2020 ◽  
Vol 378 (2166) ◽  
pp. 20190054 ◽  
Author(s):  
Tony Hey ◽  
Keith Butler ◽  
Sam Jackson ◽  
Jeyarajan Thiyagalingam
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Language Processing ◽  
High Performance ◽  
Large Scale ◽  
Materials Science ◽  
Numerical Algorithms ◽  
Scientific Data ◽  
Learning Technology ◽  
Challenges And Opportunities

This paper reviews some of the challenges posed by the huge growth of experimental data generated by the new generation of large-scale experiments at UK national facilities at the Rutherford Appleton Laboratory (RAL) site at Harwell near Oxford. Such ‘Big Scientific Data’ comes from the Diamond Light Source and Electron Microscopy Facilities, the ISIS Neutron and Muon Facility and the UK's Central Laser Facility. Increasingly, scientists are now required to use advanced machine learning and other AI technologies both to automate parts of the data pipeline and to help find new scientific discoveries in the analysis of their data. For commercially important applications, such as object recognition, natural language processing and automatic translation, deep learning has made dramatic breakthroughs. Google's DeepMind has now used the deep learning technology to develop their AlphaFold tool to make predictions for protein folding. Remarkably, it has been able to achieve some spectacular results for this specific scientific problem. Can deep learning be similarly transformative for other scientific problems? After a brief review of some initial applications of machine learning at the RAL, we focus on challenges and opportunities for AI in advancing materials science. Finally, we discuss the importance of developing some realistic machine learning benchmarks using Big Scientific Data coming from several different scientific domains. We conclude with some initial examples of our ‘scientific machine learning’ benchmark suite and of the research challenges these benchmarks will enable. This article is part of a discussion meeting issue ‘Numerical algorithms for high-performance computational science’.

Parsing Noun Phrases in the Penn Treebank

2011 ◽  
Vol 37 (4) ◽  
pp. 753-809 ◽  
Author(s):  
David Vadas ◽  
James R. Curran
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Gold Standard ◽  
High Performance ◽  
Large Scale ◽  
Complex Structure ◽  
Noun Phrases ◽  
Simple Task ◽  
Data Set ◽  
Lexical Information

Noun phrases (nps) are a crucial part of natural language, and can have a very complex structure. However, this np structure is largely ignored by the statistical parsing field, as the most widely used corpus is not annotated with it. This lack of gold-standard data has restricted previous efforts to parse nps, making it impossible to perform the supervised experiments that have achieved high performance in so many Natural Language Processing (nlp) tasks. We comprehensively solve this problem by manually annotating np structure for the entire Wall Street Journal section of the Penn Treebank. The inter-annotator agreement scores that we attain dispel the belief that the task is too difficult, and demonstrate that consistent np annotation is possible. Our gold-standard np data is now available for use in all parsers. We experiment with this new data, applying the Collins (2003) parsing model, and find that its recovery of np structure is significantly worse than its overall performance. The parser's F-score is up to 5.69% lower than a baseline that uses deterministic rules. Through much experimentation, we determine that this result is primarily caused by a lack of lexical information. To solve this problem we construct a wide-coverage, large-scale np Bracketing system. With our Penn Treebank data set, which is orders of magnitude larger than those used previously, we build a supervised model that achieves excellent results. Our model performs at 93.8% F-score on the simple task that most previous work has undertaken, and extends to bracket longer, more complex nps that are rarely dealt with in the literature. We attain 89.14% F-score on this much more difficult task. Finally, we implement a post-processing module that brackets nps identified by the Bikel (2004) parser. Our np Bracketing model includes a wide variety of features that provide the lexical information that was missing during the parser experiments, and as a result, we outperform the parser's F-score by 9.04%. These experiments demonstrate the utility of the corpus, and show that many nlp applications can now make use of np structure.

Sign in / Sign up

Export Citation Format

Share Document