Disease Named Entity Recognition in Medical Records: Analysis of Transfer Learning between Different Hospital Departments (Preprint)

BACKGROUND Electronic records provide rich clinical information for biomedical text mining. However, a system developed on one hospital department may not generalize to other departments. Here, we use hospital medical records as a research data source and explore the heterogeneous problem posed by different hospital departments. OBJECTIVE We use MIMIC-III hospital medical records as the research data source. We collaborate with medical experts to annotate the data, with 328 records being included in analyses. Disease named entity recognition (NER), which helps medical experts in consolidating diagnoses, is undertaken as a case study. METHODS To compare heterogeneity of medical records across departments, we access text from multiple departments and employ the similarity metrics. We apply transfer learning to NER in different departments’ records and test the correlation between performance and similarity metrics. We use TF-IDF cosine similarity of the named entities as our similarity metric. We use three pretrained model on the disease NER task to valid the consistency of the result. RESULTS The disease NER dataset we release consists of 328 medical records from MIMIC-III, with 95629 sentences and 8884 disease mentions in total. The inter annotator agreement Cohen’s kappa coefficient is 0.86. Similarity metrics support that medical records from different departments are heterogeneous, ranges from 0.1004 to 0.3541 compare to Medical department. In the transfer learning task using the Medical department as the training set, F1 score performs in three pretrained models average from 0.847 to 0.863. F1 scores correlate with similarity metrics with Spearman’s coefficient of 0.4285. CONCLUSIONS We propose a disease NER dataset based on medical records from MIMIC-III and demonstrate the effectiveness of transfer learning using BERT. Similarity metrics reveal noticeable heterogeneity between department records. The deep learning-based transfer learning method demonstrates good ability to generalize across departments and achieve decent NER performance thus eliminates the concern that training material from one hospital might compromise model performance when applied to another. However, the model performance does not show high correlation to the departments’ similarity.