Abstract
Objective
With COVID-19 there was a need for rapidly scalable annotation system that facilitated real-time integration with clinical decision support systems (CDS). Current annotation systems suffer from high resource utilization and poor scalability limiting real-world integration with CDS. A potential solution to mitigate these issues is to use the rule-based gazetteer developed at our institution.
Materials and Methods
Performance, resource utilization and runtime of the rule-based gazetteer was compared with five annotation systems: BioMedICUS, cTAKES, MetaMap, CLAMP and MedTagger.
Results
This rule-based gazetteer was fastest, had low resource footprint and similar performance for weighted micro-average and macro-average measures of precision, recall and f1-score compared to other annotation systems.
Discussion
Opportunities to increase its performance include fine-tuning lexical rules for symptom identification. Additionally, it could run on multiple compute nodes for faster runtime.
Conclusion
This rule-based gazetteer overcame key technical limitations facilitating real-time symptomatology identification for COVID-19 and integration of unstructured data elements into our CDS. It is ideal for large-scale deployment across a wide variety of health care settings for surveillance of acute COVID-19 symptoms for integration into prognostic modeling. Such a system is currently being leveraged for monitoring of post-acute sequelae of COVID-19 (PASC) progression in COVID-19 survivors. This study conducted the first in-depth analysis and developed a rule-based gazetteer for COVID-19 symptom extraction with the following key features: low processor and memory utilization, faster runtime and similar weighted micro-average and macro-average measures for precision, recall and f1-score compared to industry standard annotation systems.
Lay Summary
With COVID-19 came an unprecedented need to identify symptoms of COVID-19 patients under investigation (PUIs) in a time sensitive, resource-efficient and accurate manner. While available annotation systems perform well for smaller healthcare settings, they fail to scale in larger healthcare systems where 10,000+ clinical notes are generated a day. This study covers 3 improvements addressing key limitations of current annotation systems. (1) High resource utilization and poor scalability of existing annotation systems. The presented rule-based gazetteer is a high-throughput annotation system for processing high volume of notes, thus, providing opportunity for clinicians to make more informed time-sensitive decisions around patient care. (2) Equally important is our developed rule-based gazetteer performs similar or better than current annotation systems for symptom identification. (3) Due to minimal resource needs of the rule-based gazetteer, it could be deployed at healthcare sites lacking a robust infrastructure where industry standard annotation systems cannot be deployed because of low resource availability.
A Rule-Based Video Annotation System