STAT Simple Text Annotation Tool (STAT): Web-based Tool for Creating Training Data for spaCy Models

2022 ◽  
pp. 299-305
Author(s):  
Darshita Kumar ◽  
Kshitija Choudhari ◽  
Pooja Patel ◽  
Shambhavi Pandey ◽  
Aparna Hajare ◽  
...  
Keyword(s):  
Training Data ◽  
Annotation Tool ◽  
Web Based ◽  
Text Annotation

scholarly journals TeamTat: a collaborative text annotation tool

2020 ◽  
Vol 48 (W1) ◽  
pp. W5-W11
Author(s):  
Rezarta Islamaj ◽  
Dongseop Kwon ◽  
Sun Kim ◽  
Zhiyong Lu
Keyword(s):  
Project Managers ◽  
Biomedical Literature ◽  
Task Completion ◽  
Annotation Tool ◽  
Web Based ◽  
Domain Experts ◽  
Plain Text ◽  
Text Annotation ◽  
User Friendly ◽  
Input Format

Abstract Manually annotated data is key to developing text-mining and information-extraction algorithms. However, human annotation requires considerable time, effort and expertise. Given the rapid growth of biomedical literature, it is paramount to build tools that facilitate speed and maintain expert quality. While existing text annotation tools may provide user-friendly interfaces to domain experts, limited support is available for figure display, project management, and multi-user team annotation. In response, we developed TeamTat (https://www.teamtat.org), a web-based annotation tool (local setup available), equipped to manage team annotation projects engagingly and efficiently. TeamTat is a novel tool for managing multi-user, multi-label document annotation, reflecting the entire production life cycle. Project managers can specify annotation schema for entities and relations and select annotator(s) and distribute documents anonymously to prevent bias. Document input format can be plain text, PDF or BioC (uploaded locally or automatically retrieved from PubMed/PMC), and output format is BioC with inline annotations. TeamTat displays figures from the full text for the annotator's convenience. Multiple users can work on the same document independently in their workspaces, and the team manager can track task completion. TeamTat provides corpus quality assessment via inter-annotator agreement statistics, and a user-friendly interface convenient for annotation review and inter-annotator disagreement resolution to improve corpus quality.

Development of a Web-Based Crowdsourcing-Integrated Semantic Text Annotation Tool to Assist in Building a Mental Health Knowledge Base: User-Centered Design Approach (Preprint)

2020 ◽  
Author(s):  
Xing He ◽  
Hansi Zhang ◽  
Jiang Bian
Keyword(s):  
Mental Health ◽  
Scientific Literature ◽  
Usability Testing ◽  
Annotation Tool ◽  
User Centered Design ◽  
Web Based ◽  
Text Annotation ◽  
Mental Health Information ◽  
Annotation Guideline

BACKGROUND One in five U.S. adults lives with some kind of mental health condition and 4.6% of all U.S. adults have a serious mental illness in 2018. The Internet has become the first place for these people to seek online mental health information for help. However, online mental health information is not well-organized and often of low quality. There have been efforts in building evidence-based mental health knowledgebases curated with information manually extracted from the high-quality scientific literature. Manual extraction is inefficient. Crowdsourcing can potentially be a low-cost mechanism to collect labeled data from non-expert laypeople. However, there is not an existing annotation tool integrated with popular crowdsourcing platforms to perform the information extraction tasks. In our previous work, we prototyped a Semantic Text Annotation Tool (STAT) to address this gap. OBJECTIVE We aimed to refine the STAT prototype (1) to improve its usability and (2) to enhance the crowdsourcing workflow efficiency to facilitate the construction of evidence-based mental health knowledgebase, following a user-centered design (UCD) process. METHODS Following UCD principles, we conducted four design iterations to improve the initial STAT prototype. In the first two iterations, usability testing focus groups were conducted internally with 8 participants recruited from a convenient sample, and the usability was evaluated with a modified System Usability Scale (SUS). In the following two iterations, usability testing was conducted externally using the Amazon Mechanical Turk (MTurk) platform. In each iteration, we summarized the usability testing results through thematic analysis, identified usability issues, and conducted a heuristic evaluation to map identified usability issues to Jakob Nielsen’s usability heuristics. We collected suggested improvements in each of the usability testing sessions and enhanced STAT accordingly in the next UCD iteration. After four UCD iterations, we conducted a case study of the system on MTurk using mental health related scientific literature. We compared the performance of crowdsourcing workers with two expert annotators from two aspects: efficiency and quality. RESULTS At the end of two initial internal UCD iterations, the SUS score increased from 70.3 ± 12.5 to 81.1 ± 9.8 after we improved STAT following the suggested improvements. We then evaluated STAT externally through MTurk in the following two iterations. The SUS score decreased to 55.7 ± 20.1 in the third iteration, probably because of the complexity of the tasks. After further simplification of STAT and the annotation tasks with an improved annotation guideline, the SUS score increased to 73.8 ± 13.8 in the fourth iteration of UCD. In the evaluation case study, on average, the workers spent 125.5 ± 69.2 seconds on the onboarding tutorial and the crowdsourcing workers spent significantly less time on the annotation tasks compared to the two experts. In terms of annotation quality, the workers’ annotation results achieved average F1-scores ranged from 0.62 to 0.84 for the different sentences. CONCLUSIONS We successfully developed a web-based semantic text annotation tool, STAT, to facilitate the curation of semantic web knowledgebases through four UCD iterations. The lessons learned from the UCD process could serve as a guide to further enhance STAT and the development and design of other crowdsourcing-based semantic text annotation tasks. Our study also showed that a well-organized, informative annotation guideline is as important as the annotation tool itself. Further, we learned that a crowdsourcing task should consist of multiple simple microtasks rather than a complicated task.

scholarly journals Ugarit: Translation Alignment Visualization

2020 ◽  
Author(s):  
Tariq Yousef
Keyword(s):  
User Interface ◽  
Training Data ◽  
Learning Tool ◽  
Web Based ◽  
Automatic Translation ◽  
Short Overview ◽  
Alignment System ◽  
Visualization Techniques ◽  
Parallel Texts

In this paper we introduce Ugarit a web-based tool for manual translation alignment of parallel texts, the aim was to build a user interface to create training data in form of translation pairs to be used later for an automatic translation alignment system at word/phrase level, the tool now is widely used as reading environment for parallel texts and a languages learning tool. The paper gives a short overview of the visualization techniques used to present the alignment results and shows how the translation graph derived from the aligned translation pairs.

scholarly journals Detection Of Spam Comments On Instagram Using Complementary Naïve Bayes

2019 ◽  
Vol 13 (3) ◽  
pp. 263
Author(s):  
Nur Azizul Haqimi ◽  
Nur Rokhman ◽  
Sigit Priyanta
Keyword(s):  
Social Media ◽  
Support Vector Machine ◽  
Test Data ◽  
Training Data ◽  
Classification Method ◽  
Support Vector ◽  
Test Results ◽  
Imbalanced Dataset ◽  
Web Based ◽  
F Measure

Instagram (IG) is a web-based and mobile social media application where users can share photos or videos with available features. Upload photos or videos with captions that contain an explanation of the photo or video that can reap spam comments. Comments on spam containing comments that are not relevant to the caption and photos. The problem that arises when identifying spam is non-spam comments are more dominant than spam comments so that it leads to the problem of the imbalanced dataset. A balanced dataset can influence the performance of a classification method. This is the focus of research related to the implementation of the CNB method in dealing with imbalance datasets for the detection of Instagram spam comments. The study used TF-IDF weighting with Support Vector Machine (SVM) as a comparison classification. Based on the test results with 2500 training data and 100 test data on the imbalanced dataset (25% spam and 75% non-spam), the CNB accuracy was 92%, precision 86% and f-measure 93%. Whereas SVM produces 87% accuracy, 79% precision, 88% f-measure. In conclusion, the CNB method is more suitable for detecting spam comments in cases of imbalanced datasets.

scholarly journals HistoFlow: Label-Efficient and Interactive Deep Learning Cell Analysis

2020 ◽  
Author(s):  
Tim Henning ◽  
Benjamin Bergner ◽  
Christoph Lippert
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
User Interface ◽  
Network Architecture ◽  
Training Data ◽  
Cell Segmentation ◽  
Annotation Tool ◽  
Cell Analysis ◽  
Instance Segmentation

Instance segmentation is a common task in quantitative cell analysis. While there are many approaches doing this using machine learning, typically, the training process requires a large amount of manually annotated data. We present HistoFlow, a software for annotation-efficient training of deep learning models for cell segmentation and analysis with an interactive user interface.It provides an assisted annotation tool to quickly draw and correct cell boundaries and use biomarkers as weak annotations. It also enables the user to create artificial training data to lower the labeling effort. We employ a universal U-Net neural network architecture that allows accurate instance segmentation and the classification of phenotypes in only a single pass of the network. Transfer learning is available through the user interface to adapt trained models to new tissue types.We demonstrate HistoFlow for fluorescence breast cancer images. The models trained using only artificial data perform comparably to those trained with time-consuming manual annotations. They outperform traditional cell segmentation algorithms and match state-of-the-art machine learning approaches. A user test shows that cells can be annotated six times faster than without the assistance of our annotation tool. Extending a segmentation model for classification of epithelial cells can be done using only 50 to 1500 annotations.Our results show that, unlike previous assumptions, it is possible to interactively train a deep learning model in a matter of minutes without many manual annotations.

Sign in / Sign up

Export Citation Format

Share Document