EasyTurk: A User-Friendly Interface for High-Quality Linguistic Annotation with Amazon Mechanical Turk

ABSTRACT Amazon Mechanical Turk (MTurk) is a powerful tool that is more commonly being used to recruit behavioral research participants for accounting research. This manuscript provides practical and technical knowledge learned from firsthand experience to help researchers collect high-quality, defendable data for research purposes. We highlight two issues of particular importance when using MTurk: (1) accessing qualified participants, and (2) validating collected data. To address these issues, we discuss alternative methods of carrying out screens and different data validation techniques researchers may want to consider. We also demonstrate how some of the techniques discussed were implemented for a recent data collection. Finally, we contrast the use of unpaid screens with merely putting participation requirements in the MTurk instructions to examine the effectiveness of using screens. We find that screening questions significantly reduce the number of manipulation check failures as well as significantly increase the usable responses per paid participant.

Download Full-text

Amazon Mechanical Turk

Cross-Disciplinary Advances in Applied Natural Language Processing ◽

10.4018/978-1-61350-447-5.ch007 ◽

2012 ◽

pp. 90-102

Author(s):

Amber Chauncey Strain ◽

Lucille M. Booker

Keyword(s):

Mechanical Turk ◽

Amazon Mechanical Turk ◽

Quality Of Data ◽

High Quality ◽

Web Based ◽

Potential Benefits ◽

Amazon's Mechanical Turk ◽

Short Time ◽

Time Frames

One of the major challenges of ANLP research is the constant balancing act between the need for large samples, and the excessive time and monetary resources necessary for acquiring those samples. Amazon’s Mechanical Turk (MTurk) is a web-based data collection tool that has become a premier resource for researchers who are interested in optimizing their sample sizes and minimizing costs. Due to its supportive infrastructure, diverse participant pool, quality of data, and time and cost efficiency, MTurk seems particularly suitable for ANLP researchers who are interested in gathering large, high quality corpora in relatively short time frames. In this chapter, the authors first provide a broad description of the MTurk interface. Next, they describe the steps for acquiring IRB approval of MTurk experiments, designing experiments using the MTurk dashboard, and managing data. Finally, the chapter concludes by discussing the potential benefits and limitations of using MTurk for ANLP experimentation.

Download Full-text

Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup

Cornell Hospitality Quarterly ◽

10.1177/19389655211025475 ◽

2021 ◽

pp. 193896552110254

Author(s):

Lu Lu ◽

Nathan Neale ◽

Nathaniel D. Line ◽

Mark Bonn

Keyword(s):

Best Practices ◽

Data Quality ◽

Mechanical Turk ◽

Amazon Mechanical Turk ◽

Quality Data ◽

High Quality ◽

High Quality Data ◽

Amazon's Mechanical Turk ◽

Associated Data

As the use of Amazon’s Mechanical Turk (MTurk) has increased among social science researchers, so, too, has research into the merits and drawbacks of the platform. However, while many endeavors have sought to address issues such as generalizability, the attentiveness of workers, and the quality of the associated data, there has been relatively less effort concentrated on integrating the various strategies that can be used to generate high-quality data using MTurk samples. Accordingly, the purpose of this research is twofold. First, existing studies are integrated into a set of strategies/best practices that can be used to maximize MTurk data quality. Second, focusing on task setup, selected platform-level strategies that have received relatively less attention in previous research are empirically tested to further enhance the contribution of the proposed best practices for MTurk usage.

Download Full-text

Prioritising recommendations following analyses of adverse events in healthcare: a systematic review

BMJ Open Quality ◽

10.1136/bmjoq-2019-000843 ◽

2020 ◽

Vol 9 (4) ◽

pp. e000843

Author(s):

Kelly Bos ◽

Maarten J van der Laan ◽

Dave A Dongelmans

Keyword(s):

Systematic Review ◽

Adverse Events ◽

Data Extraction ◽

Data Sources ◽

Cochrane Library ◽

High Quality ◽

Data Synthesis ◽

Study Selection ◽

Personal Opinion ◽

User Friendly

PurposeThe purpose of this systematic review was to identify an appropriate method—a user-friendly and validated method—that prioritises recommendations following analyses of adverse events (AEs) based on objective features.Data sourcesThe electronic databases PubMed/MEDLINE, Embase (Ovid), Cochrane Library, PsycINFO (Ovid) and ERIC (Ovid) were searched.Study selectionStudies were considered eligible when reporting on methods to prioritise recommendations.Data extractionTwo teams of reviewers performed the data extraction which was defined prior to this phase.Results of data synthesisEleven methods were identified that are designed to prioritise recommendations. After completing the data extraction, none of the methods met all the predefined criteria. Nine methods were considered user-friendly. One study validated the developed method. Five methods prioritised recommendations based on objective features, not affected by personal opinion or knowledge and expected to be reproducible by different users.ConclusionThere are several methods available to prioritise recommendations following analyses of AEs. All these methods can be used to discuss and select recommendations for implementation. None of the methods is a user-friendly and validated method that prioritises recommendations based on objective features. Although there are possibilities to further improve their features, the ‘Typology of safety functions’ by de Dianous and Fiévez, and the ‘Hierarchy of hazard controls’ by McCaughan have the most potential to select high-quality recommendations as they have only a few clearly defined categories in a well-arranged ordinal sequence.

Download Full-text