Using MTurk to Distribute a Survey or Experiment: Methodological Considerations

ABSTRACT Amazon Mechanical Turk (MTurk) is a powerful tool that is more commonly being used to recruit behavioral research participants for accounting research. This manuscript provides practical and technical knowledge learned from firsthand experience to help researchers collect high-quality, defendable data for research purposes. We highlight two issues of particular importance when using MTurk: (1) accessing qualified participants, and (2) validating collected data. To address these issues, we discuss alternative methods of carrying out screens and different data validation techniques researchers may want to consider. We also demonstrate how some of the techniques discussed were implemented for a recent data collection. Finally, we contrast the use of unpaid screens with merely putting participation requirements in the MTurk instructions to examine the effectiveness of using screens. We find that screening questions significantly reduce the number of manipulation check failures as well as significantly increase the usable responses per paid participant.

Download Full-text

Amazon Mechanical Turk

Cross-Disciplinary Advances in Applied Natural Language Processing ◽

10.4018/978-1-61350-447-5.ch007 ◽

2012 ◽

pp. 90-102

Author(s):

Amber Chauncey Strain ◽

Lucille M. Booker

Keyword(s):

Mechanical Turk ◽

Amazon Mechanical Turk ◽

Quality Of Data ◽

High Quality ◽

Web Based ◽

Potential Benefits ◽

Amazon's Mechanical Turk ◽

Short Time ◽

Time Frames

One of the major challenges of ANLP research is the constant balancing act between the need for large samples, and the excessive time and monetary resources necessary for acquiring those samples. Amazon’s Mechanical Turk (MTurk) is a web-based data collection tool that has become a premier resource for researchers who are interested in optimizing their sample sizes and minimizing costs. Due to its supportive infrastructure, diverse participant pool, quality of data, and time and cost efficiency, MTurk seems particularly suitable for ANLP researchers who are interested in gathering large, high quality corpora in relatively short time frames. In this chapter, the authors first provide a broad description of the MTurk interface. Next, they describe the steps for acquiring IRB approval of MTurk experiments, designing experiments using the MTurk dashboard, and managing data. Finally, the chapter concludes by discussing the potential benefits and limitations of using MTurk for ANLP experimentation.

Download Full-text

Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup

Cornell Hospitality Quarterly ◽

10.1177/19389655211025475 ◽

2021 ◽

pp. 193896552110254

Author(s):

Lu Lu ◽

Nathan Neale ◽

Nathaniel D. Line ◽

Mark Bonn

Keyword(s):

Best Practices ◽

Data Quality ◽

Mechanical Turk ◽

Amazon Mechanical Turk ◽

Quality Data ◽

High Quality ◽

High Quality Data ◽

Amazon's Mechanical Turk ◽

Associated Data

As the use of Amazon’s Mechanical Turk (MTurk) has increased among social science researchers, so, too, has research into the merits and drawbacks of the platform. However, while many endeavors have sought to address issues such as generalizability, the attentiveness of workers, and the quality of the associated data, there has been relatively less effort concentrated on integrating the various strategies that can be used to generate high-quality data using MTurk samples. Accordingly, the purpose of this research is twofold. First, existing studies are integrated into a set of strategies/best practices that can be used to maximize MTurk data quality. Second, focusing on task setup, selected platform-level strategies that have received relatively less attention in previous research are empirically tested to further enhance the contribution of the proposed best practices for MTurk usage.

Download Full-text

The emotion storyboard: A method to examine social judgments of emotion

PLoS ONE ◽

10.1371/journal.pone.0249294 ◽

2021 ◽

Vol 16 (4) ◽

pp. e0249294

Author(s):

Kaitlin McCormick-Huhn ◽

Stephanie A. Shields

Keyword(s):

Response Times ◽

Mechanical Turk ◽

New Method ◽

Graphic Novel ◽

Amazon Mechanical Turk ◽

Main Character ◽

Social Judgments ◽

Manipulation Check ◽

Emotion Control ◽

Emotion Type

As perceivers, we need to understand context to make social judgments about emotion, such as judging whether emotion is appropriate. We propose a graphic novel-like method, the emotion storyboard, for use in research on social judgments of emotion. Across two studies, participants were randomly assigned to read emotion storyboards or written vignettes to compare the efficacy of the emotion storyboard to that of vignettes in studies on social judgments of emotion. In Study 1, undergraduates (N = 194) answered comprehension questions and rated story clarity and immersion. Participants also made social judgments of emotion by rating main character emotion control and appropriateness of intensity. To further compare the efficacy of the methods, in Study 2, Amazon Mechanical Turk workers (N = 213) answered comprehension questions while response times were recorded, rated clarity, answered a race manipulation check, and rated main character emotion type appropriateness. Overall, emotion storyboards resulted in greater clarity ratings, greater race manipulation check accuracy, and in some instances, enhanced comprehension and comprehension response times relative to vignettes. In emotion storyboards, main character emotion was rated more controlled and more appropriate in intensity, but not different in emotion type appropriateness, than in vignettes. Overall, the method offers a new method of examining social elements of emotion that enhances comprehension and maximizes experimental efficiency.

Download Full-text