A Technical Guide to Using Amazon's Mechanical Turk in Behavioral Accounting Research

2017 ◽  
Vol 30 (1) ◽  
pp. 111-122 ◽  
Author(s):  
Steve Buchheit ◽  
Marcus M. Doxey ◽  
Troy Pollard ◽  
Shane R. Stinson
Keyword(s):  
Third Party ◽  
Mechanical Turk ◽  
Behavioral Accounting ◽  
Online Data ◽  
Research Participants ◽  
Traditional Research ◽  
Potential Benefits ◽  
Technical Guide ◽  
Online Data Collection ◽  
Amazon's Mechanical Turk

ABSTRACT Multiple social science researchers claim that online data collection, mainly via Amazon's Mechanical Turk (MTurk), has revolutionized the behavioral sciences (Gureckis et al. 2016; Litman, Robinson, and Abberbock 2017). While MTurk-based research has grown exponentially in recent years (Chandler and Shapiro 2016), reasonable concerns have been raised about online research participants' ability to proxy for traditional research participants (Chandler, Mueller, and Paolacci 2014). This paper reviews recent MTurk research and provides further guidance for recruiting samples of MTurk participants from populations of interest to behavioral accounting researchers. First, we provide guidance on the logistics of using MTurk and discuss the potential benefits offered by TurkPrime, a third-party service provider. Second, we discuss ways to overcome challenges related to targeted participant recruiting in an online environment. Finally, we offer suggestions for disclosures that authors may provide about their efforts to attract participants and analyze responses.

Results from the Mechanical Turk Online Audio Recordings

2019 ◽  
pp. 75-112
Author(s):  
James N. Stanford
Keyword(s):  
Data Collection ◽  
New England ◽  
Large Scale ◽  
Statistical Analyses ◽  
Mechanical Turk ◽  
Online Data ◽  
Audio Recordings ◽  
Online Data Collection ◽  
Amazon's Mechanical Turk ◽  
Broad Scale

This is the first of the two chapters (Chapters 4 and 5) that present the results of the online data collection project using Amazon’s Mechanical Turk system. These projects provide a broad-scale “bird’s eye” view of New England dialect features across large distances. This chapter examines the results from 626 speakers who audio-recorded themselves reading 12 sentences two times each. The recordings were analyzed acoustically and then modeled statistically and graphically. The results are presented in the form of maps and statistical analyses, with the goal of providing a large-scale geographic overview of modern-day patterns of New England dialect features.

Online Instrument Delivery and Participant Recruitment Services: Emerging Opportunities for Behavioral Accounting Research

2013 ◽  
Vol 26 (1) ◽  
pp. 1-23 ◽  
Author(s):  
Duane M. Brandon ◽  
James H. Long ◽  
Tina M. Loraas ◽  
Jennifer Mueller-Phillips ◽  
Brian Vansant
Keyword(s):  
Mechanical Turk ◽  
Professional Organizations ◽  
Behavioral Accounting ◽  
Participant Recruitment ◽  
Accounting Research ◽  
Research Participants ◽  
Delivery Services ◽  
Geographically Distributed ◽  
Amazon's Mechanical Turk

ABSTRACT Behavioral accounting researchers have historically been constrained in their ability to reach externally valid research participants. The purpose of this paper is to familiarize researchers with two relatively new and innovative ways to overcome this issue. First, this paper discusses two online instrument delivery services provided by SurveyMonkey and Qualtrics that can be used to distribute experimental materials to geographically distributed participants quickly and inexpensively. Second, it reviews a number of participant recruitment services that behavioral accounting researchers can use to identify and recruit externally valid research participants. Specifically, this paper discusses commercial participant recruitment services provided by SurveyMonkey Audience, Qualtrics, Amazon's Mechanical Turk, and other commercial firms, as well as several non-commercial participant recruitment services associated with industry and professional organizations. Each service is evaluated against three criteria that are important to behavioral accounting researchers: (1) cost, (2) flexibility, and (3) access to populations of interest.

Review of Best Practice Recommendations for Ensuring High Quality Data with Amazon’s Mechanical Turk

2020 ◽  
Author(s):  
Brian Bauer ◽  
Kristy L. Larsen ◽  
Nicole Caulfield ◽  
Domynic Elder ◽  
Sara Jordan ◽  
...  
Keyword(s):  
Screening Method ◽  
Scientific Progress ◽  
Worked Examples ◽  
Mechanical Turk ◽  
Quality Data ◽  
Screening Methods ◽  
Online Data ◽  
Amazon's Mechanical Turk ◽  
Data Screening ◽  
Psychological Studies

Our ability to make scientific progress is dependent upon our interpretation of data. Thus, analyzing only those data that are an honest representation of a sample is imperative for drawing accurate conclusions that allow for robust, generalizable, and replicable scientific findings. Unfortunately, a consistent line of evidence indicates the presence of inattentive/careless responders who provide low-quality data in surveys, especially on popular online crowdsourcing platforms such as Amazon’s Mechanical Turk (MTurk). Yet, the majority of psychological studies using surveys only conduct outlier detection analyses to remove problematic data. Without carefully examining the possibility of low-quality data in a sample, researchers risk promoting inaccurate conclusions that interfere with scientific progress. Given that knowledge about data screening methods and optimal online data collection procedures are scattered across disparate disciplines, the dearth of psychological studies using more rigorous methodologies to prevent and detect low-quality data is likely due to inconvenience, not maleficence. Thus, this review provides up-to-date recommendations for best practices in collecting online data and data screening methods. In addition, this article includes resources for worked examples for each screening method, a collection of recommended measures, and a preregistration template for implementing these recommendations.

Results from the Mechanical Turk Online Written Questionnaires

2019 ◽  
pp. 113-138
Author(s):  
James N. Stanford
Keyword(s):  
Data Collection ◽  
New England ◽  
Mechanical Turk ◽  
Free Response ◽  
Online Data ◽  
Regional Features ◽  
Current State ◽  
Phonological Features ◽  
Online Data Collection ◽  
Different Parts

This is the second of the two chapters (Chapters 4 and 5) that present the results of the author’s online data collection project using Mechanical Turk. This chapter analyzes the results of the online written questionnaires; 534 people responded to online questions about New England dialect features, including phonological features and lexical items. The author maps the results in terms of regional features in different parts of New England, comparing them to prior surveys and to the acoustic analyses of the prior chapter. The chapter also analyzes 100 free-response answers where New Englanders gave further insights into the current state of New England English.

scholarly journals Using Mechanical Turk to Assess the Effects of Age and Spatial Proximity on Inattentional Blindness

Collabra ◽  
2015 ◽  
Vol 1 (1) ◽  
Author(s):  
Cary R. Stothart ◽  
Walter R. Boot ◽  
Daniel J. Simons
Keyword(s):  
Cognitive Aging ◽  
Inattentional Blindness ◽  
Mechanical Turk ◽  
Spatial Proximity ◽  
Online Data ◽  
Advantages And Disadvantages ◽  
Large N ◽  
Online Data Collection ◽  
First Time ◽  
The Relationship

Few studies have used online data collection to study cognitive aging. We used a large (N = 515) online sample to replicate the findings that inattentional blindness increases with age and with the distance of the unexpected object from the focus of attention. Critically, we assessed whether distance moderates the relationship between age and noticing. We replicated both age and distance effects, but found no age by distance interaction. These findings disconfirm a plausible explanation for age differences in noticing (restricted field of view), while for the first time highlighting the advantages and disadvantages of using Mechanical Turk for the study of cognitive aging.

Amazon Mechanical Turk

2012 ◽  
pp. 90-102
Author(s):  
Amber Chauncey Strain ◽  
Lucille M. Booker
Keyword(s):  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Quality Of Data ◽  
High Quality ◽  
Web Based ◽  
Potential Benefits ◽  
Amazon's Mechanical Turk ◽  
Short Time ◽  
Time Frames

One of the major challenges of ANLP research is the constant balancing act between the need for large samples, and the excessive time and monetary resources necessary for acquiring those samples. Amazon’s Mechanical Turk (MTurk) is a web-based data collection tool that has become a premier resource for researchers who are interested in optimizing their sample sizes and minimizing costs. Due to its supportive infrastructure, diverse participant pool, quality of data, and time and cost efficiency, MTurk seems particularly suitable for ANLP researchers who are interested in gathering large, high quality corpora in relatively short time frames. In this chapter, the authors first provide a broad description of the MTurk interface. Next, they describe the steps for acquiring IRB approval of MTurk experiments, designing experiments using the MTurk dashboard, and managing data. Finally, the chapter concludes by discussing the potential benefits and limitations of using MTurk for ANLP experimentation.

scholarly journals Less Trust, Moore Verification: Determining the Accuracy of Third-Party Data through an Innovative Use of Attention Checks

2017 ◽  
Author(s):  
Nathan Seltzer
Keyword(s):  
Survey Data ◽  
Selection Criteria ◽  
Research Note ◽  
Third Party ◽  
Mechanical Turk ◽  
Quality Data ◽  
Opinion Poll ◽  
Sampling Frame ◽  
Amazon's Mechanical Turk ◽  
Response Choices

Sociologists increasingly rely on third-party internet panel platforms to acquire respondents and administer questionnaires. Yet, researchers have demonstrated that even samples sourced from well-respected and widely-adopted internet platforms such as Amazon’s Mechanical Turk are often unable to screen out respondents who do not meet selection criteria requested by researchers. Here, I argue that researchers should proactively verify that third-party survey data is accurately sampled before considering it for analysis. I propose using survey “attention checks” as a methodological solution for researchers to determine whether data vendors have provided low quality data. In this short research note, I illustrate the approach by analyzing data from a consequential political opinion poll administered on behalf of an academic polling center by a third-party internet panel vendor for a special election in 2017. By assessing valid/invalid response choices of two overlapping geographic variables, I identify irregularities in the dataset that suggest that the sample included respondents who were not within the researchers’ intended sampling frame. Attention checks provide a straightforward, inexpensive tool to improve the validity of research produced with internet-drawn samples.

Caution! MTurk Workers Ahead—Fines Doubled

2015 ◽  
Vol 8 (2) ◽  
pp. 183-190 ◽  
Author(s):  
P. D. Harms ◽  
Justin A. DeSimone
Keyword(s):  
Mechanical Turk ◽  
Online Data ◽  
Organizational Research ◽  
Target Article ◽  
Long Line ◽  
Potential Problems ◽  
Using Data ◽  
Amazon's Mechanical Turk ◽  
Online Sources ◽  
Better Than

Landers and Behrend (2015) are the most recent in a long line of researchers who have suggested that online samples generated from sources such as Amazon's Mechanical Turk (MTurk) are as good as or potentially even better than the typical samples found in psychology studies. It is important that the authors caution that researchers and reviewers need to carefully reflect on the goals of research when evaluating the appropriateness of samples. However, although they argue that certain types of samples should not be dismissed out of hand, they note that there is only scant evidence demonstrating that online sources can provide usable data for organizational research and that there is a need for further research evaluating the validity of these new sources of data. Because the target article does not directly address the potential problems with such samples, we will review what is known about collecting online data (with a particular focus on MTurk) and illustrate some potential problems using data derived from such sources.

scholarly journals Workers’ task choice heuristics as a source of emergent structure in digital microwork

2019 ◽  
Author(s):  
Otto Kässi ◽  
Vili Lehdonvirta ◽  
Jean-Michel Dalle
Keyword(s):  
Labor Markets ◽  
Preferential Attachment ◽  
Mechanical Turk ◽  
High Quality ◽  
Online Data ◽  
Employment Contracts ◽  
Task Choice ◽  
Emergent Structure ◽  
Amazon's Mechanical Turk

Digital labor markets are structured around tasks and not around fixed- or long-term employment contracts. We study the consequences of the granularization of work for digital micro workers. To address this question, we combine interview data from active online micro workers and online data on open projects scraped from Amazon's Mechanical Turk platform to study how the digital micro workers choose which tasks they work on. We find evidence for preferential attachment: workers prefer to attach themselves to experienced employers who are known to offer high quality projects. In addition, workers also clearly prefer long series of repeatable tasks over one-off tasks, even when one-off tasks pay considerably more. We thus see a re-emergence of certain types of organizational structure.

Sign in / Sign up

Export Citation Format

Share Document