scholarly journals Less Trust, Moore Verification: Determining the Accuracy of Third-Party Data through an Innovative Use of Attention Checks

2017 ◽  
Author(s):  
Nathan Seltzer
Keyword(s):  
Survey Data ◽  
Selection Criteria ◽  
Research Note ◽  
Third Party ◽  
Mechanical Turk ◽  
Quality Data ◽  
Opinion Poll ◽  
Sampling Frame ◽  
Amazon's Mechanical Turk ◽  
Response Choices

Sociologists increasingly rely on third-party internet panel platforms to acquire respondents and administer questionnaires. Yet, researchers have demonstrated that even samples sourced from well-respected and widely-adopted internet platforms such as Amazon’s Mechanical Turk are often unable to screen out respondents who do not meet selection criteria requested by researchers. Here, I argue that researchers should proactively verify that third-party survey data is accurately sampled before considering it for analysis. I propose using survey “attention checks” as a methodological solution for researchers to determine whether data vendors have provided low quality data. In this short research note, I illustrate the approach by analyzing data from a consequential political opinion poll administered on behalf of an academic polling center by a third-party internet panel vendor for a special election in 2017. By assessing valid/invalid response choices of two overlapping geographic variables, I identify irregularities in the dataset that suggest that the sample included respondents who were not within the researchers’ intended sampling frame. Attention checks provide a straightforward, inexpensive tool to improve the validity of research produced with internet-drawn samples.

A Technical Guide to Using Amazon's Mechanical Turk in Behavioral Accounting Research

2017 ◽  
Vol 30 (1) ◽  
pp. 111-122 ◽  
Author(s):  
Steve Buchheit ◽  
Marcus M. Doxey ◽  
Troy Pollard ◽  
Shane R. Stinson
Keyword(s):  
Third Party ◽  
Mechanical Turk ◽  
Behavioral Accounting ◽  
Online Data ◽  
Research Participants ◽  
Traditional Research ◽  
Potential Benefits ◽  
Technical Guide ◽  
Online Data Collection ◽  
Amazon's Mechanical Turk

ABSTRACT Multiple social science researchers claim that online data collection, mainly via Amazon's Mechanical Turk (MTurk), has revolutionized the behavioral sciences (Gureckis et al. 2016; Litman, Robinson, and Abberbock 2017). While MTurk-based research has grown exponentially in recent years (Chandler and Shapiro 2016), reasonable concerns have been raised about online research participants' ability to proxy for traditional research participants (Chandler, Mueller, and Paolacci 2014). This paper reviews recent MTurk research and provides further guidance for recruiting samples of MTurk participants from populations of interest to behavioral accounting researchers. First, we provide guidance on the logistics of using MTurk and discuss the potential benefits offered by TurkPrime, a third-party service provider. Second, we discuss ways to overcome challenges related to targeted participant recruiting in an online environment. Finally, we offer suggestions for disclosures that authors may provide about their efforts to attract participants and analyze responses.

scholarly journals Studying Gambling Behaviors and Responsible Gambling Tools in a Simulated Online Casino Integrated With Amazon Mechanical Turk: Development and Initial Validation of Survey Data and Platform Mechanics of the Frescati Online Research Casino

2021 ◽  
Vol 11 ◽  
Author(s):  
Philip Lindner ◽  
Jonas Ramnerö ◽  
Ekaterina Ivanova ◽  
Per Carlbring
Keyword(s):  
Survey Data ◽  
Real Life ◽  
Responsible Gambling ◽  
Mechanical Turk ◽  
Behavioral Data ◽  
Online Gambling ◽  
Amazon Mechanical Turk ◽  
Quality Data ◽  
Online Research ◽  
Game Mechanics

Introduction: Online gambling, popular among both problem and recreational gamblers, simultaneously entails both heightened addiction risks as well as unique opportunities for prevention and intervention. There is a need to bridge the growing literature on learning and extinction mechanisms of gambling behavior, with account tracking studies using real-life gambling data. In this study, we describe the development and validation of the Frescati Online Research Casino (FORC): a simulated online casino where games, visual themes, outcome sizes, probabilities, and other variables of interest can be experimentally manipulated to conduct behavioral analytic studies and evaluate the efficacy of responsible gambling tools.Methods: FORC features an initial survey for self-reporting of gambling and gambling problems, along with several games resembling regular real-life casino games, designed to allow Pavlovian and instrumental learning. FORC was developed with maximum flexibility in mind, allowing detailed experiment specification by setting parameters using an online interface, including the display of messages. To allow convenient and rapid data collection from diverse samples, FORC is independently hosted yet integrated with the popular crowdsourcing platform Amazon Mechanical Turk through a reimbursement key mechanism. To validate the survey data quality and game mechanics of FORC, n = 101 participants were recruited, who answered an questionnaire on gambling habits and problems, then played both slot machine and card-draw type games. Questionnaire and trial-by-trial behavioral data were analyzed using standard psychometric tests, and outcome distribution modeling.Results: The expected associations among variables in the introductory questionnaire were found along with good psychometric properties, suggestive of good quality data. Only 6% of participants provided seemingly poor behavioral data. Game mechanics worked as intended: gambling outcomes showed the expected pattern of random sampling with replacement and were normally distributed around the set percentages, while balances developed according to the set return to player rate.Conclusions: FORC appears to be a valid paradigm for simulating online gambling and for collecting survey and behavioral data, offering a valuable compromise between stringent experimental paradigms with lower external validity, and real-world gambling account tracking data with lower internal validity.

Review of Best Practice Recommendations for Ensuring High Quality Data with Amazon’s Mechanical Turk

2020 ◽  
Author(s):  
Brian Bauer ◽  
Kristy L. Larsen ◽  
Nicole Caulfield ◽  
Domynic Elder ◽  
Sara Jordan ◽  
...  
Keyword(s):  
Screening Method ◽  
Scientific Progress ◽  
Worked Examples ◽  
Mechanical Turk ◽  
Quality Data ◽  
Screening Methods ◽  
Online Data ◽  
Amazon's Mechanical Turk ◽  
Data Screening ◽  
Psychological Studies

Our ability to make scientific progress is dependent upon our interpretation of data. Thus, analyzing only those data that are an honest representation of a sample is imperative for drawing accurate conclusions that allow for robust, generalizable, and replicable scientific findings. Unfortunately, a consistent line of evidence indicates the presence of inattentive/careless responders who provide low-quality data in surveys, especially on popular online crowdsourcing platforms such as Amazon’s Mechanical Turk (MTurk). Yet, the majority of psychological studies using surveys only conduct outlier detection analyses to remove problematic data. Without carefully examining the possibility of low-quality data in a sample, researchers risk promoting inaccurate conclusions that interfere with scientific progress. Given that knowledge about data screening methods and optimal online data collection procedures are scattered across disparate disciplines, the dearth of psychological studies using more rigorous methodologies to prevent and detect low-quality data is likely due to inconvenience, not maleficence. Thus, this review provides up-to-date recommendations for best practices in collecting online data and data screening methods. In addition, this article includes resources for worked examples for each screening method, a collection of recommended measures, and a preregistration template for implementing these recommendations.

scholarly journals Autism and online recruiting methods: A comparison of Mechanical Turk and discussion forums

First Monday ◽  
2019 ◽  
Author(s):  
Amelia Anderson ◽  
Nancy Everhart ◽  
Juliann Woods
Keyword(s):  
Online Survey ◽  
Autism Spectrum ◽  
Mechanical Turk ◽  
Discussion Forums ◽  
Success Rates ◽  
Sampling Frame ◽  
Online Discussion Forums ◽  
Information And Communication ◽  
Stakeholder Input ◽  
Amazon's Mechanical Turk

In a study by a team at the intersection of information and communication sciences and disorders, researchers worked to design an interactive, online professional development system for academic librarians to better serve students with autism spectrum disorder (ASD). In creating this program, it was imperative to have stakeholder input and support; recruiting members of this population, students with ASD, was critical. Amazon’s Mechanical Turk and online discussion forums, including Reddit, were used for recruitment for an online survey. While there was some overlap in results, there were also marked differences in responses based on online sampling frame. This paper details the online methods used for recruiting members of this community, and compares and contrasts success rates, challenges, and numbers associated with each method.

scholarly journals The shape of and solutions to the MTurk quality crisis

2020 ◽  
Vol 8 (4) ◽  
pp. 614-629 ◽  
Author(s):  
Ryan Kennedy ◽  
Scott Clifford ◽  
Tyler Burleigh ◽  
Philip D. Waggoner ◽  
Ryan Jewell ◽  
...  
Keyword(s):  
Data Collection ◽  
Data Quality ◽  
Treatment Effects ◽  
Mechanical Turk ◽  
Quality Data ◽  
Social Scientists ◽  
Amazon's Mechanical Turk ◽  
Gain Access

AbstractAmazon's Mechanical Turk is widely used for data collection; however, data quality may be declining due to the use of virtual private servers to fraudulently gain access to studies. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. We first analyze 38 studies and show that this fraud is not new, but has increased recently. We then show that these fraudulent respondents provide particularly low-quality data and can weaken treatment effects. Finally, we provide two solutions: an easy-to-use application for identifying fraud in the existing datasets and a method for blocking fraudulent respondents in Qualtrics surveys.

Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup

2021 ◽  
pp. 193896552110254
Author(s):  
Lu Lu ◽  
Nathan Neale ◽  
Nathaniel D. Line ◽  
Mark Bonn
Keyword(s):  
Best Practices ◽  
Data Quality ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Quality Data ◽  
High Quality ◽  
High Quality Data ◽  
Amazon's Mechanical Turk ◽  
Associated Data

As the use of Amazon’s Mechanical Turk (MTurk) has increased among social science researchers, so, too, has research into the merits and drawbacks of the platform. However, while many endeavors have sought to address issues such as generalizability, the attentiveness of workers, and the quality of the associated data, there has been relatively less effort concentrated on integrating the various strategies that can be used to generate high-quality data using MTurk samples. Accordingly, the purpose of this research is twofold. First, existing studies are integrated into a set of strategies/best practices that can be used to maximize MTurk data quality. Second, focusing on task setup, selected platform-level strategies that have received relatively less attention in previous research are empirically tested to further enhance the contribution of the proposed best practices for MTurk usage.

scholarly journals Evaluating CloudResearch’s Approved Group as a Solution for Problematic Data Quality on MTurk

2021 ◽  
Author(s):  
David Hauser ◽  
Aaron J Moss ◽  
Cheskie Rosenzweig ◽  
Shalom Noach Jaffe ◽  
Jonathan Robinson ◽  
...  
Keyword(s):  
Reading Comprehension ◽  
Data Quality ◽  
Predictive Validity ◽  
Standard Sample ◽  
Quality Measures ◽  
Third Party ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Quality Data

Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. CloudResearch, a third-party website that interfaces with MTurk, assessed ~100,000 MTurkers and categorized them into those that provide high- (~65,000, Approved) and low-(~35,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch’s vetting. Participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample, completed an array of data quality measures. Approved participants had better reading comprehension, reliability, honesty, and attentiveness scores, were less likely to cheat and satisfice, and replicated classic experimental effects more reliably than Blocked participants who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss the implications of using the Approved group for scientific studies conducted on Mechanical Turk.

An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use

2018 ◽  
Vol 13 (2) ◽  
pp. 149-154 ◽  
Author(s):  
Michael D. Buhrmester ◽  
Sanaz Talaifar ◽  
Samuel D. Gosling
Keyword(s):  
Mechanical Turk ◽  
Quality Data ◽  
Online Research ◽  
Social Scientists ◽  
High Quality Data ◽  
The Social ◽  
Research Platform ◽  
Amazon's Mechanical Turk ◽  
Effective Use ◽  
The Impact

Over the past 2 decades, many social scientists have expanded their data-collection capabilities by using various online research tools. In the 2011 article “Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?” in Perspectives on Psychological Science, Buhrmester, Kwang, and Gosling introduced researchers to what was then considered to be a promising but nascent research platform. Since then, thousands of social scientists from seemingly every field have conducted research using the platform. Here, we reflect on the impact of Mechanical Turk on the social sciences and our article’s role in its rise, provide the newest data-driven recommendations to help researchers effectively use the platform, and highlight other online research platforms worth consideration.

Sign in / Sign up

Export Citation Format

Share Document