An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use

2018 ◽  
Vol 13 (2) ◽  
pp. 149-154 ◽  
Author(s):  
Michael D. Buhrmester ◽  
Sanaz Talaifar ◽  
Samuel D. Gosling
Keyword(s):  
Mechanical Turk ◽  
Quality Data ◽  
Online Research ◽  
Social Scientists ◽  
High Quality Data ◽  
The Social ◽  
Research Platform ◽  
Amazon's Mechanical Turk ◽  
Effective Use ◽  
The Impact

Over the past 2 decades, many social scientists have expanded their data-collection capabilities by using various online research tools. In the 2011 article “Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?” in Perspectives on Psychological Science, Buhrmester, Kwang, and Gosling introduced researchers to what was then considered to be a promising but nascent research platform. Since then, thousands of social scientists from seemingly every field have conducted research using the platform. Here, we reflect on the impact of Mechanical Turk on the social sciences and our article’s role in its rise, provide the newest data-driven recommendations to help researchers effectively use the platform, and highlight other online research platforms worth consideration.

scholarly journals The shape of and solutions to the MTurk quality crisis

2020 ◽  
Vol 8 (4) ◽  
pp. 614-629 ◽  
Author(s):  
Ryan Kennedy ◽  
Scott Clifford ◽  
Tyler Burleigh ◽  
Philip D. Waggoner ◽  
Ryan Jewell ◽  
...  
Keyword(s):  
Data Collection ◽  
Data Quality ◽  
Treatment Effects ◽  
Mechanical Turk ◽  
Quality Data ◽  
Social Scientists ◽  
Amazon's Mechanical Turk ◽  
Gain Access

AbstractAmazon's Mechanical Turk is widely used for data collection; however, data quality may be declining due to the use of virtual private servers to fraudulently gain access to studies. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. We first analyze 38 studies and show that this fraud is not new, but has increased recently. We then show that these fraudulent respondents provide particularly low-quality data and can weaken treatment effects. Finally, we provide two solutions: an easy-to-use application for identifying fraud in the existing datasets and a method for blocking fraudulent respondents in Qualtrics surveys.

Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup

2021 ◽  
pp. 193896552110254
Author(s):  
Lu Lu ◽  
Nathan Neale ◽  
Nathaniel D. Line ◽  
Mark Bonn
Keyword(s):  
Best Practices ◽  
Data Quality ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Quality Data ◽  
High Quality ◽  
High Quality Data ◽  
Amazon's Mechanical Turk ◽  
Associated Data

As the use of Amazon’s Mechanical Turk (MTurk) has increased among social science researchers, so, too, has research into the merits and drawbacks of the platform. However, while many endeavors have sought to address issues such as generalizability, the attentiveness of workers, and the quality of the associated data, there has been relatively less effort concentrated on integrating the various strategies that can be used to generate high-quality data using MTurk samples. Accordingly, the purpose of this research is twofold. First, existing studies are integrated into a set of strategies/best practices that can be used to maximize MTurk data quality. Second, focusing on task setup, selected platform-level strategies that have received relatively less attention in previous research are empirically tested to further enhance the contribution of the proposed best practices for MTurk usage.

Twin Studies and Politics

2019 ◽  
Author(s):  
Levente Littvay
Keyword(s):  
Political Science ◽  
Behavioral Genetics ◽  
Twin Studies ◽  
Social Scientists ◽  
Cross Sectional ◽  
Genetic Transmission ◽  
The Social ◽  
Direction Of Causation ◽  
Social Part ◽  
The Impact

As recently as 2005, John Alford and colleagues surprised political science with their twin study that found empirical evidence of the genetic transmission of political attitudes and behaviors. Reactions in the field were mixed, but one thing is for sure: it is not time to mourn the social part of the social sciences. Genetics is not the deterministic mechanism that social scientists often assume it to be. No specific part of DNA is responsible for anything but minute, indirect effects on political orientations. Genes express themselves differently in different contexts, suggesting that the political phenomenon behavioral political scientists take for granted may be quite volatile; hence, the impact of genetics is also much less stable in its foundations than initially assumed. Twin studies can offer a unique and powerful avenue to study these behavioral processes as they are more powerful than cross-sectional (or even longitudinal) studies not only for understanding heritability but also for asserting the direction of causation, the social (and, of course, genetic) pathways that explain how political phenomena are related to each other. This chapter aims to take the reader through this journey that political science has gone through over the past decade and a half and point to the synergies behavioral political science and behavioral genetics offer to the advancement of the discipline.

American Labor Unions in the Electoral Arena By Herbert B. Asher, Eric S. Heberlig, Randall B. Ripley, and Karen Snyder. Lanham, MD: Rowman & Littlefield, 2001. 207p. $69.00 cloth, $19.95 paper.

2002 ◽  
Vol 96 (3) ◽  
pp. 630-630
Author(s):  
Glenn Perusek
Keyword(s):  
The United States ◽  
Organized Labor ◽  
Social Scientists ◽  
Race And Gender ◽  
Union Movement ◽  
The Social ◽  
And Gender ◽  
The 1960S ◽  
The Impact ◽  
American Labor

For more than a generation, as the authors rightly point out, the impact of organized labor on electoral politics has been neglected in scholarly literature. Indeed, only a tiny minority of social scientists explicitly focuses on organized labor in the United States. Although the impact of the social movements of the 1960s appeared to heighten awareness of the importance of class, race, and gender, class and its organized expression, the union movement, has received less attention, while studies of race and gender have flourished.

scholarly journals The Number of Choice Tasks and Survey Satisficing in Conjoint Experiments

2018 ◽  
Vol 26 (1) ◽  
pp. 112-119 ◽  
Author(s):  
Kirk Bansak ◽  
Jens Hainmueller ◽  
Daniel J. Hopkins ◽  
Teppei Yamamoto
Keyword(s):  
Decision Making ◽  
Survey Sampling ◽  
Mechanical Turk ◽  
Social Scientists ◽  
Response Quality ◽  
Amazon's Mechanical Turk ◽  
Survey Designs ◽  
Choice Tasks

In recent years, political and social scientists have made increasing use of conjoint survey designs to study decision-making. Here, we study a consequential question which researchers confront when implementing conjoint designs: How many choice tasks can respondents perform before survey satisficing degrades response quality? To answer the question, we run a set of experiments where respondents are asked to complete as many as 30 conjoint tasks. Experiments conducted through Amazon’s Mechanical Turk and Survey Sampling International demonstrate the surprising robustness of conjoint designs, as there are detectable but quite limited increases in survey satisficing as the number of tasks increases. Our evidence suggests that in similar study contexts researchers can assign dozens of tasks without substantial declines in response quality.

scholarly journals Studying Gambling Behaviors and Responsible Gambling Tools in a Simulated Online Casino Integrated With Amazon Mechanical Turk: Development and Initial Validation of Survey Data and Platform Mechanics of the Frescati Online Research Casino

2021 ◽  
Vol 11 ◽  
Author(s):  
Philip Lindner ◽  
Jonas Ramnerö ◽  
Ekaterina Ivanova ◽  
Per Carlbring
Keyword(s):  
Survey Data ◽  
Real Life ◽  
Responsible Gambling ◽  
Mechanical Turk ◽  
Behavioral Data ◽  
Online Gambling ◽  
Amazon Mechanical Turk ◽  
Quality Data ◽  
Online Research ◽  
Game Mechanics

Introduction: Online gambling, popular among both problem and recreational gamblers, simultaneously entails both heightened addiction risks as well as unique opportunities for prevention and intervention. There is a need to bridge the growing literature on learning and extinction mechanisms of gambling behavior, with account tracking studies using real-life gambling data. In this study, we describe the development and validation of the Frescati Online Research Casino (FORC): a simulated online casino where games, visual themes, outcome sizes, probabilities, and other variables of interest can be experimentally manipulated to conduct behavioral analytic studies and evaluate the efficacy of responsible gambling tools.Methods: FORC features an initial survey for self-reporting of gambling and gambling problems, along with several games resembling regular real-life casino games, designed to allow Pavlovian and instrumental learning. FORC was developed with maximum flexibility in mind, allowing detailed experiment specification by setting parameters using an online interface, including the display of messages. To allow convenient and rapid data collection from diverse samples, FORC is independently hosted yet integrated with the popular crowdsourcing platform Amazon Mechanical Turk through a reimbursement key mechanism. To validate the survey data quality and game mechanics of FORC, n = 101 participants were recruited, who answered an questionnaire on gambling habits and problems, then played both slot machine and card-draw type games. Questionnaire and trial-by-trial behavioral data were analyzed using standard psychometric tests, and outcome distribution modeling.Results: The expected associations among variables in the introductory questionnaire were found along with good psychometric properties, suggestive of good quality data. Only 6% of participants provided seemingly poor behavioral data. Game mechanics worked as intended: gambling outcomes showed the expected pattern of random sampling with replacement and were normally distributed around the set percentages, while balances developed according to the set return to player rate.Conclusions: FORC appears to be a valid paradigm for simulating online gambling and for collecting survey and behavioral data, offering a valuable compromise between stringent experimental paradigms with lower external validity, and real-world gambling account tracking data with lower internal validity.

Sign in / Sign up

Export Citation Format

Share Document