Caution! MTurk Workers Ahead—Fines Doubled

2015 ◽  
Vol 8 (2) ◽  
pp. 183-190 ◽  
Author(s):  
P. D. Harms ◽  
Justin A. DeSimone
Keyword(s):  
Mechanical Turk ◽  
Online Data ◽  
Organizational Research ◽  
Target Article ◽  
Long Line ◽  
Potential Problems ◽  
Using Data ◽  
Amazon's Mechanical Turk ◽  
Online Sources ◽  
Better Than

Landers and Behrend (2015) are the most recent in a long line of researchers who have suggested that online samples generated from sources such as Amazon's Mechanical Turk (MTurk) are as good as or potentially even better than the typical samples found in psychology studies. It is important that the authors caution that researchers and reviewers need to carefully reflect on the goals of research when evaluating the appropriateness of samples. However, although they argue that certain types of samples should not be dismissed out of hand, they note that there is only scant evidence demonstrating that online sources can provide usable data for organizational research and that there is a need for further research evaluating the validity of these new sources of data. Because the target article does not directly address the potential problems with such samples, we will review what is known about collecting online data (with a particular focus on MTurk) and illustrate some potential problems using data derived from such sources.

A Technical Guide to Using Amazon's Mechanical Turk in Behavioral Accounting Research

2017 ◽  
Vol 30 (1) ◽  
pp. 111-122 ◽  
Author(s):  
Steve Buchheit ◽  
Marcus M. Doxey ◽  
Troy Pollard ◽  
Shane R. Stinson
Keyword(s):  
Third Party ◽  
Mechanical Turk ◽  
Behavioral Accounting ◽  
Online Data ◽  
Research Participants ◽  
Traditional Research ◽  
Potential Benefits ◽  
Technical Guide ◽  
Online Data Collection ◽  
Amazon's Mechanical Turk

ABSTRACT Multiple social science researchers claim that online data collection, mainly via Amazon's Mechanical Turk (MTurk), has revolutionized the behavioral sciences (Gureckis et al. 2016; Litman, Robinson, and Abberbock 2017). While MTurk-based research has grown exponentially in recent years (Chandler and Shapiro 2016), reasonable concerns have been raised about online research participants' ability to proxy for traditional research participants (Chandler, Mueller, and Paolacci 2014). This paper reviews recent MTurk research and provides further guidance for recruiting samples of MTurk participants from populations of interest to behavioral accounting researchers. First, we provide guidance on the logistics of using MTurk and discuss the potential benefits offered by TurkPrime, a third-party service provider. Second, we discuss ways to overcome challenges related to targeted participant recruiting in an online environment. Finally, we offer suggestions for disclosures that authors may provide about their efforts to attract participants and analyze responses.

Review of Best Practice Recommendations for Ensuring High Quality Data with Amazon’s Mechanical Turk

2020 ◽  
Author(s):  
Brian Bauer ◽  
Kristy L. Larsen ◽  
Nicole Caulfield ◽  
Domynic Elder ◽  
Sara Jordan ◽  
...  
Keyword(s):  
Screening Method ◽  
Scientific Progress ◽  
Worked Examples ◽  
Mechanical Turk ◽  
Quality Data ◽  
Screening Methods ◽  
Online Data ◽  
Amazon's Mechanical Turk ◽  
Data Screening ◽  
Psychological Studies

Our ability to make scientific progress is dependent upon our interpretation of data. Thus, analyzing only those data that are an honest representation of a sample is imperative for drawing accurate conclusions that allow for robust, generalizable, and replicable scientific findings. Unfortunately, a consistent line of evidence indicates the presence of inattentive/careless responders who provide low-quality data in surveys, especially on popular online crowdsourcing platforms such as Amazon’s Mechanical Turk (MTurk). Yet, the majority of psychological studies using surveys only conduct outlier detection analyses to remove problematic data. Without carefully examining the possibility of low-quality data in a sample, researchers risk promoting inaccurate conclusions that interfere with scientific progress. Given that knowledge about data screening methods and optimal online data collection procedures are scattered across disparate disciplines, the dearth of psychological studies using more rigorous methodologies to prevent and detect low-quality data is likely due to inconvenience, not maleficence. Thus, this review provides up-to-date recommendations for best practices in collecting online data and data screening methods. In addition, this article includes resources for worked examples for each screening method, a collection of recommended measures, and a preregistration template for implementing these recommendations.

Inattentive Responding in MTurk and Other Online Samples

2015 ◽  
Vol 8 (2) ◽  
pp. 196-202 ◽  
Author(s):  
Avi Fleischer ◽  
Alan D. Mead ◽  
Jialin Huang
Keyword(s):  
Data Collection ◽  
University Students ◽  
Mechanical Turk ◽  
Amazon's Mechanical Turk ◽  
Online Sources

The focal article by Landers and Behrend (2015) makes the case that samples collected on microtask websites like Amazon's Mechanical Turk (MTurk) are inherently no better or worse than traditional samples of convenience from university students or organizations. We wholeheartedly agree. However, having successfully used MTurk and other online sources for data collection, we feel that the focal article was insufficient regarding the caution required in identifying inattentive respondents and the problems that can arise if such individuals are not removed from the dataset. Although we focus on MTurk, similar issues arise for most “low-stakes” assessments, including student samples, which seem to be increasingly collected online.

Results from the Mechanical Turk Online Audio Recordings

2019 ◽  
pp. 75-112
Author(s):  
James N. Stanford
Keyword(s):  
Data Collection ◽  
New England ◽  
Large Scale ◽  
Statistical Analyses ◽  
Mechanical Turk ◽  
Online Data ◽  
Audio Recordings ◽  
Online Data Collection ◽  
Amazon's Mechanical Turk ◽  
Broad Scale

This is the first of the two chapters (Chapters 4 and 5) that present the results of the online data collection project using Amazon’s Mechanical Turk system. These projects provide a broad-scale “bird’s eye” view of New England dialect features across large distances. This chapter examines the results from 626 speakers who audio-recorded themselves reading 12 sentences two times each. The recordings were analyzed acoustically and then modeled statistically and graphically. The results are presented in the form of maps and statistical analyses, with the goal of providing a large-scale geographic overview of modern-day patterns of New England dialect features.

A Convenient Solution: Using MTurk To Sample From Hard-To-Reach Populations

2015 ◽  
Vol 8 (2) ◽  
pp. 220-228 ◽  
Author(s):  
Nicholas A. Smith ◽  
Isaac E. Sabat ◽  
Larry R. Martinez ◽  
Kayla Weaver ◽  
Shi Xu
Keyword(s):  
United States ◽  
Low Socioeconomic Status ◽  
The United States ◽  
Mechanical Turk ◽  
Special Populations ◽  
Workplace Harassment ◽  
Organizational Research ◽  
Low Socioeconomic ◽  
Hard To Reach Populations ◽  
Amazon's Mechanical Turk

We agree with Landers and Behrend's (2015) proposition that Amazon's Mechanical Turk (MTurk) may provide great opportunities for organizational research samples. However, some groups are characteristically difficult to recruit because they are stigmatized or socially disenfranchised (Birman, 2005; Miller, Forte, Wilson, & Greene, 2006; Sullivan & Cain, 2004; see Campbell, Adams, & Patterson, 2008, for a review). These groups may include individuals who have not previously been the focus of much organizational research, such as those of low socioeconomic status; individuals with disabilities; lesbian, gay, bisexual, or transgender (LGBT) individuals; or victims of workplace harassment. As Landers and Behrend (2015) point out, there is an overrepresentation of research using “Western, educated, industrialized, rich, and democratic” participants. It is important to extend research beyond these samples to examine workplace phenomena that are specific to special populations. We contribute to this argument by noting the particular usefulness that MTurk can provide for sampling from hard-to-reach populations, which we characterize as groups that are in the numerical minority in terms of nationwide representation. To clarify, we focus our discussion on populations that are traditionally hard to reach in the context of contemporary organizational research within the United States.

scholarly journals Workers’ task choice heuristics as a source of emergent structure in digital microwork

2019 ◽  
Author(s):  
Otto Kässi ◽  
Vili Lehdonvirta ◽  
Jean-Michel Dalle
Keyword(s):  
Labor Markets ◽  
Preferential Attachment ◽  
Mechanical Turk ◽  
High Quality ◽  
Online Data ◽  
Employment Contracts ◽  
Task Choice ◽  
Emergent Structure ◽  
Amazon's Mechanical Turk

Digital labor markets are structured around tasks and not around fixed- or long-term employment contracts. We study the consequences of the granularization of work for digital micro workers. To address this question, we combine interview data from active online micro workers and online data on open projects scraped from Amazon's Mechanical Turk platform to study how the digital micro workers choose which tasks they work on. We find evidence for preferential attachment: workers prefer to attach themselves to experienced employers who are known to offer high quality projects. In addition, workers also clearly prefer long series of repeatable tasks over one-off tasks, even when one-off tasks pay considerably more. We thus see a re-emergence of certain types of organizational structure.

Reliability of MTurk Data From Masters and Workers

2020 ◽  
Vol 41 (1) ◽  
pp. 30-36
Author(s):  
Steven V. Rouse
Keyword(s):  
Cognitive Abilities ◽  
Online Survey ◽  
Quality Data ◽  
Standard Samples ◽  
Online Data ◽  
Approval Ratings ◽  
Cognitive Abilities Test ◽  
Elite Status ◽  
Master Status ◽  
Amazon's Mechanical Turk

Abstract. Previous research has supported the use of Amazon’s Mechanical Turk (MTurk) for online data collection in individual differences research. Although MTurk Masters have reached an elite status because of strong approval ratings on previous tasks (and therefore gain higher payment for their work) no research has empirically examined whether researchers actually obtain higher quality data when they require that their MTurk Workers have Master status. In two different online survey studies (one using a personality test and one using a cognitive abilities test), the psychometric reliability of MTurk data was compared between a sample that required a Master qualification type and a sample that placed no status-level qualification requirement. In both studies, the Master samples failed to outperform the standard samples.

PROSES KOMUNIKASI KELOMPOK DALAM METODE PEMBELAJARAN SENTRA UNTUK MEMBENTUK KEMANDIRIAN ANAK

Jurnal Common ◽  
2018 ◽  
Vol 2 (2) ◽  
Author(s):  
Rismawaty Rismawaty ◽  
Sofie Aulia Rahmah
Keyword(s):  
Data Collection ◽  
Qualitative Method ◽  
Group Communication ◽  
Communication Process ◽  
Learning Method ◽  
Online Data ◽  
Analysis Techniques ◽  
Collection Data ◽  
Descriptive Approach ◽  
Using Data

Penelitian ini dilakukan untuk mengetahui proses komunikasi kelompok dalam metode pembelajaran sentra di TK Zaid bin Tsabit. Penelitian ini mendiskusikan tentang proses komunikasi kelompok. Metode penelitian yang digunakan dalam penelitian ini adalah Metode Kualitatif dengan pendekatan Deskriptif. Teknik pengumpulan data yang dilakukan peneliti ada dengan studi pustaka, penelusuran data secara online, wawancara, observasi serta dokumentasi dengan 3 orang informan kunci yaitu guru di TK Zaid bin Tsabit serta 3 informan pendukung yaitu Kepala TK Zaid bin Tsabit dan 2 orang tua murid. Uji keabsahan data dengan peningkatan ketekunan, triangulasi dan diskusi dengan teman sejawat, teknik analisis data menggunakan pengumpulan data, reduksi data, penyajian data, penarikan kesimpulan dan evaluasi.Hasil penelitian ini bahwa Proses komunikasi yang terjadi merupakan komunikasi langsung yang terjadi dua arah dan dilakukan terus menerus untuk membentuk kemandirian anak. Proses komunikasi yang terjadi dalam kelompok metode pembelajaran sentra membentuk kemandirian anak. Proses komunikasi yang dilakukan oleh guru kepada anak dilakukan dengan memberikan arahan-arahan kepada anak serta contoh dari arahan yang telah disampaikan oleh guru.Kesimpulan pada penelitian ini adalah metode pembelajaran sentra membentuk kemandirian anak lewat komunikasi yang dilakukan guru secara terus menerus, karna melalui pembelajaran sentra anak diminta untuk melakukan segala sesuatunya sendiri dalam pengawasan guru. Saran yang diberikan adalah guru harus lebih kreatif dalam memberikan materi pada metode pembelajaran sentra serta bersikap lebih tegas dalam mendidik anak dan melakukan komunikasi yang berkelanjutan dengan orang tua murid. --------------------------------------------------------------------------------- This study was conducted to determine the process of group communication in the center learning method at TK Zaid bin Tsabit. This study discusses the process of group communication. The research method used in this study is a qualitative method with a descriptive approach. The data collection techniques carried out by the researcher were with literature studies, online data searches, interviews, observation and documentation with 3 key informants namely the teacher at TK Zaid bin Tsabit and 3 supporting informants namely TK Head Zaid bin Tsabit and 2 parents. Test the validity of data by increasing perseverance, triangulation and discussion with colleagues, data analysis techniques using data collection, data reduction, data presentation, drawing conclusions and evaluations.The results of this study that the communication process that occurs is direct communication that occurs in two directions and carried out continuously to form the independence of children. The communication process that occurs in a group of central learning methods shapes children's independence. The process of communication carried out by the teacher to the child is done by giving directions to the child as well as examples of directions that have been delivered by the teacher.The conclusion of this study is that the central learning method shapes children's independence through continuous communication by the teacher, because through central learning children are asked to do everything themselves in the supervision of the teacher. The advice given is that the teacher must be more creative in giving material to the central learning method and be more assertive in educating children and making ongoing communication with parents.

Sign in / Sign up

Export Citation Format

Share Document