Exploring crowd consistency in a mechanical turk survey

ABSTRACT Multiple social science researchers claim that online data collection, mainly via Amazon's Mechanical Turk (MTurk), has revolutionized the behavioral sciences (Gureckis et al. 2016; Litman, Robinson, and Abberbock 2017). While MTurk-based research has grown exponentially in recent years (Chandler and Shapiro 2016), reasonable concerns have been raised about online research participants' ability to proxy for traditional research participants (Chandler, Mueller, and Paolacci 2014). This paper reviews recent MTurk research and provides further guidance for recruiting samples of MTurk participants from populations of interest to behavioral accounting researchers. First, we provide guidance on the logistics of using MTurk and discuss the potential benefits offered by TurkPrime, a third-party service provider. Second, we discuss ways to overcome challenges related to targeted participant recruiting in an online environment. Finally, we offer suggestions for disclosures that authors may provide about their efforts to attract participants and analyze responses.

Download Full-text

Running Behavioral Operations Experiments Using Amazon's Mechanical Turk

SSRN Electronic Journal ◽

10.2139/ssrn.2972406 ◽

2017 ◽

Cited By ~ 1

Author(s):

Yun Lee ◽

Yong Won Seo ◽

Enno Siemsen

Keyword(s):

Mechanical Turk ◽

Behavioral Operations ◽

Amazon's Mechanical Turk

Download Full-text

Development and Validation of the Ableist Microaggression Impact Questionnaire

Rehabilitation Counseling Bulletin ◽

10.1177/00343552211014259 ◽

2021 ◽

pp. 003435522110142

Author(s):

Deniz Aydemir-Döke ◽

James T. Herbert

Keyword(s):

Factor Analysis ◽

Factor Structure ◽

Internal Consistency ◽

Internal Consistency Reliability ◽

Mechanical Turk ◽

Rehabilitation Counselor ◽

Research And Practice ◽

Amazon's Mechanical Turk ◽

Development And Validation ◽

Three Factor

Microaggressions are daily insults to minority individuals such as people with disabilities (PWD) that communicate messages of exclusion, inferiority, and abnormality. In this study, we developed a new scale, the Ableist Microaggressions Impact Questionnaire (AMIQ), which assesses ableist microaggression experiences of PWD. Data from 245 PWD were collected using Amazon’s Mechanical Turk (MTurk) platform. An exploratory factor analysis of the 25-item AMIQ revealed a three-factor structure with internal consistency reliability ranging between .87 and .92. As a more economical and psychometrically sound instrument assessing microaggression impact as it pertains to disability, the AMIQ offers promise for rehabilitation counselor research and practice.

Download Full-text

Evaluating extinction, renewal, and resurgence of operant behavior in humans with Amazon Mechanical Turk

Learning and Motivation ◽

10.1016/j.lmot.2021.101728 ◽

2021 ◽

Vol 74 ◽

pp. 101728

Author(s):

Carolyn M. Ritchey ◽

Toshikazu Kuroda ◽

Jillian M. Rung ◽

Christopher A. Podlesnik

Keyword(s):

Operant Behavior ◽

Mechanical Turk ◽

Amazon Mechanical Turk

Download Full-text

Using infographics to improve trust in science: a randomized pilot test

BMC Research Notes ◽

10.1186/s13104-021-05626-4 ◽

2021 ◽

Vol 14 (1) ◽

Author(s):

Jon Agley ◽

Yunyu Xiao ◽

Esi E. Thompson ◽

Lilian Golzarri-Arroyo

Keyword(s):

Iterative Process ◽

Behavioral Intentions ◽

Research Team ◽

Qualitative Data ◽

Mechanical Turk ◽

Full Study ◽

Amazon's Mechanical Turk ◽

Study Participants ◽

Qualitative Responses ◽

Large Randomized Trial

Abstract Objective This study describes the iterative process of selecting an infographic for use in a large, randomized trial related to trust in science, COVID-19 misinformation, and behavioral intentions for non-pharmaceutical prevenive behaviors. Five separate concepts were developed based on underlying subcomponents of ‘trust in science and scientists’ and were turned into infographics by media experts and digital artists. Study participants (n = 100) were recruited from Amazon’s Mechanical Turk and randomized to five different arms. Each arm viewed a different infographic and provided both quantitative (narrative believability scale and trust in science and scientists inventory) and qualitative data to assist the research team in identifying the infographic most likely to be successful in a larger study. Results Data indicated that all infographics were perceived to be believable, with means ranging from 5.27 to 5.97 on a scale from one to seven. No iatrogenic outcomes were observed for within-group changes in trust in science. Given equivocal believability outcomes, and after examining confidence intervals for data on trust in science and then the qualitative responses, we selected infographic 3, which addressed issues of credibility and consensus by illustrating changing narratives on butter and margarine, as the best candidate for use in the full study.

Download Full-text