Improving the Statistical Power and Reliability of Research Using Amazon Mechanical Turk

2021 ◽  
Author(s):  
Jeremiah W Bentley
Keyword(s):  
Quality Control ◽  
Statistical Power ◽  
Low Cost ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Online Platforms

Amazon Mechanical Turk (MTurk) is an increasingly popular source of experimental participants due to its convenience and low cost (relative to traditional laboratories). However, MTurk presents challenges related to statistical power and reliability. These challenges are not unique to MTurk, but are more prevalent than in research conducted with other participant pools. In this paper I discuss several reasons why research conducted with MTurk may face additional power and reliability challenges. I then present suggestions for dealing with these challenges, taking advantage of the comparative strengths of MTurk. The discussion should be of interest to PhD students and other researchers considering using MTurk or other online platforms as a source of experimental participants as well as to reviewers and editors who are considering quality control standards for research conducted with this participant pool.

Efficient Crowdsourcing via Bayes Quality Control

2014 ◽  
Vol 998-999 ◽  
pp. 1576-1580
Author(s):  
Hao Su ◽  
Shan Liao
Keyword(s):  
Quality Control ◽  
Posterior Distribution ◽  
Image Annotation ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Annotation Task ◽  
Fine Grained ◽  
Internet Age ◽  
Net Framework ◽  
Annotation Quality

Crowdsourcing has become an important tool to aggregate the wisdom of the crowd in this Internet age. A central problem in building an online crowdsourcing system is to determine the appropriate number of workers to assign tasks to. We study this problem by formulating a Bayes Net framework and introduce a quantity derived from posterior distribution to measure the convergence of crowd opinions. Using this quantity, our algorithm could stop soliciting opinions from more workers if the distribution of opinions is unlikely to change in future predictions. We empirically demonstrate the effectiveness of the designed strategy by building a challenging fine-grained image annotation task on Amazon Mechanical Turk. Experiment results show that our approach not only saves annotation cost but also guarantees high annotation quality.

scholarly journals Thermogravimetric Analysis (TGA) of Graphene Materials: Effect of Particle Size of Graphene, Graphene Oxide and Graphite on Thermal Parameters

2021 ◽  
Vol 7 (2) ◽  
pp. 41
Author(s):  
Farzaneh Farivar ◽  
Pei Lay Yap ◽  
Ramesh Udayashankar Karunagaran ◽  
Dusan Losic
Keyword(s):  
Particle Size ◽  
Quality Control ◽  
Graphene Oxide ◽  
Thermogravimetric Analysis ◽  
Low Cost ◽  
Thermal Parameters ◽  
Carbon Decomposition ◽  
Size Number ◽  
Carbon Combustion ◽  
Oxygen Groups

Thermogravimetric analysis (TGA) has been recognized as a simple and reliable analytical tool for characterization of industrially manufactured graphene powders. Thermal properties of graphene are dependent on many parameters such as particle size, number of layers, defects and presence of oxygen groups to improve the reliability of this method for quality control of graphene materials, therefore it is important to explore the influence of these parameters. This paper presents a comprehensive TGA study to determine the influence of different particle size of the three key materials including graphene, graphene oxide and graphite on their thermal parameters such as carbon decomposition range and its temperature of maximum mass change rate (Tmax). Results showed that Tmax values derived from the TGA-DTG carbon combustion peaks of these materials increasing from GO (558–616 °C), to graphene (659–713 °C) and followed by graphite (841–949 °C) The Tmax values derived from their respective DTG carbon combustion peaks increased as their particle size increased (28.6–120.2 µm for GO, 7.6–73.4 for graphene and 24.2–148.8 µm for graphite). The linear relationship between the Tmax values and the particle size of graphene and their key impurities (graphite and GO) confirmed in this study endows the use of TGA technique with more confidence to evaluate bulk graphene-related materials (GRMs) at low-cost, rapid, reliable and simple diagnostic tool for improved quality control of industrially manufactured GRMs including detection of “fake” graphene.

scholarly journals Evaluating extinction, renewal, and resurgence of operant behavior in humans with Amazon Mechanical Turk

2021 ◽  
Vol 74 ◽  
pp. 101728
Author(s):  
Carolyn M. Ritchey ◽  
Toshikazu Kuroda ◽  
Jillian M. Rung ◽  
Christopher A. Podlesnik
Keyword(s):  
Operant Behavior ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk

scholarly journals Amazon Mechanical Turk: Gold Mine or Coal Mine?

2011 ◽  
Vol 37 (2) ◽  
pp. 413-420 ◽  
Author(s):  
Karën Fort ◽  
Gilles Adda ◽  
K. Bretonnel Cohen
Keyword(s):  
Coal Mine ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Gold Mine

scholarly journals Developing and validating an isotrigon texture discrimination task using Amazon Mechanical Turk

2015 ◽  
Vol 16 (S1) ◽  
Author(s):  
John WG Seamons ◽  
Marconi S Barbosa ◽  
Jonathan D Victor ◽  
Dominique Coy ◽  
Ted Maddess
Keyword(s):  
Discrimination Task ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Texture Discrimination

Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

2011 ◽  
Author(s):  
F. Jurčíček ◽  
S. Keizer ◽  
Milica Gašić ◽  
François Mairesse ◽  
B. Thomson ◽  
...  
Keyword(s):  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
User Evaluation ◽  
Dialogue Systems ◽  
Spoken Dialogue Systems ◽  
Spoken Dialogue

Exploring Biases Between Human and Machine Generated Designs

2018 ◽  
Vol 141 (2) ◽  
Author(s):  
Christian E. Lopez ◽  
Scarlett R. Miller ◽  
Conrad S. Tucker
Keyword(s):  
Computer Simulation ◽  
Deep Learning ◽  
Domain Knowledge ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Design Tools ◽  
Generative Design ◽  
Generate Functional ◽  
Intended Function ◽  
2D Sketches

The objective of this work is to explore the possible biases that individuals may have toward the perceived functionality of machine generated designs, compared to human created designs. Toward this end, 1187 participants were recruited via Amazon mechanical Turk (AMT) to analyze the perceived functional characteristics of both human created two-dimensional (2D) sketches and sketches generated by a deep learning generative model. In addition, a computer simulation was used to test the capability of the sketched ideas to perform their intended function and explore the validity of participants' responses. The results reveal that both participants and computer simulation evaluations were in agreement, indicating that sketches generated via the deep generative design model were more likely to perform their intended function, compared to human created sketches used to train the model. The results also reveal that participants were subject to biases while evaluating the sketches, and their age and domain knowledge were positively correlated with their perceived functionality of sketches. The results provide evidence that supports the capabilities of deep learning generative design tools to generate functional ideas and their potential to assist designers in creative tasks such as ideation.

Sign in / Sign up

Export Citation Format

Share Document