scholarly journals CurmElo: The theory and practice of a forced-choice approach to producing preference rankings

PLoS ONE ◽  
2021 ◽  
Vol 16 (5) ◽  
pp. e0252145
Author(s):  
Soham Sankaran ◽  
Jacob Derechin ◽  
Nicholas A. Christakis
Keyword(s):  
Social Science ◽  
English Language ◽  
Induced Polarization ◽  
Forced Choice ◽  
Theory And Practice ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Preference Ranking ◽  
Preference Heterogeneity ◽  
Choice Approach

We introduce CurmElo, a forced-choice approach to producing a preference ranking of an arbitrary set of objects that combines the Elo algorithm with novel techniques for detecting and correcting for (1) preference heterogeneity induced polarization in preferences among raters, and (2) intransitivity in preference rankings. We detail the application of CurmElo to the problem of generating approximately preference-neutral identifiers, in this case four-letter and five-letter nonsense words patterned on the phonological conventions of the English language, using a population of Amazon Mechanical Turk workers. We find evidence that human raters have significant non-uniform preferences over these nonsense words, and we detail the consequences of this finding for social science work that utilizes identifiers without accounting for the bias this can induce. In addition, we describe how CurmElo can be used to produce rankings of arbitrary features or dimensions of preference of a set of objects relative to a population of raters.

scholarly journals CurmElo: The theory and practice of a forced-choice approach to producing preference rankings

2019 ◽  
Author(s):  
Soham Sankaran ◽  
Jacob Derechin ◽  
Nicholas Christakis
Keyword(s):  
Social Science ◽  
English Language ◽  
Induced Polarization ◽  
Forced Choice ◽  
Theory And Practice ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Preference Ranking ◽  
Preference Heterogeneity ◽  
Choice Approach

We introduce CurmElo, a forced-choice approach to producing a preference ranking of an arbitrary set of objects that combines the Elo algorithm with novel techniques for detecting and correcting for (1) preference heterogeneity induced polarization in preferences among raters, and (2) intransitivity in preference rankings. We detail the application of CurmElo to the problem of generating approximately preference-neutral identifiers, in this case four-letter and five-letter nonsense words patterned on the phonological conventions of the English language, using a population of Amazon Mechanical Turk workers. We find evidence that human raters have significant non-uniform preferences over these nonsense words, and we detail the consequences of this finding for social science work that utilizes identifiers without accounting for this effect. In addition, we describe how CurmElo can be used to produce rankings of arbitrary features or dimensions of preference of a set of objects relative to a population of raters.

scholarly journals Evaluating extinction, renewal, and resurgence of operant behavior in humans with Amazon Mechanical Turk

2021 ◽  
Vol 74 ◽  
pp. 101728
Author(s):  
Carolyn M. Ritchey ◽  
Toshikazu Kuroda ◽  
Jillian M. Rung ◽  
Christopher A. Podlesnik
Keyword(s):  
Operant Behavior ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk

scholarly journals Amazon Mechanical Turk: Gold Mine or Coal Mine?

2011 ◽  
Vol 37 (2) ◽  
pp. 413-420 ◽  
Author(s):  
Karën Fort ◽  
Gilles Adda ◽  
K. Bretonnel Cohen
Keyword(s):  
Coal Mine ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Gold Mine

scholarly journals Developing and validating an isotrigon texture discrimination task using Amazon Mechanical Turk

2015 ◽  
Vol 16 (S1) ◽  
Author(s):  
John WG Seamons ◽  
Marconi S Barbosa ◽  
Jonathan D Victor ◽  
Dominique Coy ◽  
Ted Maddess
Keyword(s):  
Discrimination Task ◽  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
Texture Discrimination

Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

2011 ◽  
Author(s):  
F. Jurčíček ◽  
S. Keizer ◽  
Milica Gašić ◽  
François Mairesse ◽  
B. Thomson ◽  
...  
Keyword(s):  
Mechanical Turk ◽  
Amazon Mechanical Turk ◽  
User Evaluation ◽  
Dialogue Systems ◽  
Spoken Dialogue Systems ◽  
Spoken Dialogue

Information seeking behaviour ofArab social science and humanities postgraduates: Adescriptive study

2021 ◽  
pp. 116-137
Author(s):  
Ahmed Maher Khafaga Shehata ◽  
Amr Hassan Fatouh Hassan
Keyword(s):  
Social Science ◽  
Search Engines ◽  
Information Seeking ◽  
English Language ◽  
Arabic Language ◽  
Information Seeking Behavior ◽  
Sources Of Information ◽  
Postgraduate Students ◽  
Social Sciences And Humanities ◽  
Seeking Behavior

The purpose of this paper is to report the findings of a study of information-seeking behavior among a group of Arab postgraduate students in social science and humanities disciplines. The paper also explores information-seeking styles and examines how information seeking is affected by external factors. The study employed a qualitative approach to explore informationseeking behavior in the sample and the sources of information used to obtain scholarly information. A sample of 33 participants was interviewed to elucidate the information-seeking behavior of the Arabic language speakers. The analysis of the interviews revealed that the participants use different methods to find information on the internet. These methods vary from using search engines to using sites that provide pirated scholarly papers. The data showed that most of the sample students use search engines and databases provided by their universities, but they should be trained in research ethics to avoid unacceptable research practices. The results also indicate that searching in other languages represents a challenge for Arab postgraduates in the social sciences and humanities. This study was conducted with social science and humanities postgraduates as part of a series of studies aiming to explore Arab language speakers' scholarly practices. The information-seeking behavior of science disciplines may differ, as the teaching language is mainly in English. This study contributes to the field by expanding our understanding of how non-English language speakers seek scholarly information and what sources are used to obtain the scholarly papers.

Sign in / Sign up

Export Citation Format

Share Document