In the last decade, the use of simple rating and comparison surveys has proliferated on social and digital media platforms to fuel recommendations. These simple surveys and their extrapolation with machine learning algorithms such as matrix factorization shed light on user preferences over large and growing pools of items such as movies, songs, and ads. Social scientists also have a long history of measuring perceptions, preferences, and opinions, typically often over smaller, discrete item sets with exhaustive rating or ranking surveys. This article introduces simple surveys for social science application. We ran experiments to compare the predictive accuracy of both individual and aggregate comparative assessments using four types of simple surveys—pairwise comparisons (PCs) and ratings on 2, 5, and continuous point scales in three contexts—perceived safety of Google Street View images, likability of artwork, and hilarity of animal GIFs. Across contexts, we find that continuous scale ratings best predict individual assessments but consume the most time and cognitive effort. Binary choice surveys are quick and best predict aggregate assessments, useful for collective decision tasks, but poorly predict personalized preferences, for which they are currently used by Netflix to recommend movies. PCs, by contrast, successfully predict personal assessments but poorly predict aggregate assessments despite being widely used to crowdsource ideas and collective preferences. We also demonstrate how findings from these surveys can be visualized in a low-dimensional space to reveal distinct respondent interpretations of questions asked in each context. We conclude by reflecting on differences between sparse, incomplete “simple surveys” and their traditional survey counterparts in terms of efficiency, information elicited, and settings in which knowing less about more may be critical for social science.
Simple Surveys: Response Retrieval Inspired by Recommendation Systems
