Following vital discussion around the replicability of published findings, researchers demanded increased efforts to improve research practices in empirical social science. Consequentially, journals publishing consumer research implemented new measures to increase the replicability of published work. Nonetheless, no systematic empirical analysis on a large sample has investigated whether published consumer research has changed along with the discussion. To address this need, we surveyed three indicators for the replicability of published consumer research over time. We used text mining to quantify sample sizes, effect sizes, and the distribution of published p-values from a sample of N = 923 articles published between 2011 and 2018 in the Journal of Marketing Research, the Journal of Consumer Psychology, and the Journal of Consumer Research. To test the developments over time, we focused on a subsample of hand-coded articles and identified central hypothesis tests herein. Results show a trend toward increased sample sizes and decreased effect sizes across all three journals in the subset as well as the entire set of articles.
Web Social Science: Concepts, Data and Tools for Social Scientists in the Digital Age
