Flawed stimulus design in additive-area heuristic studies

Studying magnitude perception using visual item arrays is notoriously difficult due to the intricate relationship between various dimensions including number, area, density, etc. When item arrays are constructed with a skewed and unbalanced distribution of their dimensional properties, false conclusions are easily made. This kind of flawed stimulus design was identified in a series of recently published studies that argue for an additive-area heuristic whereby people are more sensitive to the sum of the vertical and horizontal element axes in each item than the sum of the mathematical area of each item. By analyzing the dimensional properties of the stimuli used in the original studies (e.g., Yousif & Keil, 2019) using the mathematical framework for constructing stimulus parameters (DeWind et al., 2015) and by re-analyzing the data from another previous work on area judgment (Tomlinson et al., 2020), this paper demonstrates how skewed and unbalanced stimulus sampling leads to false conclusions.

If It’s Difficult to Pronounce, It Might Not Be Risky: The Effect of Fluency on Judgment of Risk Does Not Generalize to New Stimuli

Processing fluency is used as a basis for various types of judgment. For example, previous research has shown that people judge food additives with names that are more difficult to pronounce (i.e., that are disfluent) to be more harmful. We explored the possibility that the association between disfluency and perceived harmfulness might be in the opposite direction for some categories of stimuli. Although we found some support for this hypothesis, an improved analysis and further studies indicated that the effect was strongly dependent on the stimuli used. We then used stimulus sampling and showed that the original association between fluency and perceived safety was not replicable with the newly constructed stimuli. We found the association between fluency and perceived safety using the newly constructed stimuli in a final study, but only when pronounceability was confounded with word length. The results cast doubt on generalizability of the association between pronounceability and perceived safety and underscore the importance of treating stimulus as a random factor.

Commentaries and Rejoinder on

While direct replications such as the “Many Labs” project are extremely valuable in testing the reliability of published findings across laboratories, they reflect the common reliance in psychology on single vignettes or stimuli, which limits the scope of the conclusions that can be reached. New experimental tools and statistical techniques make it easier to routinely sample stimuli, and to appropriately treat them as random factors. We encourage researchers to get into the habit of including multiple versions of the content (e.g., stimuli or vignettes) in their designs, to increase confidence in cross-stimulus generalization and to yield more realistic estimates of effect size. We call on editors to be aware of the challenges inherent in such stimulus sampling, to expect and tolerate unexplained variability in observed effect size between stimuli, and to encourage stimulus sampling instead of the deceptively cleaner picture offered by the current reliance on single stimuli.