Different Order, Different Results? The Effects of Dimension Order in Factorial Survey Experiments

Factorial surveys are widely used in the social sciences to measure respondents’ attitudes, beliefs, or behavioral intentions. In such surveys, respondents evaluate short descriptions of hypothetical situations, persons, or objects that vary across several dimensions. An important prerequisite of the method’s validity is that respondents are able to deal with the highly complex task created by the need to consider several variable dimensions within one coherent judgment. We analyze the effects of the order in which dimensions are presented in running text vignettes. An experimental setup with four order treatments was randomly allocated to 787 respondents (based on a random sample of register data), yielding 3,119 vignette evaluations. The analyses compare respondent groups across age, education, and response speed. Overall, there is no strong evidence for order effects. However, we find a slight tendency for fast responders to be more prone to recency effects.

Download Full-text

What Can We Learn From Factorial Surveys About Human Behavior?

Methodology ◽

10.1027/1614-2241/a000161 ◽

2019 ◽

Vol 15 (1) ◽

pp. 19-30 ◽

Cited By ~ 10

Author(s):

Knut Petzold ◽

Tobias Wolbring

Keyword(s):

Social Sciences ◽

Behavioral Intentions ◽

Actual Behavior ◽

Full Time ◽

Factorial Survey ◽

Survey Experiment ◽

Intended Behavior ◽

The Social ◽

Survey Experiments ◽

High Level

Abstract. Factorial survey experiments are increasingly used in the social sciences to investigate behavioral intentions. The measurement of self-reported behavioral intentions with factorial survey experiments frequently assumes that the determinants of intended behavior affect actual behavior in a similar way. We critically investigate this fundamental assumption using the misdirected email technique. Student participants of a survey were randomly assigned to a field experiment or a survey experiment. The email informs the recipient about the reception of a scholarship with varying stakes (full-time vs. book) and recipient’s names (German vs. Arabic). In the survey experiment, respondents saw an image of the same email. This validation design ensured a high level of correspondence between units, settings, and treatments across both studies. Results reveal that while the frequencies of self-reported intentions and actual behavior deviate, treatments show similar relative effects. Hence, although further research on this topic is needed, this study suggests that determinants of behavior might be inferred from behavioral intentions measured with survey experiments.

Download Full-text