A fundamental psychological problem is identifying the idiosyncratic and shared contributions to stimulus evaluation. However, there is no established method for estimating these contributions and the existing methods have led to divergent estimates. Moreover, in many studies participants rate the stimuli only once, although at least two measurements are required to estimate idiosyncratic contributions. Here, participants rated faces or novel objects on four dimensions (beautiful, approachable, likeable, dangerous) for a total of ten blocks to better estimate the preferences of individual raters. First, we show that both intra-rater and inter-rater agreement – measures related to idiosyncratic and shared contributions, respectively – increase with repeated measures. Second, to find best practices, we compared estimates from correlation indices and variance component approaches on stimulus-generality, evaluation-generality, data preprocessing steps, and sensitivity to measurement error (a largely ignored issue). The correlation indices changed monotonically and nonlinearly with more repeated measures. Variance component analyses showed large variability in estimates from only two repeated measures, but stabilized with more measures. While there was general agreement among approaches, the correlation approach was problematic for certain stimulus types and evaluation dimensions. Our results suggest that variance component estimates are more reliable as long as one collects more than two repeated measures, which is not the current norm in psychological research, and can be implemented using mixed models with crossed random effects. Recommendations for analysis and interpretations are provided.
A confidence interval approach to investigating non-response bias and monitoring response to postal questionnaires.