Are Items More Than Indicators? An examination of psychometric homogeneity, item-specific effects, and consequences for structural equation models

Concerns of measurement error often motivate researchers to aggregate item information, using simple heuristics (e.g., sum scores) or latent variable methods, to mitigate unwanted effects such as parameter bias and attenuation. These approaches are often invoked without acknowledging that many scales in practice likely fail to possess the necessary properties for these models to be sufficient (i.e., positive conditional association and vanishing conditional dependence). We argue that measures which are not psychometrically homogeneous likely contain item specific effects particularly when examined in conjunction with external variables. We demonstrate this using a clinical empirical example assessing risk factors for suicidal ideation and show that measures constructed in alignment with principles of psychometric homogeneity are most appropriately modeled at the scale (or subscale) level while other measures should be considered at the item level. As a result, latent variable applications to such instruments are susceptible to interpretational confounding. The effects of interpretational confounding on R2, root mean square error, and model parameters are evaluated in a small simulation study. We conclude that item specific effects are not uncommon in practice and impact both explanatory and predictive research. Our findings suggest that classical approaches to addressing measurement error are insufficient to fully capture the breadth of instruments implemented in practice. Careful consideration of both the scale construction process and roles of scale items in the broader psychological theory are necessary prior to the application of traditional measurement methods.