This chapter presents a general approach to assessing the reliability of measurement of survey questions—those in common use in many surveys. The approach, which relies on a robust set of longitudinal design requirements, applies the quasi-Markov simplex model to multi-wave data in the evaluation of measurement errors for survey questions. Under particular assumptions, this model produces a set of estimates that conform to the psychometric definition of measurement reliability, defined as the ratio of true variance to observed variance. These models attribute some of the over-time inconsistency in measurements to unreliability and some to true change. This strategy rejects traditional notions of reliability that rely on internal consistency estimates for composite variables, as well as the simple test–retest approach to estimating reliability. Rather, the emphasis is on the separation of unreliability from true change in observations made over time. The importance of meeting several design requirements for using these over-time statistical models is also emphasized. These include the use of large-scale panel studies representative of known populations, with a minimum of three waves of measurement, separated by lengthy re-interview intervals, and limited to exactly replicated questions over the multiple waves. Results are presented from several three-wave panel studies that have employed this design, which provide evidence for the utility of the approach in the evaluation of the quality of survey measurement with respect to question content, context, and form.
Do Temporary Dropouts Improve the Composition of Panel Data? An Analysis of “Gap Interviews” in the German Family Panel pairfam
