The Implications of Functional Form Choice on Model Misspecification in Longitudinal Survey Mode Adjustments

In this chapter, we develop theory and methodology to support mode adjustment and hindcasting/forecasting in the presence of different possible mode effect types, including additive effects and odds-multiplicative effects. Mode adjustment is particularly important if the ultimate goal is to report one aggregate estimate of response parameters, and also to allow for comparison to historical surveys performed with different modes. Effect type has important consequences for inferential validity when the baseline response changes over time (i.e. when there is a time trend or time effect). We present a methodology to provide inference for additive and odds-multiplicative effect types, and demonstrate its performance in a simulation study. We also show that if the wrong effect type is assumed, the resulting inference can be invalid as confidence interval coverage is greatly reduced and estimates can be biased.

Comparison of Reliability in Seventeen European Countries Using the Quasi-Simplex Model

The aim of this chapter is to illustrate the application of the quasi-simplex model (QSM) for reliability estimation in longitudinal data and to employ it to obtain information about the reliability of the European Union—Survey on Income and Living Conditions (EU-SILC) data collected between 2012 and 2017. Reliability of two survey questions is analysed: one which asks respondents about the financial situation in their households, and one which requests information about respondents’ health. Employing the QSM on the two items resulted in 80 reliability estimates from 17 and 11 European countries, respectively. Results revealed statistically significant differences in reliability between post-communist Central and Eastern European (CEE) countries and the rest of Europe, and similar patterns of the size of reliability estimates were observed for both items. The highest reliability (i.e. reliability over 0.8) was observed in CEE countries such as Bulgaria, Romania, Czechia, Poland, and Hungary. The lowest reliability (i.e. reliability lower than 0.7) was observed for data from Sweden, Slovenia, Norway, Spain, Portugal, Austria, Italy, and the Netherlands. The remarkable variation in longitudinal reliability across culturally and historically different European regions is discussed both from substantive and methodological perspectives.

Developing Reliable Measures

This chapter presents a general approach to assessing the reliability of measurement of survey questions—those in common use in many surveys. The approach, which relies on a robust set of longitudinal design requirements, applies the quasi-Markov simplex model to multi-wave data in the evaluation of measurement errors for survey questions. Under particular assumptions, this model produces a set of estimates that conform to the psychometric definition of measurement reliability, defined as the ratio of true variance to observed variance. These models attribute some of the over-time inconsistency in measurements to unreliability and some to true change. This strategy rejects traditional notions of reliability that rely on internal consistency estimates for composite variables, as well as the simple test–retest approach to estimating reliability. Rather, the emphasis is on the separation of unreliability from true change in observations made over time. The importance of meeting several design requirements for using these over-time statistical models is also emphasized. These include the use of large-scale panel studies representative of known populations, with a minimum of three waves of measurement, separated by lengthy re-interview intervals, and limited to exactly replicated questions over the multiple waves. Results are presented from several three-wave panel studies that have employed this design, which provide evidence for the utility of the approach in the evaluation of the quality of survey measurement with respect to question content, context, and form.

Disappearing Errors in a Conversion Model

The same basic differential equation model has been adapted for time-dependent conversions of members of a population among different states. The conversion model has been applied in different contexts such as epidemiological infections, the Bass model for the diffusion of innovations, and the ideodynamic model for public opinion. For example, the ideodynamic version of the model predicts changes in public opinions in response to persuasive messages extending back to an indefinite past. All messages are measured with error, and this chapter discusses how errors in message measurements disappear with time so that predicted opinion values gradually become unaffected by past measurement errors. Prediction uncertainty is discussed using formal statistics, sensitivity analysis, and bootstrap variance calculations. This chapter presents ideodynamic predictions for opinion time series about the Toyota car manufacturer calculated from daily Twitter scores over two-and-a-half years. During this time, there was a sudden onslaught of bad news for Toyota, and the model was able to accurately predict the accompanying drop in favourable public opinion towards Toyota and rise in unfavourable opinion.

On Total Least Squares Estimation for Longitudinal Errors-in-Variables Models

The objective of this study is to evaluate the total least squares (TLS) estimator for the linear mixed model when the design matrix is subject to measurement errors, with special focus on models for longitudinal or repeated-measures data. We consider measurement errors only in the design matrix concerning the fixed part of the model and estimate its corresponding parameter vector under the TLS set up. After treating two variants of the general case, the random coefficient model is discussed as a special case. We evaluate conditions, on the design matrices as well as on variance component parameters, under which a reasonable TLS estimator can be expected in such models. Analysis of a real data example is also provided.

Longitudinal Measurement (Non)Invariance in Latent Constructs

The chapter discusses the basic principles and core problems of latent variable panel modelling, with a focus on the specification of error structures and (the evaluation of) longitudinal measurement invariance. We address alternative specifications of autocorrelative error structures, and demonstrate how to decompose the indicators’ residual variances into indicator-specific and random error components. Furthermore, besides describing the conventional global testing strategy for measurement (non)invariance, we contribute to the literature by integrating theoretical and analytical elements not yet extensively discussed outside the respective disciplines. We (i) introduce response shift theory as viable theoretical basis for the occurrence of noninvariance across time; (ii) provide a detailed description of model and scale identification strategies, accompanied by a critical reflection of their potential to adequately detect noninvariant parameters; and (iii) discuss the concepts of partial and approximate measurement invariance as well as the decomposition of response shifts and true change as different strategies of how to deal with measurement noninvariance.

A Methodological Framework for the Analysis of Panel Conditioning Effects

Panel conditioning refers to the phenomenon whereby respondents’ attitudes, behaviour, reporting of behaviour and/or knowledge are changed by repeated participation in a panel survey. Uncovering such effects, however, is difficult due to three major methodological challenges. First, researchers need to disentangle changes in behaviour and attitudes from changes in the reporting of behaviour and attitudes as panel conditioning may result in both, even at the same time and in opposite directions. Second, the identification of the causal effect of panel participation on the various forms of change mentioned above is complicated as it requires comparisons of panel respondents with control groups of people who have not been interviewed before. Third, other sources of error in (panel) surveys may easily be mistaken for panel conditioning if not properly accounted for. Such error sources are panel attrition, mode effects, and interviewer effects. In this chapter the challenges mentioned above are reviewed in detail and a methodological framework for the analysis of panel conditioning effects is provided by identifying the strengths and weaknesses of the various designs that researchers have developed to address the challenges. The chapter concludes with a discussion of a future research agenda on panel conditioning effects in longitudinal surveys.

Modelling Mode Effects for a Panel Survey in Transition

Funding reductions combined with increasing data-collection costs required that Wave V of the USA’s National Longitudinal Study of Adolescent to Adult Health (Add Health) abandon its traditional approach of in-person interviewing and adopt a more cost-effective method. This approach used the mail/web mode in Phase 1 of data collection and in-person interviewing for a random sample of nonrespondents in Phase 2. In addition, to facilitate the comparison of modes, a small random subsample served as the control and received the traditional in-person interview. We show that concerns about reduced data quality as a result of the redesign effort were unfounded based on findings from an analysis of the survey data. In several important respects, the new two-phase, mixed-mode design outperformed the traditional design with greater measurement accuracy, improved weighting adjustments for mitigating the risk of nonresponse bias, reduced residual (or post-adjustment) nonresponse bias, and substantially reduced total-mean-squared error of the estimates. This good news was largely unexpected based upon the preponderance of literature suggesting data quality could be adversely affected by the transition to a mixed mode. The bad news is that the transition comes with a high risk of mode effects for comparing Wave V and prior wave estimates. Analytical results suggest that significant differences can occur in longitudinal change estimates about 60 % of the time purely as an artifact of the redesign. This begs the question: how, then, should a data analyst interpret significant findings in a longitudinal analysis in the presence of mode effects? This chapter presents the analytical results and attempts to address this question.

Self-Evaluation, Differential Item Functioning, and Longitudinal Anchoring Vignettes

Anchoring vignettes are a powerful instrument to detect systematic differences in the use of self-reported ordinal survey responses. Not taking into account the (non-random) heterogeneity in reporting styles across different respondents may systematically bias the measurement of the variables of interest. The presence of such individual heterogeneity leads respondents to interpret, understand, or use the response categories for the same question differently. This phenomenon is defined as differential item functioning (DIF) in the psychometric literature. A growing amount of cross-sectional studies apply the anchoring vignette approach to tackle this issue but its use is still limited in the longitudinal context. This chapter introduces longitudinal anchoring vignettes for DIF correction, as well as the statistical approaches available when working with such data and how to investigate stability over time of individual response scales.

Estimating Mode Effects in Panel Surveys

An increasing number of longitudinal studies worldwide cover societal change over decades. The fieldwork of these panel surveys experienced notable technological advancements and often switched modes of data collection over time. A large body of literature documents that the mode of data collection matters for respondent behaviour. However, empirical evidence of mode effects on estimates of reliability and validity is surprisingly scarce. This chapter proposes to adapt the experimental multitrait–multimethod (MTMM) approach to observational data. For this purpose, we use the longitudinal data from the German Socio-Economic Panel (SOEP), an annual panel study started in 1984 with interviewer-administered paper-and-pencil interviews and nowadays additionally using computer-assisted personal interviewing as well as self-administered questionnaires, both mailed and web-based. While mode changes in the SOEP are not randomly assigned to respondents, we augment our approach by propensity score weighting to reduce selection bias in estimates of mode effects. Our analysis suggests moderate but statistically significant differences in reliability and validity estimates between modes of data collection.