Toward a Causally Informative Fit Index of Longitudinal Models: A Within-Study Design Approach

In structural equation modeling, plausible competing theories can imply similar or equivalent covariance matrices and thus show similar or identical model fit indices, despite making very different causal predictions. We propose a method for selecting among longitudinal models on the basis of causal information. We use a within-study design approach and present an index of causal fit for choosing among models on the basis of their fit with causally informative estimates, in cases in which research designs allow for strong causal estimates. We test for the usefulness and validity of the approach by applying it to data from three randomized controlled trials of early math interventions with longitudinal follow-up assessments. We find that, across datasets, some models consistently outperform other models at forecasting later experimental impacts, traditional fit indices are not strongly related to our index of causal fit, and models show consistent patterns of similarity and discrepancy between statistical fit and causal fit. A simulation study finds that when assumptions are met, the index of causal fit can recover the generating model at rates higher than those of statistical fit indices, and is less redundant with statistical fit indices than they are with each other. Results support the validity of our proposed approach and suggest that it can be useful for choosing among models.