Science suffers from a reproducibility crisis. Specification Curve Analysis (SCA) helps address this crisis by preventing the selective reporting of results and arbitrary data analysis choices. SCA plots the variability (or heterogeneity) of treatment effects against all ‘reasonable specifications’ (ways to conduct analysis). However, SCA has also been used for formal statistical inference on a type of global average (median) treatment effect (ATE), leading a study by Orben & Przybylski to conclude that ‘the association of [adolescent mental] well-being with regularly eating potatoes was nearly as negative as the association with technology use.’ In contrast, we find relevant associations between certain technologies and well-being, and sharp discrepancies between parent and teenager assessments. These heterogeneous effects are masked by taking medians. In layman’s terms, an ATE may appear practically irrelevant due to averaging over apples and oranges. In addition, the SCA median can have large bias and variance, due to over-weighting statistically implausible control variable specifications. With the Bayesian Specification Curve Analysis (BSCA) we extend SCA to estimate both individual and, if desired, average treatment effects, with controls weighted via Bayesian Model Averaging. The strategy allows to test individual effects, a missing feature in SCA, while improving statistical properties and protecting against false positives. We provide R code that implements BSCA and reproduces our analyses.
Covariate selection and model averaging in semiparametric estimation of treatment effects