Panel experiments and dynamic causal effects: A finite population perspective

In panel experiments, we randomly assign units to different interventions, measuring their outcomes, and repeating the procedure in several periods. Using the potential outcomes framework, we define finite population dynamic causal effects that capture the relative effectiveness of alternative treatment paths. For a rich class of dynamic causal effects, we provide a nonparametric estimator that is unbiased over the randomization distribution and derive its finite population limiting distribution as either the sample size or the duration of the experiment increases. We develop two methods for inference: a conservative test for weak null hypotheses and an exact randomization test for sharp null hypotheses. We further analyze the finite population probability limit of linear fixed effects estimators. These commonly‐used estimators do not recover a causally interpretable estimand if there are dynamic causal effects and serial correlation in the assignments, highlighting the value of our proposed estimator.

When possible, report a Fisher-exactPvalue and display its underlying null randomization distribution

In randomized experiments, Fisher-exactPvalues are available and should be used to help evaluate results rather than the more commonly reported asymptoticPvalues. One reason is that using the latter can effectively alter the question being addressed by including irrelevant distributional assumptions. The Fisherian statistical framework, proposed in 1925, calculates aPvalue in a randomized experiment by using the actual randomization procedure that led to the observed data. Here, we illustrate this Fisherian framework in a crossover randomized experiment. First, we consider the first period of the experiment and analyze its data as a completely randomized experiment, ignoring the second period; then, we consider both periods. For each analysis, we focus on 10 outcomes that illustrate important differences between the asymptotic and Fisher tests for the null hypothesis of no ozone effect. For some outcomes, the traditionalPvalue based on the approximating asymptotic Student’stdistribution substantially subceeded the minimum attainable Fisher-exactPvalue. For the other outcomes, the Fisher-exact null randomization distribution substantially differed from the bell-shaped one assumed by the asymptoticttest. Our conclusions: When researchers choose to reportPvalues in randomized experiments, 1) Fisher-exactPvalues should be used, especially in studies with small sample sizes, and 2) the shape of the actual null randomization distribution should be examined for the recondite scientific insights it may reveal.

The analysis of randomized experiments with orthogonal block structure. I. Block structure and the null analysis of variance

Nearly all the experimental designs so far proposed assume that the experimental units are or can be grouped together in blocks in ways that can be described in terms of nested and cross-classifications. When every nesting classification employed has equal numbers of subunits nested in each unit, then the experimental units are said to have a simple block structure . Every simple block structure has a complete randomization theory. Any permutation of the labelling of the units is permissible which preserves the block structure, and this defines a valid randomization procedure. In a null experiment all units receive the same treatment, and the population of all possible vectors of yields generated by the randomization procedure gives the null randomization distribution. The covariance matrix of this distribution, the null analysis of variance, and the expectations of the various mean squares in it are all derivable from the initial description of the block structure. All simple block structures can be characterized by a set of mutually orthogonal idempotent matrices C i . Such sets of matrices may also exist for non-simple block structures (e. g. those having unequal numbers of plots in a block), and the existence of such a set defines an orthogonal block structure. Non-simple orthogonal block structures do not have a complete randomization theory and inferences from them require further assumptions.