Always Valid Inference: Continuous Monitoring of A/B Tests

A/B tests are typically analyzed via frequentist p-values and confidence intervals, but these inferences are wholly unreliable if users endogenously choose samples sizes by continuously monitoring their tests. We define always valid p-values and confidence intervals that let users try to take advantage of data as fast as it becomes available, providing valid statistical inference whenever they make their decision. Always valid inference can be interpreted as a natural interface for a sequential hypothesis test, which empowers users to implement a modified test tailored to them. In particular, we show in an appropriate sense that the measures we develop trade off sample size and power efficiently, despite a lack of prior knowledge of the user’s relative preference between these two goals. We also use always valid p-values to obtain multiple hypothesis testing control in the sequential context. Our methodology has been implemented in a large-scale commercial A/B testing platform to analyze hundreds of thousands of experiments to date.

Evaluating Stochastic Seeding Strategies in Networks

When trying to maximize the adoption of a behavior in a population connected by a social network, it is common to strategize about where in the network to seed the behavior, often with an element of randomness. Selecting seeds uniformly at random is a basic but compelling strategy in that it distributes seeds broadly throughout the network. A more sophisticated stochastic strategy, one-hop targeting, is to select random network neighbors of random individuals; this exploits a version of the friendship paradox, whereby the friend of a random individual is expected to have more friends than a random individual, with the hope that seeding a behavior at more connected individuals leads to more adoption. Many seeding strategies have been proposed, but empirical evaluations have demanded large field experiments designed specifically for this purpose and have yielded relatively imprecise comparisons of strategies. Here we show how stochastic seeding strategies can be evaluated more efficiently in such experiments, how they can be evaluated “off-policy” using existing data arising from experiments designed for other purposes, and how to design more efficient experiments. In particular, we consider contrasts between stochastic seeding strategies and analyze nonparametric estimators adapted from policy evaluation and importance sampling. We use simulations on real networks to show that the proposed estimators and designs can substantially increase precision while yielding valid inference. We then apply our proposed estimators to two field experiments, one that assigned households to an intensive marketing intervention and one that assigned students to an antibullying intervention. This paper was accepted by Gui Liberali, Special Issue on Data-Driven Prescriptive Analytics.

LIMIT THEOREMS FOR FACTOR MODELS

This paper establishes central limit theorems (CLTs) and proposes how to perform valid inference in factor models. We consider a setting where many counties/regions/assets are observed for many time periods, and when estimation of a global parameter includes aggregation of a cross-section of heterogeneous microparameters estimated separately for each entity. The CLT applies for quantities involving both cross-sectional and time series aggregation, as well as for quadratic forms in time-aggregated errors. This paper studies the conditions when one can consistently estimate the asymptotic variance, and proposes a bootstrap scheme for cases when one cannot. A small simulation study illustrates performance of the asymptotic and bootstrap procedures. The results are useful for making inferences in two-step estimation procedures related to factor models, as well as in other related contexts. Our treatment avoids structural modeling of cross-sectional dependence but imposes time-series independence.

The Rule of Contradictory Pairs, Insolubles and Validity

The Oxford Calculator Roger Swyneshed put forward three provocative claims in his treatise on insolubles, written in the early 1330s, of which the second states that there is a formally valid inference with true premises and false conclusion. His example deployed the Liar paradox as the conclusion of the inference: ‘The conclusion of this inference is false, so this conclusion is false’. His account of insolubles supported his claim that the conclusion is false, and so the premise, referring to the conclusion, would seem to be true. But what is his account of validity that can allow true premises to lead to a false conclusion? This article considers Roger’s own account, as well as that of Paul of Venice, writing some sixty years later, whose account of the truth and falsehood of insolubles followed Roger’s closely. Paul endorsed Roger’s three claims. But their accounts of validity were different. The question is whether these accounts are coherent and support Paul’s claim in his Logica Magna that he endorsed all the normal rules of inference.

FINITE-SAMPLE SIZE CONTROL OF IVX-BASED TESTS IN PREDICTIVE REGRESSIONS

In predictive regressions with variables of unknown persistence, the use of extended IV (IVX) instruments leads to asymptotically valid inference. Under highly persistent regressors, the standard normal or chi-squared limiting distributions for the usual t and Wald statistics may, however, differ markedly from the actual finite-sample distributions which exhibit in particular noncentrality. Convergence to the limiting distributions is shown to occur at a rate depending on the choice of the IVX tuning parameters and can be very slow in practice. A characterization of the leading higher-order terms of the t statistic is provided for the simple regression case, which motivates finite-sample corrections. Monte Carlo simulations confirm the usefulness of the proposed methods.

A simple estimator for quantile panel data models using smoothed quantile regressions

Summary This paper considers panel data models where the idiosyncratic errors are subject to conditonal quantile restrictions. We propose a two-step estimator based on smoothed quantile regressions that is easy to implement. The asymptotic distribution of the estimator is established, and the analytical expression of its asymptotic bias is derived. Building on these results, we show how to make asymptotically valid inference on the basis of both analytical and split-panel jackknife bias corrections. Finite-sample simulations are used to support our theoretical analysis and to illustrate the importance of bias correction in quantile regressions for panel data. Finally, in an empirical application, the proposed method is used to study the growth effects of foreign direct investment.

Sequent-Type Calculi for Three-Valued and Disjunctive Default Logic

Default logic is one of the basic formalisms for nonmonotonic reasoning, a well-established area from logic-based artificial intelligence dealing with the representation of rational conclusions, which are characterised by the feature that the inference process may require to retract prior conclusions given additional premisses. This nonmonotonic aspect is in contrast to valid inference relations, which are monotonic. Although nonmonotonic reasoning has been extensively studied in the literature, only few works exist dealing with a proper proof theory for specific logics. In this paper, we introduce sequent-type calculi for two variants of default logic, viz., on the one hand, for three-valued default logic due to Radzikowska, and on the other hand, for disjunctive default logic, due to Gelfond, Lifschitz, Przymusinska, and Truszczyński. The first variant of default logic employs Łukasiewicz’s three-valued logic as the underlying base logic and the second variant generalises defaults by allowing a selection of consequents in defaults. Both versions have been introduced to address certain representational shortcomings of standard default logic. The calculi we introduce axiomatise brave reasoning for these versions of default logic, which is the task of determining whether a given formula is contained in some extension of a given default theory. Our approach follows the sequent method first introduced in the context of nonmonotonic reasoning by Bonatti, which employs a rejection calculus for axiomatising invalid formulas, taking care of expressing the consistency condition of defaults.

Estimation and Inference for Linear Models with Two-Way Fixed Effects and Sparsely Matched Data

Models with multiway fixed effects are frequently used to address selection on unobservables. The data used for estimating these models often contain few observations per value of either indexing variable (sparsely matched data). I show that this sparsity has important implications for inference and propose an asymptotically valid inference method based on subsetting. Sparsity also has important implications for point estimation when covariates or instrumental variables are sequentially exogenous (e.g., dynamic models), and I propose a new estimator for these models. Finally, I illustrate these methods by providing estimates of the effect of class size reductions on student achievement.