Combating Effort Avoidance in Computer Adaptive Practicing: Does a Problem Skipping Restriction Promote Learning?

Effort is key in learning, evidenced by its omnipresence in both empirical findings and educational theories. At the same time, students are consistently found to avoid effort. In this study, we investigate whether limiting effort avoidance improves learning outcomes, and explore for whom this would be the case. In a large-scale computer adaptive practice system for primary education, over 150,000 participants were distributed across four conditions in which a problem-skipping option was delayed for 0, 3, 6, or 9 seconds. The results show that after a 14 week period, no average treatment effects in learning outcomes can be found between conditions. A substantive typology of students, based on the expected target mechanisms of the intervention, neither shows consistent conditional average treatment effects. Nevertheless, the substantive typology is shown to be meaningful, as the different types—toilers, skippers, and rushers—differ with respect to their learning outcomes. We argue that although the scale of the experiment suggests a precise null finding, the cumulative nature of the effect of problem skipping cautions against generalizing this finding to sustained intervention.

Debiased machine learning of conditional average treatment effects and other causal functions

Summary This paper provides estimation and inference methods for the best linear predictor (approximation) of a structural function, such as conditional average structural and treatment effects, and structural derivatives, based on modern machine learning tools. We represent this structural function as a conditional expectation of an unbiased signal that depends on a nuisance parameter, which we estimate by modern machine learning techniques. We first adjust the signal to make it insensitive (Neyman-orthogonal) with respect to the first-stage regularisation bias. We then project the signal onto a set of basis functions, which grow with sample size, to get the best linear predictor of the structural function. We derive a complete set of results for estimation and simultaneous inference on all parameters of the best linear predictor, conducting inference by Gaussian bootstrap. When the structural function is smooth and the basis is sufficiently rich, our estimation and inference results automatically target this function. When basis functions are group indicators, the best linear predictor reduces to the group average treatment/structural effect, and our inference automatically targets these parameters. We demonstrate our method by estimating uniform confidence bands for the average price elasticity of gasoline demand conditional on income.