ManyClasses 1: Assessing the Generalizable Effect of Immediate Feedback Versus Delayed Feedback Across Many College Classes

Psychology researchers have long attempted to identify educational practices that improve student learning. However, experimental research on these practices is often conducted in laboratory contexts or in a single course, which threatens the external validity of the results. In this article, we establish an experimental paradigm for evaluating the benefits of recommended practices across a variety of authentic educational contexts—a model we call ManyClasses. The core feature is that researchers examine the same research question and measure the same experimental effect across many classes spanning a range of topics, institutions, teacher implementations, and student populations. We report the first ManyClasses study, in which we examined how the timing of feedback on class assignments, either immediate or delayed by a few days, affected subsequent performance on class assessments. Across 38 classes, the overall estimate for the effect of feedback timing was 0.002 (95% highest density interval = [−0.05, 0.05]), which indicates that there was no effect of immediate feedback compared with delayed feedback on student learning that generalizes across classes. Furthermore, there were no credibly nonzero effects for 40 preregistered moderators related to class-level and student-level characteristics. Yet our results provide hints that in certain kinds of classes, which were undersampled in the current study, there may be modest advantages for delayed feedback. More broadly, these findings provide insights regarding the feasibility of conducting within-class randomized experiments across a range of naturally occurring learning environments.

Rhinovirus prevalence as indicator for efficacy of measures against SARS-CoV-2

Background Non-pharmaceutical measures to control the spread of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) should be carefully tuned as they can impose a heavy social and economic burden. To quantify and possibly tune the efficacy of these anti-SARS-CoV-2 measures, we have devised indicators based on the abundant historic and current prevalence data from other respiratory viruses. Methods We obtained incidence data of 17 respiratory viruses from hospitalized patients and outpatients collected by 37 clinics and laboratories between 2010-2020 in Germany. With a probabilistic model for Bayes inference we quantified prevalence changes of the different viruses between months in the pre-pandemic period 2010-2019 and the corresponding months in 2020, the year of the pandemic with noninvasive measures of various degrees of stringency. Results We discovered remarkable reductions δ in rhinovirus (RV) prevalence by about 25% (95% highest density interval (HDI) [−0.35,−0.15]) in the months after the measures against SARS-CoV-2 were introduced in Germany. In the months after the measures began to ease, RV prevalence increased to low pre-pandemic levels, e.g. in August 2020 δ=−0.14 (95% HDI [−0.28,0.12]). Conclusions RV prevalence is negatively correlated with the stringency of anti-SARS-CoV-2 measures with only a short time delay. This result suggests that RV prevalence could possibly be an indicator for the efficiency for these measures. As RV is ubiquitous at higher prevalence than SARS-CoV-2 or other emerging respiratory viruses, it could reflect the efficacy of noninvasive measures better than such emerging viruses themselves with their unevenly spreading clusters.

Predicting individual shelter dog behaviour after adoption using longitudinal behavioural assessment: a hierarchical Bayesian approach

Predicting the behaviour of shelter dogs after adoption is an important, but difficult, endeavour. Differences between shelter and post-adoption environments, between- and within-individual heterogeneity in behaviour, uncertainty in behavioural predictions and measurement error all hinder the accurate assessment of future behaviour. This study integrates 1) a longitudinal behavioural assessment with 2) a novel joint hierarchical Bayesian mixture model that accounts for individual variation, missing data and measurement error to predict behaviour post-adoption. We analysed shelter observations (> 28,000 records) and post-adoption reports (from telephone surveys) on the behaviour of 241 dogs across eight contexts. Dog behaviour at the shelter correlated positively with behaviour post-adoption within contexts (r = 0.38; 95% highest density interval: [0.20, 0.55]), and behavioural repeatability was approximately 20% higher post-adoption for behaviour within contexts. Although measurement error was higher post-adoption than at the shelter, we found few differences in individual-level, latent probabilities of different behaviours post-adoption versus at the shelter. This good predictive ability was aided by accurate representation of uncertainty in individual-level predictions. We conclude that longitudinal assessment paired with a sufficient inferential framework to model latent behavioural profiles with uncertainty enables reasonably accurate estimation of post-adoption behaviour.

Abstract P579: Fluid Attenuated Inversion Recovery Correlates With Time of Onset of Arterial Ischemic Stroke in Childhood

Background: In adults, time since stroke onset correlates with efficacy and risk of recanalization therapies; beyond this temporal window there is increased risk of adverse events in particular hemorrhage with thrombolysis and decreased benefit of recanalization due to irreversible tissue injury. In adults the appearance of fluid attenuated inversion recovery (FLAIR) signal is a proxy for time, and is typically present by 6 hours. The time to FLAIR signal hyperintensity in childhood stroke is unknown but is of interest with potential utility due to the often greater uncertainty as to timing symptom onset. Methods: Time to FLAIR signal hyperintensity on brain MRI performed on children within 24 hours of stroke onset was studied with logistic regression. Results: A total of 68 MRIs with FLAIR imaging were available from 54 children (27 female), age 0.8 to 17.9 years, median 12.0 years. Seventy-four percent (40/54) of children and 72% (49/68) of scans had anterior circulation stroke. Interquartile range for time to FLAIR presence was 7.8 to 19.1 hours. The 90% probability of FLAIR change was reached at 11.2 hours for all strokes (Figure, dotted line; 80% highest density interval (HDI): 1.2-11.2 hours), and 9.3 hours for anterior circulation only strokes (Figure, solid line; 80% HDI: 3.3-9.3 hours), though nearly all had FLAIR change by 6 hours. FLAIR change was absent in 4 children after 6 hours, two with anterior circulation stroke (16 year-old at 6.1 hours, 10 year-old at 7.0 hours) and 2 with posterior circulation stroke (15 year-old at 7.3 hours, 9 month-old at 18.2 hours). Conclusion: Similar to adults, FLAIR hyperintensity can be used to estimate time since stroke ictus in childhood stroke. Children may have somewhat delayed time to FLAIR signal change compared with adults, suggesting that they may have a longer window for effective recanalization therapies.

Decisions About Equivalence: A Comparison of TOST, HDI-ROPE, and the Bayes Factor

Some important research questions require the ability to find evidence for two conditions being practically equivalent. This is impossible to accomplish within the traditional frequentist null hypothesis significance testing framework; hence, other methodologies must be utilized. We explain and illustrate three approaches for finding evidence for equivalence: The frequentist two one-sided tests procedure, the Bayesian highest density interval region of practical equivalence procedure, and the Bayes factor interval null procedure. We compare the classification performances of these three approaches for various plausible scenarios. The results indicate that the Bayes factor interval null approach compares favorably to the other two approaches in terms of statistical power. Critically, compared to the Bayes factor interval null procedure, the two one-sided tests and the highest density interval region of practical equivalence procedures have limited discrimination capabilities when the sample size is relatively small: specifically, in order to be practically useful, these two methods generally require over 250 cases within each condition when rather large equivalence margins of approximately 0.2 or 0.3 are used; for smaller equivalence margins even more cases are required. Because of these results, we recommend that researchers rely more on the Bayes factor interval null approach for quantifying evidence for equivalence, especially for studies that are constrained on sample size.

Hazard from Himalayan glacier lake outburst floods

Sustained glacier melt in the Himalayas has gradually spawned more than 5,000 glacier lakes that are dammed by potentially unstable moraines. When such dams break, glacier lake outburst floods (GLOFs) can cause catastrophic societal and geomorphic impacts. We present a robust probabilistic estimate of average GLOFs return periods in the Himalayan region, drawing on 5.4 billion simulations. We find that the 100-y outburst flood has an average volume of 33.5+3.7/−3.7 × 106 m3 (posterior mean and 95% highest density interval [HDI]) with a peak discharge of 15,600+2,000/−1,800 m3⋅s−1. Our estimated GLOF hazard is tied to the rate of historic lake outbursts and the number of present lakes, which both are highest in the Eastern Himalayas. There, the estimated 100-y GLOF discharge (∼14,500 m3⋅s−1) is more than 3 times that of the adjacent Nyainqentanglha Mountains, and at least an order of magnitude higher than in the Hindu Kush, Karakoram, and Western Himalayas. The GLOF hazard may increase in these regions that currently have large glaciers, but few lakes, if future projected ice loss generates more unstable moraine-dammed lakes than we recognize today. Flood peaks from GLOFs mostly attenuate within Himalayan headwaters, but can rival monsoon-fed discharges in major rivers hundreds to thousands of kilometers downstream. Projections of future hazard from meteorological floods need to account for the extreme runoffs during lake outbursts, given the increasing trends in population, infrastructure, and hydropower projects in Himalayan headwaters.

HDI-Forest: Highest Density Interval Regression Forest

By seeking the narrowest prediction intervals (PIs) that satisfy the specified coverage probability requirements, the recently proposed quality-based PI learning principle can extract high-quality PIs that better summarize the predictive certainty in regression tasks, and has been widely applied to solve many practical problems. Currently, the state-of-the-art quality-based PI estimation methods are based on deep neural networks or linear models. In this paper, we propose Highest Density Interval Regression Forest (HDI-Forest), a novel quality-based PI estimation method that is instead based on Random Forest. HDI-Forest does not require additional model training, and directly reuses the trees learned in a standard Random Forest model. By utilizing the special properties of Random Forest, HDI-Forest could efficiently and more directly optimize the PI quality metrics. Extensive experiments on benchmark datasets show that HDI-Forest significantly outperforms previous approaches, reducing the average PI width by over 20% while achieving the same or better coverage probability.

Accounting for Estimation Uncertainty and Shrinkage in Bayesian Within-Subject Intervals: A Comment on Nathoo, Kilshaw, and Masson (2018)

To facilitate the interpretation of systematic mean differences in within-subject designs, Nathoo, Kilshaw, and Masson (2018, Journal of Mathematical Psychology, 86, 1-9) proposed a Bayesian within-subject highest-density interval (HDI). However, their approach rests on independent maximum-likelihood estimates for the random effects which do not take estimation uncertainty and shrinkage into account. I propose an extension of Nathoo et al.'s method using a fully Bayesian, two-step approach. First, posterior samples are drawn for the linear mixed model. Second, the within-subject HDI is computed repeatedly based on the posterior samples, thereby accounting for estimation uncertainty and shrinkage. After marginalizing over the posterior distribution, the two-step approach results in a Bayesian within-subject HDI with a width similar to that of the classical within-subject confidence interval proposed by Loftus and Masson (1994, Psychonomic Bulletin & Review, 1, 476-490).

Rejecting or Accepting Parameter Values in Bayesian Estimation

This article explains a decision rule that uses Bayesian posterior distributions as the basis for accepting or rejecting null values of parameters. This decision rule focuses on the range of plausible values indicated by the highest density interval of the posterior distribution and the relation between this range and a region of practical equivalence (ROPE) around the null value. The article also discusses considerations for setting the limits of a ROPE and emphasizes that analogous considerations apply to setting the decision thresholds for p values and Bayes factors.

Does the 5-HT1A rs6295 polymorphism influence the safety and efficacy of citalopram therapy in the oldest old?

Major depressive disorder (MDD) in older people is a relatively common, yet hard to treat problem. In this study, we aimed to establish if a single nucleotide polymorphism in the 5-HT1A receptor gene (rs6295) determines antidepressant response in patients aged > 80 years (the oldest old) with MDD. Nineteen patients aged at least 80 years with a new diagnosis of MDD were monitored for response to citalopram 20 mg daily over 4 weeks and genotyped for the rs6295 allele. Both a frequentist and Bayesian analysis was performed on the data. Bayesian analysis answered the clinically relevant question: ‘What is the probability that an older patient would enter remission after commencing selective serotonin reuptake inhibitor (SSRI) treatment, conditional on their rs6295 genotype?’ Individuals with a CC (cytosine–cytosine) genotype showed a significant improvement in their Geriatric Depression Score ( p = 0.020) and cognition ( p = 0.035) compared with other genotypes. From a Bayesian perspective, we updated reports of antidepressant efficacy in older people with our data and calculated that the 4-week relative risk of entering remission, given a CC genotype, is 1.9 [95% highest-density interval (HDI) 0.7–3.5], compared with 0.52 (95% HDI 0.1–1.0) for the CG (cytosine–guanine) genotype. The sample size of n = 19 is too small to draw any firm conclusions, however, the data suggest a trend indicative of a relationship between the rs6295 genotype and response to citalopram in older patients, which requires further investigation.