Covariate Adjustment in Randomized Controlled Trials

Introduction & Objective: Unadjusted analyses, fully adjusted analyses, or adjusted analyses based on tests of significance on covariate imbalance are recommended for covariate adjustment in randomized controlled trials. It has been indicated that the tests of significance on baseline comparability is inappropriate, rather it is important to indicate the strength of relationship with outcomes. Our goal is to understand when the adjustment should be used in randomized controlled trials. Methods: Unadjusted analysis, fully adjusted analysis, and adjusted analysis based on baseline comparability were examined under null and alternative hypothesis by simulation studies. Each data set was simulated 3000 times for a total of 9 scenarios for sample sizes of 20, 40, and 100 each with baseline thresholds of 0.05, 0.1, and 0.2. Each scenario was examined by the change in magnitude of correlation from 0.1 to 0.9. Results: Power of fully adjusted analysis under alternative hypothesis was increased as the correlation increased while adjusted analysis based on the covariate imbalance did not compare favorably to the unadjusted analysis. Type 1 error was decreased in adjusted analysis based on the covariate imbalance under null hypothesis. It was then observed that p-value does not follow a uniform distribution under the null hypothesis. Conclusion: Unadjusted and fully adjusted analyses were valid analyses. Full adjustment could potentially increase the power if adjustment is known. However, adjusted analysis based on the test of significance on covariate imbalance may not be a valid analysis. Tests of significance should not be used for comparing baseline comparability.

That’s not a two-sided test! It’s two one-sided tests!

When carrying out and reporting tests of significance, researchers might claim to have conducted a two-sided test when in fact they have conducted one-sided tests. Mark Rubin explains the confusion and how to avoid it.

Tests of Statistical Significance

Chapter 3 provides a brief overview of null hypothesis significance testing and points out its primary defects. It then outlines the neo-Fisherian account of tests of statistical significance, along with a second option contained in the philosophy of statistics known as the error-statistical philosophy, both of which are defensible. Tests of statistical significance are the most widely used means for evaluating hypotheses and theories in psychology. A massive critical literature has developed in psychology, and the behavioral sciences more generally, regarding the worth of these tests. The chapter provides a list of important lessons learned from the ongoing debates about tests of significance.