Improving replicability using interaction with laboratories: a multi-lab experimental assessment
AbstractPhenotyping inbred and genetically-engineered mouse lines has become a central strategy for discovering mammalian gene function and evaluating pharmacological treatment. Yet the utility of any findings critically depends on their replicability in other laboratories. In previous publications we proposed a statistical approach for estimating the inter-laboratory replicability of novel discoveries in a single laboratory, and demonstrated that previous phenotyping results from multi-lab databases can be used to derive a Genotype-by-Lab (GxL) adjustment factor to ensure the replicability of single-lab results, for similarly measured phenotypes, even before making the effort of replicating the new finding in additional laboratories.The demonstration above, however, still raised several important questions that could only be answered by an additional large-scale prospective experiment: Does GxL-adjustment works in single-lab experiments that were not intended to be standardized across laboratories? With genotypes that were not included in the previous experiments? And can it be used to adjust the results of pharmacological treatment experiments? We replicated results from five studies in the Mouse Phenome Database (MPD), in three behavioral tests, across three laboratories, offering 212 comparisons including 60 involving a pharmacological treatment: 18 mg/kg/day fluoxetine. In addition, we define and use a dimensionless GxL factor, derived from dividing the GxL variance by the standard deviation between animals within groups, as the more robust vehicle to transfer the adjustment from the multi-lab analysis to very different labs and genotypes.For genotype comparisons, GxL-adjustment reduced the rate of non-replicable discoveries from 60% to 12%, for the price of reducing the power to make replicable discoveries from 87% to 66%. Another way to look at these results is noting that the adjustment could have prevented 23 failures to replicate, for the price of missing only three replicated ones. The tools and data needed for deployment of this method across other mouse experiments are publicly available in the Mouse Phenome Database.Our results further point at some phenotypes as more prone to produce non-replicable results, while others, known to be more difficult to measure, are as likely to produce replicable results (once adjusted) as the physiological body weight is.
