Improving replicability using interaction with laboratories: a multi-lab experimental assessment

Abstract

Experimentation with mouse and rat models has become a central strategy for discovering mammalian gene function, and for preclinical testing of pharmacological treatments, yet the utility of any findings critically depends on their replicability in other laboratories. In previous publications we proposed a statistical approach for estimating the inter-laboratory replicability of novel discoveries made in a single laboratory. We demonstrated that previous phenotyping results from multi-lab databases can be used to derive a Genotype-by-Lab (GxL) adjustment factor to greatly enhance the replicability of the single-lab findings, for similarly measured phenotypes, even before making the effort of replicating these finding in additional laboratories. This demonstration, however, still raised several important questions that could only be answered by an additional large-scale prospective experiment: 1) Does GxL-adjustment work in single-lab experiments that were not intended to be standardized across laboratories, and with genotypes that were not included in the previous experiments? And 2) Can it be used to adjust the results of pharmacological experiments? We investigated these questions by attempting to replicate, across three laboratories, results from five single-lab studies in the Mouse Phenome Database (MPD), offering 212 comparisons, including 60 involving a pharmacological treatment: 18 mg/kg/day fluoxetine. In addition, we define and use a dimensionless GxL factor, by dividing the GxL variance by the standard deviation between animals within groups, as a more robust vehicle to transfer the adjustment from the multi-lab analysis to very different labs and genotypes. For genotype comparisons, GxL-adjustment reduced the rate of non-replicable discoveries from 60% to 12%, for the price of reducing the power to make replicable discoveries from 87% to 66%. In absolute numbers, the adjustment prevented 23 non-replicable discoveries for the price of missing only three replicated ones. Tools and data needed for deployment of this method across other mouse experiments are publicly available in MPD. Our results further point at some phenotypes as more prone to produce non-replicable results, while others, known to be more difficult to measure, are as likely to produce replicable results (once adjusted) such as the physiological measure, body weight.