The paltry power of priors versus populations

Abstract

Genome-wide tests, including genome-wide association studies (GWAS) of germ-line genetic variants, driver tests of cancer somatic mutations, and transcriptome-wide association tests of RNA-Seq data, carry a high multiple testing burden. This burden can be overcome by enrolling larger cohorts or alleviated by using prior biological knowledge to favor some hypotheses over others. Here we compare these two methods in terms of their abilities to boost the power of hypothesis testing. We provide a quantitative estimate for progress in cohort sizes, and present a theoretical analysis of the power of oracular hard priors: priors that select a subset of hypotheses for testing, with an oracular guarantee that all true positives are within the tested subset. This theory demonstrates that for GWAS, strong priors that limit testing to 100-1000 genes provide less power than typical annual 20-40% increases in cohort sizes. These theoretical results explain the continued dominance of simple, unbiased univariate hypothesis tests for RNA-Seq studies and GWAS: if a statistical question can be answered by larger cohort sizes, it should be answered by larger cohort sizes rather than by more complicated biased methods involving priors. We suggest that priors are better suited for non-statistical aspects of biology, such as pathway structure and causality, that are not yet easily captured by standard hypothesis tests. Author summaryBiological experiments often test thousands to millions of hypotheses. Gene-based tests for human RNA-Seq data, for example, involve approximately 20,000 tests; genome-wide association studies (GWAS) involve about 1 million effective tests. A robust approach is to perform individual tests and then apply a Bonferroni correction to account for multiple testing. This approach implies a single-test p-value of 2.5 x 10-6 for RNA-Seq experiments, and a p-value of 5 x 10-8 for GWAS, to control the false-positive rate at a conventional value of 0.05. Many methods have been proposed to alleviate the multiple-testing burden by incorporating a prior probability that boosts the significance for a subset of candidate genes or variants. At the extreme limit, only hypotheses within a candidate set are tested, corresponding to a decreased multiple testing burden. Despite decades of methods development, prior-based tests have not been generally used. Here we compare the power increase possible with a prior with the power increase from a much simpler strategy of increasing a study size. We show that increasing the population size is exponentially more valuable than increasing the strength of prior, even when the true prior is known exactly. Furthermore, even modest yearly increases in actual GWAS cohorts can yield power gains beyond the reach of any reasonable prior. These results provide a rigorous explanation for the continued use of simple, robust methods rather than more sophisticated approaches. They suggest that the value of priors is not in multiple hypothesis testing but rather in non-statistical aspects of interpretation including pathway structure and causality.