Replicability of bulk RNA-Seq differential expression and enrichment analysis results in cancer research

Abstract

The high-dimensional and heterogeneous nature of transcriptomics data from RNA sequencing (RNA-Seq) experiments poses a challenge to routine down-stream analysis steps, such as differential expression analysis and enrichment analysis. Additionally, due to practical and financial constraints, RNA-Seq experiments are often limited to a small number of biological replicates; three replicates is a commonly employed minimum cohort size. In light of recent studies on the low replicability of preclinical cancer research, it is essential to understand how the combination of population heterogeneity and underpowered cohort sizes affects the replicability of RNA-Seq research. Using 7000 simulated RNA-Seq experiments based on real gene expression data from seven different cancer types, we find that the analysis results from underpowered experiments exhibit inflated effect sizes and are unlikely to replicate well. However, the ground-truth results obtained by analyzing large cohorts show that the precision of differentially expressed genes can be high even for small cohort sizes. The poor replicability of underpowered experiments is thus a direct consequence of their low recall (sensitivity). In other words, the low replicability of underpowered RNA-Seq cancer studies does not necessarily indicate a high prevalence of false positives. Instead, the results obtained from such studies are limited to small and mostly random subsets of a larger ground truth. We conclude with a set of practical recommendations to alleviate problems with underpowered RNA-Seq studies. Author SummaryTranscriptomics data from RNA sequencing (RNA-Seq) experiments are complex and challenging to analyze due to their high dimensionality and variability. These experiments often involve limited biological replicates due to practical and financial constraints. Recent concerns about the replicability of cancer research highlight the need to explore how this combination of limited cohort sizes and population heterogeneity impacts the reliability of RNA-Seq studies. To investigate these issues, we conducted 7000 simulated RNA-Seq experiments based on real gene expression data from seven different cancer types. We show that experiments with small cohort sizes tend to produce results with exaggerated effects that can be difficult to replicate. We further found that while underpowered studies with few replicates indeed lead to little-replicable results, the identified differentially expressed genes are reliable as shown by low rates of false positives. Each underpowered study thus discovers a small subset of the ground truth. Our study concludes with practical recommendations for RNA-Seq studies with small cohort sizes.