A field-wide assessment of differential high throughput sequencing reveals widespread bias

Abstract

Here we assess inferential quality in the field of differential expression profiling by high throughput sequencing, based on analysis of datasets submitted 2008-2020 to the NCBI GEO data repository. We take advantage of the parallel differential expression testing over thousands of genes, whereby each experiment leads to a large set of p values, the distribution of which can indicate the validity of assumptions behind the test. Moreover, from a well-behaved p value set{pi} 0, the fraction of genes that are not differentially expressed, can be estimated. We found that only 25% of experiments resulted in theoretically expected p value histogram shapes, although there is a marked improvement over time. Uniform p value histogram shapes, indicative of < 100 true effects, were extremely few. Furthermore, although many HT-seq workflows assume that most genes are not differentially expressed we found 37% of experiments to have{pi} 0-s of less than 0.5, as if most genes changed their expression level. Restricting our analysis to studies involving cancer or transcription factors, expected to lead to real changes in expression of many genes, did not result in meaningfully different distributions of{pi} 0-s. Both the fractions of different p value histogram types and the{pi} 0 values are strongly associated with the differential expression analysis program used by the original authors. While we could double the proportion of theoretically expected p value distributions by removing low-count features from analysis, this treatment did not remove the association with the analysis program. Taken together, our results indicate widespread bias in differential expression profiling field.