Integrating single-cell and single-nucleus datasets improves bulk RNA-seq deconvolution

Abstract

Bulk RNA-seq deconvolution typically uses single-cell RNA-sequencing (scRNA-seq) references, but some cell types are only detectable through single-nucleus RNA sequencing (snRNA-seq). Because snRNA-seq captures nuclear, but not cytoplasmic, transcripts, direct use as a reference could reduce deconvolution accuracy. Here, we systematically benchmark strategies to integrate both modalities, focusing on transformations and gene-filtering approaches that harmonize snRNA-seq with scRNA-seq references. Across four diverse tissues, we evaluated principal component-based shifts, conditional and non-conditional variational autoencoders (scVI), and the removal of cross-modality differentially expressed genes (DEGs). While all methods improved performance relative to untransformed snRNA-seq, filtering consistent cross-modality DEGs delivered the greatest gains, often matching or surpassing scRNA-only references. Conditional scVI performed comparably and was especially effective when matched scRNA-snRNA cell types were unavailable. In real adipose bulk samples without ground truth, DEG pruning and conditional scVI provided the most robust cell-fraction estimates across donors and transformations. Together, these results demonstrate that scRNA-seq should be prioritized as the reference when available, with snRNA-seq appended only after filtering cross-modality DEGs. For less-characterized systems where DEG information is limited, conditional scVI offers a practical alternative. Our findings provide clear guidelines for modality-aware integration, enabling near-scRNA-seq accuracy in bulk deconvolution workflows.