NicheCorr: Identifying Niche-Specific Differential Gene Co-Expression in Spatial Transcriptomics

Abstract

Cell-cell communication is central to biological processes such as immune responses and tumor progression, yet the role of spatial proximity between distinct cell types in shaping gene co-expression remains poorly understood. Existing approaches primarily identify differences in average gene expression across tissue regions, often overlooking how local cellular microenvironments reshape gene-gene interactions within cells. To address this gap, we introduce NicheCorr, a statistical procedure that identifies cell type-specific niche-associated gene co-expression changes within distinct spatial niches. In simulations, NicheCorr attained the highest power for detecting niche-dependent co-expression. It was also the only method evaluated that maintained Type I error control when mean expression varied nonlinearly with the niche, a setting in which CS-CORE, scHOT without mean adjustment, and the Pearson test all exhibited inflated false positive rates. Furthermore, NicheCorr integrates ligand-receptor (LR) inference, enabling a comprehensive view of spatially structured signaling interactions. We applied the framework to the CosMx SMI Human Liver FFPE Dataset and the Xenium Human Pancreatic Cancer Dataset, identifying 52 gene pairs in liver tumor cells whose co-expression changes with T-cell niche composition and 48 gene pairs in pancreatic cancer whose fibroblast niche-dependent co-expression differs between tumor and CFTR- tumor cells, and uncovering potential LR interactions that may underlie these differential co-expression patterns.