Detecting cell segmentation errors using doublet methods
In spatial transcriptomics, cell segmentation is used to draw boundaries around cells. Molecules located within a cell's boundary are assigned to it, making its gene expression profile dependent on segmentation accuracy. To identify potentially problematic cells, studies increasingly use doublet detection methods, a class of algorithms developed to recognize molecular admixture from two cells in scRNA-seq. To evaluate their suitability in spatial data, we model cell segmentation errors as a continuum of partial molecular admixture between neighboring cells, generated by varying the loss of a cell's own transcripts and the gain of transcripts from its neighbor. Evaluating 8 doublet methods across 16 spatial datasets, we characterize the conditions under which they perform well and identify their failure modes. Detection improves with increasing molecular admixture and transcriptional dissimilarity between neighboring cells, with cxds2, scDblFinder.score, and a simple baseline (unique_genes) performing best. These patterns are consistent across datasets spanning different gene panels, technological platforms, staining techniques, segmentation algorithms, and error frequencies. In unperturbed spatial data, elevated doublet scores localize to regions consistent with segmentation problems, suggesting that they can help prioritize cells or regions for inspection, segmentation refinement, or transcript reassignment. Our results establish when doublet methods can provide useful quality control signals for cell segmentation, supporting their use in spatial transcriptomics.