Detecting cell segmentation errors using doublet methods
- 1University of Toronto
Abstract
In spatial transcriptomics, cell segmentation is used to draw boundaries around cells. Molecules located within a cell's boundary are assigned to it, making its gene expression profile dependent on segmentation accuracy. To identify potentially problematic cells, studies increasingly use doublet detection methods, a class of algorithms developed to recognize molecular admixture from two cells in scRNA-seq. To evaluate their suitability in spatial data, we model cell segmentation errors as a continuum of partial molecular admixture between neighboring cells, generated by varying the loss of a cell's own transcripts and the gain of transcripts from its neighbor. Evaluating 8 doublet methods across 16 spatial datasets, we characterize the conditions under which they perform well and identify their failure modes. Detection improves with increasing molecular admixture and transcriptional dissimilarity between neighboring cells, with cxds2, scDblFinder.score, and a simple baseline (unique_genes) performing best. These patterns are consistent across datasets spanning different gene panels, technological platforms, staining techniques, segmentation algorithms, and error frequencies. In unperturbed spatial data, elevated doublet scores localize to regions consistent with segmentation problems, suggesting that they can help prioritize cells or regions for inspection, segmentation refinement, or transcript reassignment. Our results establish when doublet methods can provide useful quality control signals for cell segmentation, supporting their use in spatial transcriptomics.
Related
- A high-resolution blood immune cell doublet atlas via imaging spectral cytometry
- Order Matters: Ambient RNA correction confounds doublet detection in single-cell RNA sequencing
- From segmentation to morphometry: A post-processed U-Net framework for corneal endothelial cell analysis
- Agentic AI enables cell segmentation-free cell-type inference and cell-type-guided cell segmentation in spatial transcriptomics
- vafSOMA: Detecting cell-layer-fixed somatic mutations in plants
BibTeX
@misc{sarwar2026detecting,
title = {{Detecting cell segmentation errors using doublet methods}},
author = {A. Sarwar and J. Gillis},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.15.751798},
url = {https://www.biorxiv.org/content/10.64898/2026.09.15.751798v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Detecting cell segmentation errors using doublet methods
AU - A. Sarwar
AU - J. Gillis
PY - 2026
DA - 2026/09/15
PB - bioRxiv
DO - 10.64898/2026.09.15.751798
UR - https://www.biorxiv.org/content/10.64898/2026.09.15.751798v1
AB - In spatial transcriptomics, cell segmentation is used to draw boundaries around cells. Molecules located within a cell's boundary are assigned to it, making its gene expression profile dependent on segmentation accuracy. To identify potentially problematic cells, studies increasingly use doublet detection methods, a class of algorithms developed to recognize molecular admixture from two cells in scRNA-seq. To evaluate their suitability in spatial data, we model cell segmentation errors as a continuum of partial molecular admixture between neighboring cells, generated by varying the loss of a cell's own transcripts and the gain of transcripts from its neighbor. Evaluating 8 doublet methods across 16 spatial datasets, we characterize the conditions under which they perform well and identify their failure modes. Detection improves with increasing molecular admixture and transcriptional dissimilarity between neighboring cells, with cxds2, scDblFinder.score, and a simple baseline (unique_genes) performing best. These patterns are consistent across datasets spanning different gene panels, technological platforms, staining techniques, segmentation algorithms, and error frequencies. In unperturbed spatial data, elevated doublet scores localize to regions consistent with segmentation problems, suggesting that they can help prioritize cells or regions for inspection, segmentation refinement, or transcript reassignment. Our results establish when doublet methods can provide useful quality control signals for cell segmentation, supporting their use in spatial transcriptomics.
ER -