Overcoming segmentation bottlenecks in brain tissue for spatial proteomic inference at population scale
- 1University of Pittsburgh
Abstract
In spatial proteomics across large cohorts, downstream estimates from tens of thousands of images depend on accurate cell segmentation. Yet pretrained segmentation models struggle with morphologically complex cell bodies in human brain tissue, and how domain adaptation affects downstream discovery remains unclear. Here, we introduce SPInfer, a workflow for spatial proteomic inference developed using 5,832 manually annotated cell bodies from 120 immunofluorescence images of the human prefrontal cortex. After benchmarking eight segmentation methods spanning classical, convolutional, and foundation models, we adapted Cellpose-SAM, raising cell segmentation F1 from 0.37 to 0.84 and recovering cell compositions concordant with matched single-nucleus RNA-seq and spatial transcriptomics. Across 14,765 images from 69 postmortem donors, identical Alzheimer's disease pathology models found no significant associations after false discovery rate correction using zero-shot masks, versus four using fine-tuned masks; three linked higher OLIG2-positive fractions to greater neurofibrillary tangle and plaque burden. Using the fine-tuned masks, a graph attention network over nuclear neighborhoods classified four cell types without marker information, and differential expression analysis linked cell morphology to gene expression. This provides a pipeline for fine-tuning foundation models to segment complex tissues, enabling biological discovery in spatial proteomics at the population scale.
Related
- Selfing-aware demographic inference highlights local population structure and bottlenecks in Hawaiian Caenorhabditis elegans
- Discovery of a Virulent Population of Meloidogyne graminicola Overcoming the Rice Resistance Gene MG1 and Population Genetic Analysis Based on mtCOI Sequences
- Agentic AI enables cell segmentation-free cell-type inference and cell-type-guided cell segmentation in spatial transcriptomics
- Proteomic machine-learning signatures associated with carotid plaque: A population-based hypothesis-generating study
- Correcting for ascertainment bias in the inference of population structure
BibTeX
@misc{you2026overcoming,
title = {{Overcoming segmentation bottlenecks in brain tissue for spatial proteomic inference at population scale}},
author = {L. You and J. L. Durbin and G. Meng and W. Stallaert and G. Li and S. Uttam and H. Feng and J. R. Glausier and J. Wang},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.22.761199},
url = {https://www.biorxiv.org/content/10.64898/2026.10.22.761199v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Overcoming segmentation bottlenecks in brain tissue for spatial proteomic inference at population scale
AU - L. You
AU - J. L. Durbin
AU - G. Meng
AU - W. Stallaert
AU - G. Li
AU - S. Uttam
AU - H. Feng
AU - J. R. Glausier
AU - J. Wang
PY - 2026
DA - 2026/10/22
PB - bioRxiv
DO - 10.64898/2026.10.22.761199
UR - https://www.biorxiv.org/content/10.64898/2026.10.22.761199v1
AB - In spatial proteomics across large cohorts, downstream estimates from tens of thousands of images depend on accurate cell segmentation. Yet pretrained segmentation models struggle with morphologically complex cell bodies in human brain tissue, and how domain adaptation affects downstream discovery remains unclear. Here, we introduce SPInfer, a workflow for spatial proteomic inference developed using 5,832 manually annotated cell bodies from 120 immunofluorescence images of the human prefrontal cortex. After benchmarking eight segmentation methods spanning classical, convolutional, and foundation models, we adapted Cellpose-SAM, raising cell segmentation F1 from 0.37 to 0.84 and recovering cell compositions concordant with matched single-nucleus RNA-seq and spatial transcriptomics. Across 14,765 images from 69 postmortem donors, identical Alzheimer's disease pathology models found no significant associations after false discovery rate correction using zero-shot masks, versus four using fine-tuned masks; three linked higher OLIG2-positive fractions to greater neurofibrillary tangle and plaque burden. Using the fine-tuned masks, a graph attention network over nuclear neighborhoods classified four cell types without marker information, and differential expression analysis linked cell morphology to gene expression. This provides a pipeline for fine-tuning foundation models to segment complex tissues, enabling biological discovery in spatial proteomics at the population scale.
ER -