Peer-reviewedCC BYNature Methods
Beyond benchmarking: an expert-guided consensus approach to spatially aware clustering
40 affiliations
- 1Biomedical Data Science Center, Centre Hospitalier Universitaire Vaudois, Lausanne, Switzerland
- 2Biomedical Data Science Center, Centre Hospitalier Universitaire Vaudois, Rue du Bugnon 2, 0 Lausanne, Switzerland
- 3Delft Bioinformatics Lab, Delft University of Technology, Delft, the Netherlands
- 4Department of Human Genetics, Leiden University Medical Center, Leiden, the Netherlands
- 5Delft Bioinformatics Lab, Delft University of Technology, Van Mourik Broekmanweg 6, 2628 XE Delft, Netherlands
- 6Department of Human Genetics, Leiden University Medical Center, Einthovenweg 0, 333 ZC Leiden, Netherlands
- 7Department of Molecular Life Sciences and SIB Swiss Institute of Bioinformatics, University of Zurich, Zurich, Switzerland
- 8Department of Molecular Life Sciences and SIB Swiss Institute of Bioinformatics, University of Zurich, Winterthurerstrasse 190, 8057 Zurich, Switzerland
- 9Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Center of Digital, Berlin, Germany
- 10Berlin Institute of Health at Charité – Universitätsmedizin Berlin, Center of Digital Health, Charitéplatz 1, 10117 Berlin, Germany
- 11Department of Nephrology, Rheumatology, and Clinical Immunology, University Hospital RWTH Aachen, Aachen, Germany
- 12Department of Nephrology, Rheumatology, and Clinical Immunology, University Hospital RWTH Aachen, Pauwelsstraße 30, Aachen, Germany
- 13Allen Institute for Brain Science, Seattle, WA, USA
- 14Allen Institute for Brain Science, 615 Westlake Ave N, Seattle, WA, USA 98109
- 15DTU Health Tech, Technical University of Denmark, Ørsteds Plads, Kongens Lyngby, Denmark
- 16LEO Foundation Skin Immunology Research Center, Department of Immunology and Microbiology, University of Copenhagen, Copenhagen, Denmark
- 17German Cancer Research Center (DKFZ), Division of Computational Genomics and Systems Genetics, Heidelberg, Germany
- 18The German Human Genome-Phenome Archive, Heidelberg, Germany
- 19German Cancer Research Center (DKFZ), Division of Computational Genomics and Systems Genetics, Im Neuenheimer Feld 280, 69120 Heidelberg, Germany
- 20Division of Cell Biology, Histology and Embryology, Gottfried Schatz Research Center, Medical University of Graz, Graz, Austria
- 21Diagnostic and Research Institute of Pathology, Medical University of Graz, Graz, Austria
- 22Department of Internal Medicine, Division of Nephrology, University of Michigan, Ann Arbor, MI, USA
- 23Computational Biology Department, Carnegie Mellon University, Pittsburgh, PA, USA
- 24CIMA Universidad de Navarra, Pamplona, Spain
- 25Department of Biomedical Informatics, College of Medicine, Ohio State University, Columbus, OH, USA
- 26Allen Institute for Neural Dynamics, Seattle, WA, USA
- 27Genome Institute of Singapore, Agency for Science, Technology and Research (A*STAR), Singapore, Singapore
- 28Nature Genetics, Berlin, Germany
- 29Institute of Applied Biosciences, Centre for Research and Technology Hellas, Thessaloniki, Greece
- 30Biozentrum, University of Basel, Basel, Switzerland
- 31SIB Swiss Institute of Bioinformatics, Quartier Sorge Batiment Amphipole, Lausanne, Switzerland
- 32Helmholtz Zentrum München, Institute of Computational Biology, Neuherberg, Germany
- 33CSEM, Allschwil, Switzerland
- 34ETHZ, BMIC - Biomedical Image Computing lab, Zurich, Switzerland
- 35OMAPiX GmbH Raiffeisenstraße 3, Langenfeld, Germany
- 36Technical University of Munich, School of Medicine, Institute of Virology, Munich, Germany
- 37University of Michigan, Undergraduate Research Opportunity Program, Ann Arbor, MI, USA
- 38Department of Informatics and Chemistry, Faculty of Chemical Technology, University of Chemistry and Technology, Prague, Czech Republic
- 39Laboratory of Genomics and Bioinformatics, Institute of Molecular Genetics of the Czech Academy of Sciences, Prague, Czech Republic
- 40Department of Biology and Biological Engineering, Chalmers University of Technology, Gothenburg, Sweden
Abstract
Spatial omics technologies have revolutionized the study of tissue architecture and cellular heterogeneity by integrating molecular profiles with spatial localization. In spatially resolved transcriptomics, delineating higher-order anatomical structures is critical for understanding how cellular organization affects tissue and organ function. Since 2020, more than 50 spatially aware clustering (SAC) methods have been developed for this purpose. However, the reliability of current benchmarks is undermined by their narrow focus on Visium and brain tissue datasets, as well as incorrect interpretation of manual annotation as ground truth. Here, we present SACCELERATOR, a community-driven, extensible framework that standardizes data formatting, method integration, and metric evaluation, and is designed to rapidly incorporate new methods and datasets. SACCELERATOR currently includes 22 SAC methods applied to 15 datasets spanning 9 technologies and diverse tissue types. Our analysis revealed substantial limitations in the generalizability and reproducibility of SAC methods across tissues and platforms. We also demonstrate that anatomical labels commonly used as ground truths are often biased, potentially error-prone, and, in some cases, unsuitable for benchmarking efforts. Rather than scoring and comparing methods, we propose a consensus-guided workflow that aggregates clustering results to generate consensus representations. Descriptive spatial metrics highlight areas of high entropy where method disagreement is highest, enabling targeted feedback for tissue experts. Applied to brain and cancer datasets, this approach uncovered biologically meaningful patterns overlooked by individual methods and manual annotations. Our results underscore the need for iterative, expert-in-the-loop analysis and reveal that traditional evaluation metrics do not always capture the subjective qualities of results. By improving tissue annotation and addressing key benchmarking limitations, SACCELERATOR provides a robust foundation for advancing spatial omics research.
Related
- Bayesian Learning of Distance Metrics Beyond RMSD for Biomolecule Alignment, Clustering, and Domain Identification.
- DeepEnzyKat: Turnover number prediction through enzyme-substrate interaction modeling and adaptive expert ensembles
- DICD-IE: an expert-annotated benchmark for evaluating pretrained biomedical language models in dental implant literature and clinical knowledge extraction
- Establishing standardized evaluation protocols for peptide-target interaction prediction: the PTI-TAPE consensus framework eDelphi
- Optimization of the NFκB-DNA ensemble leads to high affinity binding of partial consensus sites
BibTeX
@article{sun2026beyond,
title = {{Beyond benchmarking: an expert-guided consensus approach to spatially aware clustering}},
author = {Jieran Sun and Kirti Biharie and Peiying Cai and Niklas Müller‐Bötticher and Paul Kießling and Meghan A. Turner and Søren Helweg Dam and Florian Heyl and Sarusan Kathirchelvan and Martin Emons and Samuel Gunz and Sven Olaf Twardziok and Amin El Heliebi and Martin Zacharias and SpaceHack 2.0 participants and Søren Helweg Dam and Fadhl Alakwaa and Shahul Alam and Maria Calleja and Yuzhou Chang and Thomas Chartrand and Nigel Chou and Estella Y. Dong and Michael Fletcher and George Gavriilidis and Alexander Kanitz and Sameesh Kher and Louis Benedikt Kuemmerle and Francesca A. Luongo and Qirong Mao and Giorgia Moranzoni and Mar M. Moreno and Anastasiia Okhtienko and Lena Perry and Lucie Pfeiferová and Daryna Pikulska and Shyam Prabhakar and Rasool Saghaleyni and Zaira Seferbekova and Vipul Singhal and Divya Sitani and Charlotte Soneson and Sebastian Tiesmeyer and Marco Varrone and Siao-Han Wong and Liya Zaygerman and Teresa Zulueta-Coarasa and Roland Eils and Marcel J. T. Reinders and Raphaël Gottardo and Christoph Kuppe and Brian Long and Ahmed Essam Mahfouz and Mark D. Robinson and Naveed Ishaque},
journal = {Nature Methods},
year = {2026},
month = aug,
doi = {10.1038/s41592-026-03194-8},
url = {https://doi.org/10.1038/s41592-026-03194-8},
note = {CC BY},
}
RIS
TY - JOUR
TI - Beyond benchmarking: an expert-guided consensus approach to spatially aware clustering
AU - Jieran Sun
AU - Kirti Biharie
AU - Peiying Cai
AU - Niklas Müller‐Bötticher
AU - Paul Kießling
AU - Meghan A. Turner
AU - Søren Helweg Dam
AU - Florian Heyl
AU - Sarusan Kathirchelvan
AU - Martin Emons
AU - Samuel Gunz
AU - Sven Olaf Twardziok
AU - Amin El Heliebi
AU - Martin Zacharias
AU - SpaceHack 2.0 participants
AU - Søren Helweg Dam
AU - Fadhl Alakwaa
AU - Shahul Alam
AU - Maria Calleja
AU - Yuzhou Chang
AU - Thomas Chartrand
AU - Nigel Chou
AU - Estella Y. Dong
AU - Michael Fletcher
AU - George Gavriilidis
AU - Alexander Kanitz
AU - Sameesh Kher
AU - Louis Benedikt Kuemmerle
AU - Francesca A. Luongo
AU - Qirong Mao
AU - Giorgia Moranzoni
AU - Mar M. Moreno
AU - Anastasiia Okhtienko
AU - Lena Perry
AU - Lucie Pfeiferová
AU - Daryna Pikulska
AU - Shyam Prabhakar
AU - Rasool Saghaleyni
AU - Zaira Seferbekova
AU - Vipul Singhal
AU - Divya Sitani
AU - Charlotte Soneson
AU - Sebastian Tiesmeyer
AU - Marco Varrone
AU - Siao-Han Wong
AU - Liya Zaygerman
AU - Teresa Zulueta-Coarasa
AU - Roland Eils
AU - Marcel J. T. Reinders
AU - Raphaël Gottardo
AU - Christoph Kuppe
AU - Brian Long
AU - Ahmed Essam Mahfouz
AU - Mark D. Robinson
AU - Naveed Ishaque
PY - 2026
DA - 2026/08/24
JO - Nature Methods
DO - 10.1038/s41592-026-03194-8
UR - https://doi.org/10.1038/s41592-026-03194-8
AB - Spatial omics technologies have revolutionized the study of tissue architecture and cellular heterogeneity by integrating molecular profiles with spatial localization. In spatially resolved transcriptomics, delineating higher-order anatomical structures is critical for understanding how cellular organization affects tissue and organ function. Since 2020, more than 50 spatially aware clustering (SAC) methods have been developed for this purpose. However, the reliability of current benchmarks is undermined by their narrow focus on Visium and brain tissue datasets, as well as incorrect interpretation of manual annotation as ground truth. Here, we present SACCELERATOR, a community-driven, extensible framework that standardizes data formatting, method integration, and metric evaluation, and is designed to rapidly incorporate new methods and datasets. SACCELERATOR currently includes 22 SAC methods applied to 15 datasets spanning 9 technologies and diverse tissue types. Our analysis revealed substantial limitations in the generalizability and reproducibility of SAC methods across tissues and platforms. We also demonstrate that anatomical labels commonly used as ground truths are often biased, potentially error-prone, and, in some cases, unsuitable for benchmarking efforts. Rather than scoring and comparing methods, we propose a consensus-guided workflow that aggregates clustering results to generate consensus representations. Descriptive spatial metrics highlight areas of high entropy where method disagreement is highest, enabling targeted feedback for tissue experts. Applied to brain and cancer datasets, this approach uncovered biologically meaningful patterns overlooked by individual methods and manual annotations. Our results underscore the need for iterative, expert-in-the-loop analysis and reveal that traditional evaluation metrics do not always capture the subjective qualities of results. By improving tissue annotation and addressing key benchmarking limitations, SACCELERATOR provides a robust foundation for advancing spatial omics research.
SN - 1548-7091
SN - 1548-7105
ER -