Cohort Effects, Not Treatment Response, Dominate Public Immunotherapy Cohorts: Evidence from Microbiome and Tumor Genomic Data
Abstract
Researchers often pool public gut microbiome datasets from immunotherapy studies to get around small sample sizes, assuming that adding cohorts adds statistical power faster than it adds differences between studies. We test that assumption. We processed four public shotgun metagenomic cohorts of patients receiving immune checkpoint blockade through one pipeline and pooled them step by step, from 39 to 283 patients. Because the PERMANOVA effect size R^2 is above zero even when there is no real effect, we compared every result with its exact chance expectation. Measured as raw variance explained, cohort membership accounted for 8 to 27 times more variance than treatment response, although much of that ratio comes from sample size alone. At every pooling step, treatment response explained no more variance than chance (within 0.2 percentage points) and was never significant, while cohort membership explained 6.0 to 9.5 percentage points beyond chance (p = 0.001 each time). Classification stayed at chance across three model types, none of six leak-free batch-correction strategies recovered signal, and no genus reached significance in a random-effects meta-analysis. The largest single cohort (n = 165) was also non-significant, although simulation shows it had at least 90% power to detect an effect at its own 95% upper bound; within it, recruiting site alone explained 7.0 percentage points beyond chance. In three tumor genomic cohorts (n = 223), a small but real response effect sat next to a cohort effect about eighteen times larger, so the problem is not specific to microbiome data. We release batch_detector, an open tool that reports chance-corrected cohort and response effects before pooling, and test it on an independent colorectal cancer meta-analysis.
Related
- The cervicovaginal microbiome as a candidate correlate and potential predictor of HPV clearance and treatment response after HSIL therapy: a scoping review
- Immunotherapy restores therapeutic efficacy of fecal microbiome transplants to treat Clostridioides difficile
- Long non-coding RNA prompts immune-related adverse events risk and immunotherapy response in pan-cancer
- Provisional extracellular matrix signaling network regulates tumor lymphoneogenesis and response to immunotherapy
- Trained immunity in cancer immunotherapy and vaccine development: mechanisms, evidence gaps, and translational opportunities
BibTeX
@misc{garg2026cohort,
title = {{Cohort Effects, Not Treatment Response, Dominate Public Immunotherapy Cohorts: Evidence from Microbiome and Tumor Genomic Data}},
author = {Amey Garg},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.33010},
eprint = {2610.33010},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.33010},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Cohort Effects, Not Treatment Response, Dominate Public Immunotherapy Cohorts: Evidence from Microbiome and Tumor Genomic Data
AU - Amey Garg
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.33010
UR - https://arxiv.org/abs/2610.33010
AB - Researchers often pool public gut microbiome datasets from immunotherapy studies to get around small sample sizes, assuming that adding cohorts adds statistical power faster than it adds differences between studies. We test that assumption. We processed four public shotgun metagenomic cohorts of patients receiving immune checkpoint blockade through one pipeline and pooled them step by step, from 39 to 283 patients. Because the PERMANOVA effect size R^2 is above zero even when there is no real effect, we compared every result with its exact chance expectation. Measured as raw variance explained, cohort membership accounted for 8 to 27 times more variance than treatment response, although much of that ratio comes from sample size alone. At every pooling step, treatment response explained no more variance than chance (within 0.2 percentage points) and was never significant, while cohort membership explained 6.0 to 9.5 percentage points beyond chance (p = 0.001 each time). Classification stayed at chance across three model types, none of six leak-free batch-correction strategies recovered signal, and no genus reached significance in a random-effects meta-analysis. The largest single cohort (n = 165) was also non-significant, although simulation shows it had at least 90% power to detect an effect at its own 95% upper bound; within it, recruiting site alone explained 7.0 percentage points beyond chance. In three tumor genomic cohorts (n = 223), a small but real response effect sat next to a cohort effect about eighteen times larger, so the problem is not specific to microbiome data. We release batch_detector, an open tool that reports chance-corrected cohort and response effects before pooling, and test it on an independent colorectal cancer meta-analysis.
ER -