Contributions of Site Heterogeneity in Multi-site Brain-Phenotype Predictions
- 1Vanderbilt University Medical Center
Abstract
Machine learning studies of brain-phenotype associations typically summarize accuracy with a single Pearson correlation, yet in harmonized multi-site data this metric is shaped by training-set size, feature and outcome variability, and true effect heterogeneity. We analyzed structural MRI data from the Reproducible Brain Charts consortium. After preprocessing and quality control, 3486 participants from 9 sites were retained. Three training set compositions were used to understand contributions of site heterogeneity after harmonization: within-site (Local), size-downsampled multi-site (Multi-Down), and full multi-site (Multi-Full). Another site-agnostic strategy (Overall) pooled all sites into a single dataset. All strategies included the target site in the training and testing samples. These analyses separate the contribution of the site effect from that of sample size. Between-site heterogeneity was extensive (age I^2 {approx} 95%). The overall correlation of age prediction (r = 0.73) exceeded the within-site meta-analytic estimate (Multi-Full r = 0.63). The improvement of correlation in the pooled dataset was driven by a mechanical inflation from the larger outcome variance. At matched sample size, within-site training outperformed harmonized multi-site training. Increasing the multi-site training sample size did not yield a significant advantage over within-site training. The apparent advantage of cross-dataset training thus appears to reflect training-set size rather than a benefit of multi-site composition per se. The same pattern generally held for the weaker brain-p-factor signals. Together, these results indicate that between-site heterogeneity in machine learning brain-phenotype prediction persists after harmonization. We recommend reporting site-level prediction accuracy and feature dispersion and standard deviation of the outcome variable alongside any pooled estimate to improve inter-site and inter-study comparison of machine learning prediction accuracy.
Related
- Learning from Error Predictions Rather Than Prediction Errors: A Perspective on Self-Supervised Learning in the Brain
- The Mottling Phenotype in Chickens Shows Genetic Heterogeneity and Is Caused by Mutations at the EDNRB2 Locus
- Clinical Heterogeneity in Myotonic Dystrophy Type 1: Identifying Phenotype Categories Using Unsupervised Clustering.
- Information processing-like phenotype of human brain organoids
- Quantifying Uncertainty in Brain Age Predictions via Conformal Prediction
BibTeX
@misc{chen2026contributions,
title = {{Contributions of Site Heterogeneity in Multi-site Brain-Phenotype Predictions}},
author = {G. Chen and M. Jones and I. Gadiyar and T. D. Satterthwaite and A. Chen and S. Vandekar},
howpublished = {bioRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.01.763540},
url = {https://www.biorxiv.org/content/10.64898/2026.11.01.763540v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Contributions of Site Heterogeneity in Multi-site Brain-Phenotype Predictions
AU - G. Chen
AU - M. Jones
AU - I. Gadiyar
AU - T. D. Satterthwaite
AU - A. Chen
AU - S. Vandekar
PY - 2026
DA - 2026/11/01
PB - bioRxiv
DO - 10.64898/2026.11.01.763540
UR - https://www.biorxiv.org/content/10.64898/2026.11.01.763540v1
AB - Machine learning studies of brain-phenotype associations typically summarize accuracy with a single Pearson correlation, yet in harmonized multi-site data this metric is shaped by training-set size, feature and outcome variability, and true effect heterogeneity. We analyzed structural MRI data from the Reproducible Brain Charts consortium. After preprocessing and quality control, 3486 participants from 9 sites were retained. Three training set compositions were used to understand contributions of site heterogeneity after harmonization: within-site (Local), size-downsampled multi-site (Multi-Down), and full multi-site (Multi-Full). Another site-agnostic strategy (Overall) pooled all sites into a single dataset. All strategies included the target site in the training and testing samples. These analyses separate the contribution of the site effect from that of sample size. Between-site heterogeneity was extensive (age I^2 {approx} 95%). The overall correlation of age prediction (r = 0.73) exceeded the within-site meta-analytic estimate (Multi-Full r = 0.63). The improvement of correlation in the pooled dataset was driven by a mechanical inflation from the larger outcome variance. At matched sample size, within-site training outperformed harmonized multi-site training. Increasing the multi-site training sample size did not yield a significant advantage over within-site training. The apparent advantage of cross-dataset training thus appears to reflect training-set size rather than a benefit of multi-site composition per se. The same pattern generally held for the weaker brain-p-factor signals. Together, these results indicate that between-site heterogeneity in machine learning brain-phenotype prediction persists after harmonization. We recommend reporting site-level prediction accuracy and feature dispersion and standard deviation of the outcome variable alongside any pooled estimate to improve inter-site and inter-study comparison of machine learning prediction accuracy.
ER -