Partial pooling beats synthetic respondents as estimators, not as covariates: a preregistered small-sample tournament
Abstract
Survey estimates for population segments, such as media use by sex and age, rest on few respondents per cell, and a growing literature offers synthetic respondents generated by large language models to fill the gap. That literature has not tested them against the small-area estimators a methodologist would otherwise use. We ran ten estimators, preregistered, on identical household-clustered splits of a Korean media panel across five calibration budgets and three holdout regimes. A multilevel partial-pooling model of the MRP family led every calibration-only estimator at every budget, matching at one hundred respondents what direct estimation reaches with roughly four times as many. On unobserved segments, information fixed as a per-segment prior was inert for synthetic panel and previous wave alike; attached as a bias correction it transferred, at the price of estimation noise. No registered synthetic advantage over MRP survived the paired decision rule. Entered instead as a covariate inside the multilevel model, an exploratory hybrid, the same panel improved on MRP on unobserved segments: the information survives where the registered constructions did not. Interval coverage stayed below nominal throughout and worsened with budget for partial pooling, contrary to prediction.
Related
- Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
- Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents
- Every tournament T of order at least 3 admits infinitely many T-irregular oriented graphs
- Multivariate Bernstein duality for tournament Wright--Fisher diffusions
- A Resource Allocation Tournament
BibTeX
@misc{kim2026partial,
title = {{Partial pooling beats synthetic respondents as estimators, not as covariates: a preregistered small-sample tournament}},
author = {Howard Kim},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.14526},
eprint = {2611.14526},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.14526},
}
RIS
TY - UNPB
TI - Partial pooling beats synthetic respondents as estimators, not as covariates: a preregistered small-sample tournament
AU - Howard Kim
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.14526
UR - https://arxiv.org/abs/2611.14526
AB - Survey estimates for population segments, such as media use by sex and age, rest on few respondents per cell, and a growing literature offers synthetic respondents generated by large language models to fill the gap. That literature has not tested them against the small-area estimators a methodologist would otherwise use. We ran ten estimators, preregistered, on identical household-clustered splits of a Korean media panel across five calibration budgets and three holdout regimes. A multilevel partial-pooling model of the MRP family led every calibration-only estimator at every budget, matching at one hundred respondents what direct estimation reaches with roughly four times as many. On unobserved segments, information fixed as a per-segment prior was inert for synthetic panel and previous wave alike; attached as a bias correction it transferred, at the price of estimation noise. No registered synthetic advantage over MRP survived the paired decision rule. Entered instead as a covariate inside the multilevel model, an exploratory hybrid, the same panel improved on MRP on unobserved segments: the information survives where the registered constructions did not. Interval coverage stayed below nominal throughout and worsened with budget for partial pooling, contrary to prediction.
ER -