Graph-Based Testing of Conditional Mean Independence with Applications to Large Language Model Evaluation
Abstract
We study whether a predictor X provides additional information for predicting a scalar response Y after adjustment for other variables Z, formalized by testing H0:E(Y| X,Z)=E(Y| Z). Existing methods often require estimating nuisance functions involving X, which can be difficult for high-dimensional or structured predictors. We propose a graph-based test that extends the edge count principle to covariate adjustment. It represents X through pairwise dissimilarities, estimates only the baseline regression E(Y| Z), and aggregates products of held-out residuals across graph edges. We develop sample-splitting and two-fold cross-fitting versions. Under regularity conditions, both have standard normal null limits, yielding analytic p-values, and are consistent under graph-smooth and graph-rough alternatives. Simulations with vector, image, and network predictors show good size control and power advantages over competing procedures, with cross-fitting improving power across the main alternatives. Applications examine human quality ratings, mathematical error labels, and LLM response preferences. Our method detects associations with quality ratings and an LLM judge's preferences after adjustment for text length and other recorded characteristics.
Related
BibTeX
@misc{liu2026graphbased,
title = {{Graph-Based Testing of Conditional Mean Independence with Applications to Large Language Model Evaluation}},
author = {Mingshuo Liu and Qihua Zhu and Jingchi Liao and Lu Tian and Doudou Zhou},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.01773},
eprint = {2611.01773},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.01773},
}
RIS
TY - UNPB
TI - Graph-Based Testing of Conditional Mean Independence with Applications to Large Language Model Evaluation
AU - Mingshuo Liu
AU - Qihua Zhu
AU - Jingchi Liao
AU - Lu Tian
AU - Doudou Zhou
PY - 2026
DA - 2026/11/01
PB - arXiv
DO - 10.48550/arxiv.2611.01773
UR - https://arxiv.org/abs/2611.01773
AB - We study whether a predictor X provides additional information for predicting a scalar response Y after adjustment for other variables Z, formalized by testing H_0:E(Y| X,Z)=E(Y| Z). Existing methods often require estimating nuisance functions involving X, which can be difficult for high-dimensional or structured predictors. We propose a graph-based test that extends the edge count principle to covariate adjustment. It represents X through pairwise dissimilarities, estimates only the baseline regression E(Y| Z), and aggregates products of held-out residuals across graph edges. We develop sample-splitting and two-fold cross-fitting versions. Under regularity conditions, both have standard normal null limits, yielding analytic p-values, and are consistent under graph-smooth and graph-rough alternatives. Simulations with vector, image, and network predictors show good size control and power advantages over competing procedures, with cross-fitting improving power across the main alternatives. Applications examine human quality ratings, mathematical error labels, and LLM response preferences. Our method detects associations with quality ratings and an LLM judge's preferences after adjustment for text length and other recorded characteristics.
ER -