Embedding-Bias in Conditional Independence Testing
Abstract
To test conditional independence of X and Y given a text or an image Z, one conditions on an embedding ψ(Z) in place of Z. The embedded test is valid if Z is independent of X or of Y given ψ(Z), which cannot be confirmed from data, and when this fails, the rejection probability under the null hypothesis can tend to one. We study this failure, and show that focusing on a specific form of dependence relaxes what the embedding must retain. For a residual correlation test inspired by the Generalised Covariance Measure, validity only requires that the parts of 𝔼[X | Z] and 𝔼[Y | Z] missed by 𝔼[X | ψ(Z)] and 𝔼[Y | ψ(Z)] are uncorrelated. Otherwise, we treat the discarded information as an omitted variable. Under the null hypothesis, the bias equals the absolute correlation of the missed parts times the geometric mean of two partial R2 values. This identity yields a robust test valid under a declared tolerance for the geometric mean, which, like a sensitivity parameter, is not identified from the data. On synthetic data and text embeddings, the robust test holds its level approximately. On text generated by a language model, under an exact null hypothesis, every embedding, even the generator's own states, biases the embedded test.
Related
- Sequential Conditional Independence Testing with Machine Learning Models
- Graph-Based Testing of Conditional Mean Independence with Applications to Large Language Model Evaluation
- Conditional independence testing through marginal permutation
- Conditional Independence Testing in Time Series
- Domain-Adapted Diffusion Models for Conditional Independence Testing
BibTeX
@misc{thams2026embeddingbias,
title = {{Embedding-Bias in Conditional Independence Testing}},
author = {Nikolaj Thams and Anton Rask Lundborg},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.11584},
eprint = {2610.11584},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.11584},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Embedding-Bias in Conditional Independence Testing
AU - Nikolaj Thams
AU - Anton Rask Lundborg
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.11584
UR - https://arxiv.org/abs/2610.11584
AB - To test conditional independence of X and Y given a text or an image Z, one conditions on an embedding ψ(Z) in place of Z. The embedded test is valid if Z is independent of X or of Y given ψ(Z), which cannot be confirmed from data, and when this fails, the rejection probability under the null hypothesis can tend to one. We study this failure, and show that focusing on a specific form of dependence relaxes what the embedding must retain. For a residual correlation test inspired by the Generalised Covariance Measure, validity only requires that the parts of 𝔼[X | Z] and 𝔼[Y | Z] missed by 𝔼[X | ψ(Z)] and 𝔼[Y | ψ(Z)] are uncorrelated. Otherwise, we treat the discarded information as an omitted variable. Under the null hypothesis, the bias equals the absolute correlation of the missed parts times the geometric mean of two partial R^2 values. This identity yields a robust test valid under a declared tolerance for the geometric mean, which, like a sensitivity parameter, is not identified from the data. On synthetic data and text embeddings, the robust test holds its level approximately. On text generated by a language model, under an exact null hypothesis, every embedding, even the generator's own states, biases the embedded test.
ER -