Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches
Abstract
Conditional independence tests (CITs) test for conditional dependence between two random objects X and Y given a third random object Z. Existing CITs have limited applicability to high-dimensional data, especially multimodal data like text. However, we show that such tests are of interest for large language model (LLM) outputs, where we test whether an output X generated from a source text Z carries information about an attribute Y beyond Z itself. For this purpose, we propose embedded CITs (eCITs), which embed X and Z and apply an existing CIT to the resulting representations and to Y. We show that, provided the embedding of Z is sufficient, i.e. retains the information Z carries about either Y or the representation of X, the null hypothesis transfers from X and Z to their representations, so that a CIT valid for the embedded hypothesis is valid for the original one. We further give conditions for equivalence of the two hypotheses, and show that sufficiency weakens to mean sufficiency when the embedded test targets conditional mean independence. We propose a semi-synthetic simulation design to assess type I error (T1E) control and power of the eCITs for given embedding maps on a specific dataset and task, and use it to evaluate them on our application. Applying the eCITs to German Parliament speeches, we find for all combinations of embedding maps considered that the summaries of two LLMs contain information about the speaker's faction and gender beyond the speech they were generated from.
Related
- A Kernel-Based Test for Multi-Way Conditional Independence
- Conditional independence and graphical models for rankings
- Sequential Conditional Independence Testing with Machine Learning Models
- Causal Discovery beyond Conditional Independence: Identifiability, Algorithms, and Statistical Inference for Directed Acyclic and Cyclic Graphs
- On the Relaxation of Conditional Independence Assumption for Image Segmentation
BibTeX
@misc{simnacher2026embedded,
title = {{Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches}},
author = {Marco Simnacher and Georg Keilbar and Benjamin König and Christoph Lippert and Sonja Greven},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.00946},
eprint = {2609.00946},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.00946},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches
AU - Marco Simnacher
AU - Georg Keilbar
AU - Benjamin König
AU - Christoph Lippert
AU - Sonja Greven
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.00946
UR - https://arxiv.org/abs/2609.00946
AB - Conditional independence tests (CITs) test for conditional dependence between two random objects X and Y given a third random object Z. Existing CITs have limited applicability to high-dimensional data, especially multimodal data like text. However, we show that such tests are of interest for large language model (LLM) outputs, where we test whether an output X generated from a source text Z carries information about an attribute Y beyond Z itself. For this purpose, we propose embedded CITs (eCITs), which embed X and Z and apply an existing CIT to the resulting representations and to Y. We show that, provided the embedding of Z is sufficient, i.e. retains the information Z carries about either Y or the representation of X, the null hypothesis transfers from X and Z to their representations, so that a CIT valid for the embedded hypothesis is valid for the original one. We further give conditions for equivalence of the two hypotheses, and show that sufficiency weakens to mean sufficiency when the embedded test targets conditional mean independence. We propose a semi-synthetic simulation design to assess type I error (T1E) control and power of the eCITs for given embedding maps on a specific dataset and task, and use it to evaluate them on our application. Applying the eCITs to German Parliament speeches, we find for all combinations of embedding maps considered that the summaries of two LLMs contain information about the speaker's faction and gender beyond the speech they were generated from.
ER -