SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics
Abstract
Evaluating text generation requires measuring how well the generated distribution matches the data distribution. For autoregressive models, this is done by the perplexity. Diffusion and flow-based language models can only provide a likelihood bound, whose tightness differs between model families. Sample-based substitutes such as generative perplexity with entropy do not consider the distribution fit. We propose SOL, a distance between text distributions. Each sequence is represented by the empirical measure of its hidden states under a fixed transformer and the distributions of these measures are compared by the double sliced Wasserstein distance. We prove that SOL is a metric if the transformer is injective. Experiments show that SOL detects distributional failures, recovers expected model trends, and provides stable sample-based estimates. We put forward SOL to fill the gap in the current evaluation protocol used for non auto-regressive models. As a first step we use SOL to re-evaluate a variety of models trained on OpenWebText.
Related
- λ-geodesic convexity of the squared sliced-Wasserstein distance
- Spherical Harmonic Sliced Wasserstein Displacement Interpolation for Acoustic Source and Reflection Density Modeling
- Fast rigid rotational alignment of voxelized 3-D density maps in sliced Wasserstein distance
- Sliced Wasserstein Barycenters: The Analysis Approach within the Barycentric Coding Model in the Wasserstein Space
- Minimax Rates for Estimating Powered Sliced Wasserstein Functionals
BibTeX
@misc{kornhardt2026measuring,
title = {{SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics}},
author = {Gregor Kornhardt and Moritz Piening and Jannis Chemseddine and Gabriele Steidl},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06513},
eprint = {2610.06513},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.06513},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics
AU - Gregor Kornhardt
AU - Moritz Piening
AU - Jannis Chemseddine
AU - Gabriele Steidl
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2610.06513
UR - https://arxiv.org/abs/2610.06513
AB - Evaluating text generation requires measuring how well the generated distribution matches the data distribution. For autoregressive models, this is done by the perplexity. Diffusion and flow-based language models can only provide a likelihood bound, whose tightness differs between model families. Sample-based substitutes such as generative perplexity with entropy do not consider the distribution fit. We propose SOL, a distance between text distributions. Each sequence is represented by the empirical measure of its hidden states under a fixed transformer and the distributions of these measures are compared by the double sliced Wasserstein distance. We prove that SOL is a metric if the transformer is injective. Experiments show that SOL detects distributional failures, recovers expected model trends, and provides stable sample-based estimates. We put forward SOL to fill the gap in the current evaluation protocol used for non auto-regressive models. As a first step we use SOL to re-evaluate a variety of models trained on OpenWebText.
ER -