No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse
Abstract
Iterative fine-tuning on synthetic data causes \emph{model collapse}: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we develop a new approach grounded in mathematical information theory: the non-parametric Kontoyiannis entropy rate estimator hk, computed entirely from raw text via match-length statistics, with no model of any kind. We show that this is in fact a \emph{superior} training-data filter on text-diversity metrics in a fully-synthetic, single-lineage fine-tuning setting. In a six-generation QLoRA collapse experiment on Llama-3.1-8B, logprob-based filtering (the most established model-access-requiring baseline) provides no significant text-diversity benefit on any metric (p > 0.23), whereas hk-filtering yields +42% unique trigrams, +30% vocabulary, and −19% repetition (all p < 0.001). We validate hk as a cross-domain entropy proxy (β= 0.924, R2 = 0.746) and collapse detector (ρ= +0.454, p < 0.0001) across 4~domains, 2~temperatures, 2~generator--scorer model pairs, and 1{,}520 generated documents. Our results demonstrate that information theoretic approaches to collapse mitigation are efficient, and suggest new approaches for maintaining multi-agent diversity.
Related
- Model Collapse Is Avoidable Even as Real Data Vanishes: A Sharp Criterion for Iterative Gaussian Estimation
- Grad-div stabilized finite element Picard-Newton iterative methods for the Navier-Stokes equations
- Solving the Yamabe problem by an iterative method on a small Riemannian domain
- Iterative Methods for Solving New Generalized Absolute Value Equations
- Oscillation of a Second-Order Differential Equation as a Consequence of Iterative Monotonicities
BibTeX
@misc{mitchell2026model,
title = {{No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse}},
author = {Lewis Mitchell},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.01493},
eprint = {2610.01493},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.01493},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse
AU - Lewis Mitchell
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.01493
UR - https://arxiv.org/abs/2610.01493
AB - Iterative fine-tuning on synthetic data causes \emph{model collapse}: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we develop a new approach grounded in mathematical information theory: the non-parametric Kontoyiannis entropy rate estimator h_k, computed entirely from raw text via match-length statistics, with no model of any kind. We show that this is in fact a \emph{superior} training-data filter on text-diversity metrics in a fully-synthetic, single-lineage fine-tuning setting. In a six-generation QLoRA collapse experiment on Llama-3.1-8B, logprob-based filtering (the most established model-access-requiring baseline) provides no significant text-diversity benefit on any metric (p > 0.23), whereas h_k-filtering yields +42% unique trigrams, +30% vocabulary, and −19% repetition (all p < 0.001). We validate h_k as a cross-domain entropy proxy (β= 0.924, R^2 = 0.746) and collapse detector (ρ= +0.454, p < 0.0001) across 4~domains, 2~temperatures, 2~generator--scorer model pairs, and 1{,}520 generated documents. Our results demonstrate that information theoretic approaches to collapse mitigation are efficient, and suggest new approaches for maintaining multi-agent diversity.
ER -