{
  "abstract": "Iterative fine-tuning on synthetic data causes \\emph{model collapse}: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we develop a new approach grounded in mathematical information theory: the non-parametric Kontoyiannis entropy rate estimator $h_k$, computed entirely from raw text via match-length statistics, with no model of any kind. We show that this is in fact a \\emph{superior} training-data filter on text-diversity metrics in a fully-synthetic, single-lineage fine-tuning setting. In a six-generation QLoRA collapse experiment on Llama-3.1-8B, logprob-based filtering (the most established model-access-requiring baseline) provides no significant text-diversity benefit on any metric ($p > 0.23$), whereas $h_k$-filtering yields $+42\\%$ unique trigrams, $+30\\%$ vocabulary, and $-19\\%$ repetition (all $p < 0.001$). We validate $h_k$ as a cross-domain entropy proxy ($β= 0.924$, $R^2 = 0.746$) and collapse detector ($ρ= +0.454$, $p < 0.0001$) across 4~domains, 2~temperatures, 2~generator--scorer model pairs, and 1{,}520 generated documents. Our results demonstrate that information theoretic approaches to collapse mitigation are efficient, and suggest new approaches for maintaining multi-agent diversity.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2610.01493",
  "author_total": 1,
  "authors": [
    {
      "name": "Lewis Mitchell"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/2zfh2ja3qzp4z.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2610.01493",
  "fulltext_url": null,
  "id": "2zfh2ja3qzp4z",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2610.01493",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "http://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.AI"
    ],
    [
      "arxiv",
      "cs.CL"
    ],
    [
      "arxiv",
      "cs.IT"
    ],
    [
      "arxiv",
      "math.IT"
    ],
    [
      "arxiv",
      "physics.data-an"
    ],
    [
      "arxiv",
      "stat.ML"
    ],
    [
      "openalex",
      "1702|Topic Modeling"
    ]
  ],
  "openalex": "W7216292525",
  "pdf_url": "https://arxiv.org/pdf/2610.01493",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-01",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/2zfh2ja3qzp4z.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:26:03Z",
      "native_id": "2610.01493",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2610.01493"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:stat.ML",
      "subfield": "statistics"
    },
    {
      "field": "physics",
      "origin": "arxiv:physics.data-an",
      "subfield": "computational"
    }
  ],
  "title": "No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse",
  "title_text": "No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse",
  "type": "preprint",
  "updated_at": "2026-10-30T12:26:14Z",
  "url": "https://research.draconitic.com/paper/2zfh2ja3qzp4z",
  "venue": "arXiv",
  "versions": []
}