Optimal Weighting of Training Data in Adaptive Coding
Abstract
Adaptive coders are commonly primed with training data, turned into context tables or shipped dictionaries. The training data generally do not match the source being encoded, and the coder must decide how much to trust them. We make that trust a design variable: a Krichevsky--Trofimov (KT) estimator over an m-ary alphabet whose training counts are scaled by ξ∈[0,1]. For a training sequence of length ℓ at Kullback--Leibler divergence D nats per symbol from the message source, the redundancy-minimizing weight is ξ*=d/(2ℓ D+d), d=m−1. The effective training length ξ*ℓ follows a harmonic law: any mismatch caps the usable training information at d/(2D) symbols, however much was collected. Three implementable selectors supply the unknown D: offline, from the spread of the training set; by a plug-in loop, from the decoded prefix; and by a twice-universal mixture, with no estimation at all. On the ten English texts of the Calgary and Canterbury corpora, weighting removes up to 22.7% of the redundancy of the classical KT code, the ξ=0 endpoint, and beats both endpoints on every file. An open-source implementation is provided.
Related
- Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation
- Co-Adaptive Entropy Modeling and Rate-Distortion-Optimal Bit Allocation for Perceptual Audio Coding
- Keeping Score: Adaptive, Tuning-Free Loss Weighting for Score-Augmented Neural Ratio Estimation
- Token-Level Attribution of Multi-Mode LLM Watermarks: Optimal Rule and Belief Weighting
- Adaptive spline-based weighting functions for blended survival curve extrapolation
BibTeX
@misc{reznik2026optimal,
title = {{Optimal Weighting of Training Data in Adaptive Coding}},
author = {Yuriy Reznik},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.27783},
eprint = {2609.27783},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.27783},
}
RIS
TY - UNPB
TI - Optimal Weighting of Training Data in Adaptive Coding
AU - Yuriy Reznik
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.27783
UR - https://arxiv.org/abs/2609.27783
AB - Adaptive coders are commonly primed with training data, turned into context tables or shipped dictionaries. The training data generally do not match the source being encoded, and the coder must decide how much to trust them. We make that trust a design variable: a Krichevsky--Trofimov (KT) estimator over an m-ary alphabet whose training counts are scaled by ξ∈[0,1]. For a training sequence of length ℓ at Kullback--Leibler divergence D nats per symbol from the message source, the redundancy-minimizing weight is ξ^*=d/(2ℓ D+d), d=m−1. The effective training length ξ^*ℓ follows a harmonic law: any mismatch caps the usable training information at d/(2D) symbols, however much was collected. Three implementable selectors supply the unknown D: offline, from the spread of the training set; by a plug-in loop, from the decoded prefix; and by a twice-universal mixture, with no estimation at all. On the ten English texts of the Calgary and Canterbury corpora, weighting removes up to 22.7% of the redundancy of the classical KT code, the ξ=0 endpoint, and beats both endpoints on every file. An open-source implementation is provided.
ER -