Implicit Target Shift in Online Learning: Characterization and Correction
Abstract
Online learning from a stream of data is a defining feature of intelligence, yet modern machine learning systems often struggle in this setting, especially under distributional shift. To understand its basic properties, we study the relationship between online and offline learning in the context of kernel regression by deriving a closed-form expression for the function learned by online kernel regression. We reveal that online kernel regression is equivalent to offline regression with shifted, inaccurate target outputs. Conversely, we show that by compensating for this implicit target shift in the teaching signal through target correction, online kernel-based learning can provably learn the same predictor as its offline counterpart. We derive both a closed-form expression for this target correction and an iterative form that can be applied sequentially. Applying this framework to continual image classification tasks on domain-incremental Split CIFAR-10 and CORe50, we show that online stochastic gradient descent with iteratively corrected targets outperforms learning with the true targets. This work therefore provides a basic framework for analyzing and improving online learning in non-stationary environments from the perspective of implicit target shift.
Related
- Reference-State Bootstrap Correction and Kernel Characterization of Residual-Based Fit Indices in Structural Equation Models
- Estimating Implicit Regularization in Deep Learning
- Online Packet Scheduling with Deadlines and Learning
- Online tensor learning: Computational and statistical trade-offs, adaptivity and optimal regret
- Meta-LinEXP3: Online-within-Online Learning for Adversarial Linear Contextual Bandits
BibTeX
@misc{li2026implicit,
title = {{Implicit Target Shift in Online Learning: Characterization and Correction}},
author = {Ziyan Li and Naoki Hiratani},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.07886},
eprint = {2605.07886},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.07886},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Implicit Target Shift in Online Learning: Characterization and Correction
AU - Ziyan Li
AU - Naoki Hiratani
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2605.07886
UR - https://arxiv.org/abs/2605.07886
AB - Online learning from a stream of data is a defining feature of intelligence, yet modern machine learning systems often struggle in this setting, especially under distributional shift. To understand its basic properties, we study the relationship between online and offline learning in the context of kernel regression by deriving a closed-form expression for the function learned by online kernel regression. We reveal that online kernel regression is equivalent to offline regression with shifted, inaccurate target outputs. Conversely, we show that by compensating for this implicit target shift in the teaching signal through target correction, online kernel-based learning can provably learn the same predictor as its offline counterpart. We derive both a closed-form expression for this target correction and an iterative form that can be applied sequentially. Applying this framework to continual image classification tasks on domain-incremental Split CIFAR-10 and CORe50, we show that online stochastic gradient descent with iteratively corrected targets outperforms learning with the true targets. This work therefore provides a basic framework for analyzing and improving online learning in non-stationary environments from the perspective of implicit target shift.
ER -