An Identifiability Theory of Masked Prediction: Mode Blindness and Mask Schedules
Abstract
Masked prediction learns to infer missing variables from visible context selected by a mask schedule. When does small excess risk guarantee recovery of the true joint distribution? We study this question on finite product spaces under masked-block log loss, with conditionals induced by a single joint distribution. To quantify recovery, we introduce an ε-identifiability modulus measuring the worst-case error among joint distributions with population excess risk at most ε. For data with separated modes pinned down by sufficiently large visible contexts, schedules that always retain such contexts can permit nonvanishing mode-weight errors while incurring exponentially small excess risk. An information decomposition explains why: the loss captures mode-weight mismatch only where the visible context leaves the mode uncertain. We prove two-sided bounds showing that, over a fixed range of mode weights, sensitivity to mode reweighting is governed by residual mode uncertainty averaged over the mask schedule. Assigning schedule mass to low-visibility masks that retain this uncertainty yields recovery bounds within the reweighting family. Beyond this family, positive full-mask probability characterizes uniform control of joint KL divergence by excess risk. Exact calculations and controlled gradient optimization validate these predictions.
Related
- Serial Depth of Adaptive Parallel Sampling: Lower Bounds and Schedules for Masked Diffusion Models
- The hidden advantage of mask resampling: a theory of masked autoencoders
- Random Width and Brightness: Polyhedral Density Theory, Reconstruction, and Gaussian Identifiability
- The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexity
- Accelerated Concatenation of Stepsize Schedules for Smooth Strongly Convex Gradient Descent
BibTeX
@misc{cai2026identifiability,
title = {{An Identifiability Theory of Masked Prediction: Mode Blindness and Mask Schedules}},
author = {Yichao Cai and Javen Qinfeng Shi},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.01383},
eprint = {2608.01383},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.01383},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - An Identifiability Theory of Masked Prediction: Mode Blindness and Mask Schedules
AU - Yichao Cai
AU - Javen Qinfeng Shi
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2608.01383
UR - https://arxiv.org/abs/2608.01383
AB - Masked prediction learns to infer missing variables from visible context selected by a mask schedule. When does small excess risk guarantee recovery of the true joint distribution? We study this question on finite product spaces under masked-block log loss, with conditionals induced by a single joint distribution. To quantify recovery, we introduce an ε-identifiability modulus measuring the worst-case error among joint distributions with population excess risk at most ε. For data with separated modes pinned down by sufficiently large visible contexts, schedules that always retain such contexts can permit nonvanishing mode-weight errors while incurring exponentially small excess risk. An information decomposition explains why: the loss captures mode-weight mismatch only where the visible context leaves the mode uncertain. We prove two-sided bounds showing that, over a fixed range of mode weights, sensitivity to mode reweighting is governed by residual mode uncertainty averaged over the mask schedule. Assigning schedule mass to low-visibility masks that retain this uncertainty yields recovery bounds within the reweighting family. Beyond this family, positive full-mask probability characterizes uniform control of joint KL divergence by excess risk. Exact calculations and controlled gradient optimization validate these predictions.
ER -