Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention
Abstract
Estimating a decision threshold requires locating observations near an unknown boundary. We study how gradient-based pretraining learns this statistical rule in a two-parameter softmax-attention model with a fixed feature and inequality direction. Pretraining uses labeled contexts and their true thresholds; a fresh threshold must be inferred from context alone. Under a large-resolution initialization, constant-step gradient descent on m tasks with n examples each produces a frozen estimator with error Õ((m∧ n)−1+N−1) for each fixed interior threshold and every fresh-context size N. The two terms separate finite-pretraining accuracy from fresh-context localization. The mechanism is coordinated parameter divergence: population training calibrates the relative label and feature scores, then increases the attention scale as t1/4, giving population threshold error O(t−1/4). To transfer this mechanism to a fixed finite corpus, we control gradient errors relative to the shrinking directions of progress at successive parameter scales. This certifies a growing training interval without requiring long-time tracking of the population trajectory. We also identify the boundary limitation of the one-head model and explain statistically what a reflected symmetrization could achieve.
Related
- High-Dimensional Learning Dynamics of Attention-Indexed Models
- Beyond Multiplicity Correction: Sharp Retrieval Thresholds for Attention with Correlated Distractors
- What can linear attention learn from nonlinear teachers in-context?
- In-context classification: Calibration and noise in linear attention with softmax readout
- Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms
BibTeX
@misc{le2026learning,
title = {{Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention}},
author = {Hong Ha Le and Jackie Lok and Atsushi Nitanda and Yan Shuo Tan},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.07074},
eprint = {2610.07074},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07074},
}
RIS
TY - UNPB
TI - Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention
AU - Hong Ha Le
AU - Jackie Lok
AU - Atsushi Nitanda
AU - Yan Shuo Tan
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.07074
UR - https://arxiv.org/abs/2610.07074
AB - Estimating a decision threshold requires locating observations near an unknown boundary. We study how gradient-based pretraining learns this statistical rule in a two-parameter softmax-attention model with a fixed feature and inequality direction. Pretraining uses labeled contexts and their true thresholds; a fresh threshold must be inferred from context alone. Under a large-resolution initialization, constant-step gradient descent on m tasks with n examples each produces a frozen estimator with error Õ((m∧ n)^−1+N^−1) for each fixed interior threshold and every fresh-context size N. The two terms separate finite-pretraining accuracy from fresh-context localization. The mechanism is coordinated parameter divergence: population training calibrates the relative label and feature scores, then increases the attention scale as t^1/4, giving population threshold error O(t^−1/4). To transfer this mechanism to a fixed finite corpus, we control gradient errors relative to the shrinking directions of progress at successive parameter scales. This certifies a growing training interval without requiring long-time tracking of the population trajectory. We also identify the boundary limitation of the one-head model and explain statistically what a reflected symmetrization could achieve.
ER -