Cognitive Load-Informed Attention Budgets Across Self-Attention and Token–Context Attention
- 1Department of Computer Science, Texas Tech University, Lubbock, TX 79409, USA
Abstract
We investigate a theory-informed attention framework combining Cognitive Load Theory (CLT)-inspired resource regulation with Token–Context Attention (TCA) and a TCA-specific application of root-mean-square (RMS) branch normalization. Seven primary conditions were evaluated across SNLI, SST-2, IMDb, AG News, and DBPedia 14 using 10 paired seeds and locked data manifests, with a validation-calibrated fixed-gate control for adaptive TCA-RMS. A separately locked secondary ablation added entropy-only CLT (CLT-E) to self-attention, TCA, and TCA-RMS under the same records and seeds. RMS branch normalization increased mean raw-TCA accuracy on all five datasets, with Holm-adjusted raw-TCA accuracy support on IMDb and AG News. In contrast, none of the prespecified direct mixed-CLT accuracy additions survived the within-dataset Holm correction; CLT-related evidence was localized to particular architectures, datasets, and metrics. No adaptive-versus-fixed accuracy contrast was detected after correction, and the entropy-only ablation shows that useful attenuation can arise without the full entropy–uncertainty–IDF composition. Accordingly, the results support branch-scale normalization and bounded attention-output gating as separable architectural interventions, while neither universal accuracy improvement nor human cognitive-load construct validity is claimed. These conclusions are restricted to lightweight, from-scratch classifiers trained under locked limited-data protocols; whether pretrained encoders change the need for TCA branch normalization or the effects of the CLT-inspired gate remains an open empirical question.
Related
- What can linear attention learn from nonlinear teachers in-context?
- In-context classification: Calibration and noise in linear attention with softmax readout
- Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention
- Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms
- Universality and Generalization of Causal Transformers Across Context Lengths
BibTeX
@article{graham2026cognitive,
title = {{Cognitive Load-Informed Attention Budgets Across Self-Attention and Token–Context Attention}},
author = {Jarrod Graham and Victor S. Sheng},
journal = {Mathematics},
year = {2026},
month = nov,
doi = {10.3390/math14214076},
url = {https://www.mdpi.com/2227-7390/14/21/4076},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Cognitive Load-Informed Attention Budgets Across Self-Attention and Token–Context Attention
AU - Jarrod Graham
AU - Victor S. Sheng
PY - 2026
DA - 2026/11/06
JO - Mathematics
DO - 10.3390/math14214076
UR - https://www.mdpi.com/2227-7390/14/21/4076
AB - We investigate a theory-informed attention framework combining Cognitive Load Theory (CLT)-inspired resource regulation with Token–Context Attention (TCA) and a TCA-specific application of root-mean-square (RMS) branch normalization. Seven primary conditions were evaluated across SNLI, SST-2, IMDb, AG News, and DBPedia 14 using 10 paired seeds and locked data manifests, with a validation-calibrated fixed-gate control for adaptive TCA-RMS. A separately locked secondary ablation added entropy-only CLT (CLT-E) to self-attention, TCA, and TCA-RMS under the same records and seeds. RMS branch normalization increased mean raw-TCA accuracy on all five datasets, with Holm-adjusted raw-TCA accuracy support on IMDb and AG News. In contrast, none of the prespecified direct mixed-CLT accuracy additions survived the within-dataset Holm correction; CLT-related evidence was localized to particular architectures, datasets, and metrics. No adaptive-versus-fixed accuracy contrast was detected after correction, and the entropy-only ablation shows that useful attenuation can arise without the full entropy–uncertainty–IDF composition. Accordingly, the results support branch-scale normalization and bounded attention-output gating as separable architectural interventions, while neither universal accuracy improvement nor human cognitive-load construct validity is claimed. These conclusions are restricted to lightweight, from-scratch classifiers trained under locked limited-data protocols; whether pretrained encoders change the need for TCA branch normalization or the effects of the CLT-inspired gate remains an open empirical question.
SN - 2227-7390
ER -