A Layered Simplex Architecture for Large Alphabets
Abstract
A learning architecture induces a prior over its expressible models. Recent work links architectural success to a broad complexity spectrum: model probabilities spanning many orders of magnitude. Understanding how depth shapes these priors and prediction may help explain the success of neural networks and transformers. We study this connection in large-alphabet prediction through the *layered simplex architecture* (LSA), which multiplies independent uniform simplex vectors coordinate-wise and normalizes. At depth zero, the prior is concentrated on the uniform distribution; at depth one, the Bayesian predictor is Laplace's rule. Each additional layer shifts prior mass toward more concentrated distributions. We derive exact Bayesian marginal likelihood and regret expressions using reusable depth-specific kernels. Mixing over depths up to a cutoff logarithmic in alphabet size yields an explicit predictor with rigorous guarantees. For targets uniform on a specified sparse support, the mixture's prior complexity is asymptotic to the support-description cost at every fixed positive KL resolution, establishing a broad complexity spectrum. We also prove a regret lower bound for every label-invariant predictor: the expected description length of the discovered symbol set minus the source's uncertainty about its identity given its size. For Zipf sources in a growing-alphabet regime, the depth mixture attains this bound to leading order. Thus, the symbol-discovery rate determines the regret exponent. Although introduced to study depth, the LSA predictor competes with established large-alphabet methods, including Good--Turing, Ristad's law and an adaptive Dirichlet mixture, while substantially outperforming classical add-constant rules on concentrated sources.
Related
- Systematic Codes Correcting Two Deletions with Three Redundant Symbols over Large Alphabets
- Co-designing a Layered Control Architecture
- On high-girth layered graphs of large Turán density in a hypercube
- The probabilistic bound for perfect hashing is sharp only for binary alphabets
- Unique ergodicity, and not, for primitive substitutions on compact alphabets containing isolated points
BibTeX
@misc{feder2026layered,
title = {{A Layered Simplex Architecture for Large Alphabets}},
author = {Meir Feder and Yaniv Fogel and Ruediger Urbanke},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2608.19908},
eprint = {2608.19908},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.19908},
}
RIS
TY - UNPB
TI - A Layered Simplex Architecture for Large Alphabets
AU - Meir Feder
AU - Yaniv Fogel
AU - Ruediger Urbanke
PY - 2026
DA - 2026/11/03
PB - arXiv
DO - 10.48550/arxiv.2608.19908
UR - https://arxiv.org/abs/2608.19908
AB - A learning architecture induces a prior over its expressible models. Recent work links architectural success to a broad complexity spectrum: model probabilities spanning many orders of magnitude. Understanding how depth shapes these priors and prediction may help explain the success of neural networks and transformers. We study this connection in large-alphabet prediction through the *layered simplex architecture* (LSA), which multiplies independent uniform simplex vectors coordinate-wise and normalizes. At depth zero, the prior is concentrated on the uniform distribution; at depth one, the Bayesian predictor is Laplace's rule. Each additional layer shifts prior mass toward more concentrated distributions. We derive exact Bayesian marginal likelihood and regret expressions using reusable depth-specific kernels. Mixing over depths up to a cutoff logarithmic in alphabet size yields an explicit predictor with rigorous guarantees. For targets uniform on a specified sparse support, the mixture's prior complexity is asymptotic to the support-description cost at every fixed positive KL resolution, establishing a broad complexity spectrum. We also prove a regret lower bound for every label-invariant predictor: the expected description length of the discovered symbol set minus the source's uncertainty about its identity given its size. For Zipf sources in a growing-alphabet regime, the depth mixture attains this bound to leading order. Thus, the symbol-discovery rate determines the regret exponent. Although introduced to study depth, the LSA predictor competes with established large-alphabet methods, including Good--Turing, Ristad's law and an adaptive Dirichlet mixture, while substantially outperforming classical add-constant rules on concentrated sources.
ER -