Near-Floor Geometry Is Generic: Leverage Dispersion in Trained Overcomplete Codes
Abstract
An overcomplete code packs F features into d<F dimensions, so linear readout of one feature picks up cross-talk from the others, bounded below by the rank-trace floor W(F,d)=(F−d)/(d(F−1)). Proximity to this floor is read as evidence of an efficient arrangement. It is not: an i.i.d. code of the same shape attains it at every shape and load we measure (1.0000-1.0005 over 7 matched controls), and the reason is exact. For the gain-calibrated pseudoinverse the attainment ratio is d/(F(F−d))(∑i hi−1−F) with ∑i hi=d, so it equals one precisely when the leverage hi is equalised across features: distance from the floor IS leverage dispersion. On 40 released decoder matrices - 24 sparse-autoencoder decoders plus 16 MLP down-projections of a hybrid model - all sit above their matched control (1.0055-1.2018), and all reach guaranteed affine failure 4 to 40 sparsity levels earlier: where a random code of the same shape is still safe, a released dictionary already has a feature no affine rule recovers. Autoencoders trained by the same recipe on a randomly initialised copy of the same model return to the floor (1.0038-1.0065) in all 10 paired layers, so the excess reflects what the model learned, not the act of fitting an autoencoder. Under controlled conditions the loss decides whether an interface exists at all: L2 reaches Rgeom=18.5635 to 32.3131 with its frontier collapsed to $1$, against $1.0002$ for L4, and that frontier follows a rate we registered before the widest run (10.79 predicted, 10.510 measured). Two limits are stated rather than implied: frontier statements are worst cases, not typical-case claims; and because the probe class contains the network's own decoder, our decoder comparison bounds supervised fitting rather than affine expressiveness.
Related
- Approximate Matrix Multiplication through Joint Leverage Score Sampling
- With Leverage, the Gambler is Not the One Who Gets Ruined
- High-Frequency Estimation of the Leverage Effect under Rough Volatility with Microstructure Noise
- Generic small-scale creation in the two-dimensional Euler equation
- Modularity of higher theta series I: cohomology of the generic fiber
BibTeX
@misc{borobia2026nearfloor,
title = {{Near-Floor Geometry Is Generic: Leverage Dispersion in Trained Overcomplete Codes}},
author = {Hector Borobia and Elies Seguí-Mas and Guillermina Tormo-Carbó},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.01192},
eprint = {2605.01192},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.01192},
}
RIS
TY - UNPB
TI - Near-Floor Geometry Is Generic: Leverage Dispersion in Trained Overcomplete Codes
AU - Hector Borobia
AU - Elies Seguí-Mas
AU - Guillermina Tormo-Carbó
PY - 2026
DA - 2026/09/05
PB - arXiv
DO - 10.48550/arxiv.2605.01192
UR - https://arxiv.org/abs/2605.01192
AB - An overcomplete code packs F features into d<F dimensions, so linear readout of one feature picks up cross-talk from the others, bounded below by the rank-trace floor W(F,d)=(F−d)/(d(F−1)). Proximity to this floor is read as evidence of an efficient arrangement. It is not: an i.i.d. code of the same shape attains it at every shape and load we measure (1.0000-1.0005 over 7 matched controls), and the reason is exact. For the gain-calibrated pseudoinverse the attainment ratio is d/(F(F−d))(∑_i h_i^−1−F) with ∑_i h_i=d, so it equals one precisely when the leverage h_i is equalised across features: distance from the floor IS leverage dispersion. On 40 released decoder matrices - 24 sparse-autoencoder decoders plus 16 MLP down-projections of a hybrid model - all sit above their matched control (1.0055-1.2018), and all reach guaranteed affine failure 4 to 40 sparsity levels earlier: where a random code of the same shape is still safe, a released dictionary already has a feature no affine rule recovers. Autoencoders trained by the same recipe on a randomly initialised copy of the same model return to the floor (1.0038-1.0065) in all 10 paired layers, so the excess reflects what the model learned, not the act of fitting an autoencoder. Under controlled conditions the loss decides whether an interface exists at all: L^2 reaches R_geom=18.5635 to 32.3131 with its frontier collapsed to $1$, against $1.0002$ for L^4, and that frontier follows a rate we registered before the widest run (10.79 predicted, 10.510 measured). Two limits are stated rather than implied: frontier statements are worst cases, not typical-case claims; and because the probe class contains the network's own decoder, our decoder comparison bounds supervised fitting rather than affine expressiveness.
ER -