The Geometry of Updates: Fisher Alignment at Vocabulary Scale
Abstract
Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in prediction targets. This creates an activation-dark regime: representation-similarity metrics can be uninformative without assumptions about label-conditioned error geometry, while classical update-geometry metrics are computationally prohibitive at vocabulary scale. We show that, in a shared-output head setting, representation metrics (e.g., CKA) are non-identifiable for transfer; models can share identical representations yet have orthogonal head updates. The key identity is that head Fisher alignment is exactly a cosine between kernel mean embeddings in the joint activation-error space, exposing activation, error, and coupling factors rather than requiring a materialized Fisher matrix. FisherSketch estimates this cosine directly in a single streaming pass, making K=128,256 head Fisher alignment practical with a 16 KB task signature (m=4096) and a 192 KB per-task streaming state, small enough to store next to a model hash, but encoding transfer-relevant update structure. Beyond source selection, the same signatures and marginals provide a diagnostic instrument for studying whether LLM task similarity is driven by activations, errors, or their coupling; shared-parameter and internal-layer validations, together with Llama-3.1-8B verbalizer-shift experiments, show that FisherSketch remains informative when activation similarity cannot distinguish tasks.
Related
- Geometry-Aware Multi-Objective Optimization: Norm-Constrained Oracles with Entropic Weight Updates
- Graph-diameter criteria and sharp dissipation in discrete Wasserstein-Fisher geometry
- Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods
- Fisher Geometry of Circular Wrapped Normals: Completion and Curvature
- Spectral duality structures and the Fisher--Rao geometry of reset distributions
BibTeX
@misc{sweeney2026geometry,
title = {{The Geometry of Updates: Fisher Alignment at Vocabulary Scale}},
author = {John Sweeney},
howpublished = {arXiv},
year = {2026},
month = jun,
doi = {10.48550/arxiv.2606.27242},
eprint = {2606.27242},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.27242},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The Geometry of Updates: Fisher Alignment at Vocabulary Scale
AU - John Sweeney
PY - 2026
DA - 2026/06/25
PB - arXiv
DO - 10.48550/arxiv.2606.27242
UR - https://arxiv.org/abs/2606.27242
AB - Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in prediction targets. This creates an activation-dark regime: representation-similarity metrics can be uninformative without assumptions about label-conditioned error geometry, while classical update-geometry metrics are computationally prohibitive at vocabulary scale. We show that, in a shared-output head setting, representation metrics (e.g., CKA) are non-identifiable for transfer; models can share identical representations yet have orthogonal head updates. The key identity is that head Fisher alignment is exactly a cosine between kernel mean embeddings in the joint activation-error space, exposing activation, error, and coupling factors rather than requiring a materialized Fisher matrix. FisherSketch estimates this cosine directly in a single streaming pass, making K=128,256 head Fisher alignment practical with a 16 KB task signature (m=4096) and a 192 KB per-task streaming state, small enough to store next to a model hash, but encoding transfer-relevant update structure. Beyond source selection, the same signatures and marginals provide a diagnostic instrument for studying whether LLM task similarity is driven by activations, errors, or their coupling; shared-parameter and internal-layer validations, together with Llama-3.1-8B verbalizer-shift experiments, show that FisherSketch remains informative when activation similarity cannot distinguish tasks.
ER -