Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison
Abstract
The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and grouped layers. We measure the singular-value spectra of all four mode unfoldings on Mistral-7B-v0.3 and LLaMA-2-13B and compare four standard tensor decompositions: Tucker, CP, tensor train, and t-SVD, at matched storage. The spectra partition the four axes into two classes. The token and feature modes carry low-rank structure, particularly for keys. The head and layer modes are nearly full-rank and resist compression at any practical error level. Among the four decompositions, Tucker achieves the lowest reconstruction error at every compression ratio from 2× to 5×, because it can leave the full-rank modes untouched. Comparisons with two-dimensional unfolding baselines show that the preferred representation differs between keys and values: 2D methods achieve lower key error, while four-way Tucker achieves lower value error at matched storage. A mode-pinning theorem certifies the full-rank preservation from the measured spectra alone. Two further spectral properties affect the compressible modes without touching the full-rank ones: values reach a higher error floor than keys at every ratio, and post-RoPE keys lose 41% - 64% of their pre-RoPE compressibility on both models.
Related
- Eigensolvers for polynomial roots and tensor decomposition
- Heteroskedastic Canonical Polyadic Tensor Decomposition
- PAR2COX: Survival-Informed Tensor Decomposition for Phenotyping and Risk Prediction from Irregular Longitudinal Data
- A Fast Algorithm for Quaternion Tensor Singular Value Decomposition under the QT-Product with Applications to Color Video Processing
- Beyond Kruskal: Polynomial-Time Tensor Decomposition under the Lovitz-Petrov Condition
BibTeX
@misc{krishnan2026tensor,
title = {{Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison}},
author = {Rahul Krishnan and Volker Schulz},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.28029},
eprint = {2609.28029},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.28029},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison
AU - Rahul Krishnan
AU - Volker Schulz
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.28029
UR - https://arxiv.org/abs/2609.28029
AB - The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and grouped layers. We measure the singular-value spectra of all four mode unfoldings on Mistral-7B-v0.3 and LLaMA-2-13B and compare four standard tensor decompositions: Tucker, CP, tensor train, and t-SVD, at matched storage. The spectra partition the four axes into two classes. The token and feature modes carry low-rank structure, particularly for keys. The head and layer modes are nearly full-rank and resist compression at any practical error level. Among the four decompositions, Tucker achieves the lowest reconstruction error at every compression ratio from 2× to 5×, because it can leave the full-rank modes untouched. Comparisons with two-dimensional unfolding baselines show that the preferred representation differs between keys and values: 2D methods achieve lower key error, while four-way Tucker achieves lower value error at matched storage. A mode-pinning theorem certifies the full-rank preservation from the measured spectra alone. Two further spectral properties affect the compressible modes without touching the full-rank ones: values reach a higher error floor than keys at every ratio, and post-RoPE keys lose 41% - 64% of their pre-RoPE compressibility on both models.
ER -