Sequential Functional Structured Tucker Compression for Large Language Model Attentions
Abstract
Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.
Related
- Sequential Lossy Compression With Causal Conditional Perception
- Three-dimensional animation compression optimization based on structured processing and adaptive spatiotemporal segmentation algorithm
- Language Modeling is Monotone Compression
- The Functional Structure of Post-Compression Recovery in Low-Rank LLMs
- Adaptive Functional Clustering with Structured Dependence via Variational Inference
BibTeX
@misc{chen2026sequential,
title = {{Sequential Functional Structured Tucker Compression for Large Language Model Attentions}},
author = {Jiangfeng Chen and Xinyu Wang and Tianshuo Yan and Hanwei Wu and Xiao-Wen Chang and Yang Zhang and Lei Ding},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.00717},
eprint = {2610.00717},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.00717},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Sequential Functional Structured Tucker Compression for Large Language Model Attentions
AU - Jiangfeng Chen
AU - Xinyu Wang
AU - Tianshuo Yan
AU - Hanwei Wu
AU - Xiao-Wen Chang
AU - Yang Zhang
AU - Lei Ding
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.00717
UR - https://arxiv.org/abs/2610.00717
AB - Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.
ER -