Many Tenants, One Cache: Per-Tenant Rotation of the Attention Cache in a Production Serving Engine

Abstract

Language model serving engines put many customers into one process on one accelerator, sharing one attention cache. Per-tenant rotation stores each tenant’s cached rows under a tenant-specific orthogonal transform and rotates the tenant’s query the same way, so attention is unchanged for the tenant and a read under any other key returns nothing usable. We measure what this costs, and whether it survives the features that make modern serving dense, in a production engine (vLLM) on one confidential H100, across five models and a set of pre-registered phases.

Three results stand out. First, rotation extends to multi-head latent attention: on Moonlight 16B, a memory-bound ceiling of 793 tenants against 822 without rotation at 2,048 tokens, a prefill gap of 14.3 ms and a decode gap of 0.74 to 1.35 ms per token, with every isolation gate passed. Second, one H100 holds 4,000 registered Moonlight tenants at a 2-second service target, with or without rotation, by parking idle tenants off the card and reviving them in 35.3 ms against 32.8 ms without rotation. The ceiling is host memory for the parked tenants, not the GPU. Third, a shared-prefix tier that holds a common system prompt once halves a new tenant’s first-request time (24.9 against 48.0 ms) and makes a parked tenant about 17 times smaller (4.0 against 67.7 MB). On the key path, fusion and CUDA graphs bring rotation’s decode cost to 0.8 to 2.2 ms per token, and an FP8 cache doubles the tenant ceiling under rotation.

The overhead targets we registered failed in every boot, and the Moonlight density figures rest on 64 keys. Both are reported in full. What rotation does not hide is measured in the companion Paper B.

Companion paper: “What Per-Tenant Rotation Leaks” (doi:10.5281/zenodo.23452292).

What Per-Tenant Rotation Leaks: Red-Teaming a Rotated Attention Cache, and a Confined Byte Layer That Closes the Gap
Luis Carranza
Zenodo18 Oct 2026PreprintCC BY 4.0Full textZenodo
Innovative management program for achieving high yield and emission reduction: a case study of wheat/maize rotation production in the North China Plain
Yanchen Gong, Xiaoqin Jiang, Yongheng Du +6
Frontiers in plant science16 Oct 2026Peer-reviewedCC BYFull textPlant biologyEurope PMC
Diversified cropping rotation increases soil carbon sequestration via fungal-mediated necromass production and mineral association
Cheng Ji, Xinhu Guo, Kevin Z. Mganga +10
Communications Earth & Environment30 Oct 2026Peer-reviewedCC BY-NC-ND 4.0General earth scienceNatureCrossref
RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention
Shiyu Ji, Peng Ling, Chen Lin +4
arXiv14 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Coherence Rather Than Error Rate Governs Privacy in Multi-Tenant Quantum Computing
Farhad Farokhi
arXiv1 Oct 2026PreprintarXiv licenceFull textQuantum physics & quantum informationarXiv

Cite

BibTeX RIS

BibTeX

@misc{carranza2026many,
  title = {{Many Tenants, One Cache: Per-Tenant Rotation of the Attention Cache in a Production Serving Engine}},
  author = {Luis Carranza},
  howpublished = {Zenodo},
  year = {2026},
  month = oct,
  doi = {10.5281/zenodo.23452278},
  url = {https://zenodo.org/records/23452278},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Many Tenants, One Cache: Per-Tenant Rotation of the Attention Cache in a Production Serving Engine
AU  - Luis Carranza
PY  - 2026
DA  - 2026/10/18
PB  - Zenodo
DO  - 10.5281/zenodo.23452278
UR  - https://zenodo.org/records/23452278
AB  - Language model serving engines put many customers into one process on one accelerator, sharing one attention cache. Per-tenant rotation stores each tenant’s cached rows under a tenant-specific orthogonal transform and rotates the tenant’s query the same way, so attention is unchanged for the tenant and a read under any other key returns nothing usable. We measure what this costs, and whether it survives the features that make modern serving dense, in a production engine (vLLM) on one confidential H100, across five models and a set of pre-registered phases.  Three results stand out. First, rotation extends to multi-head latent attention: on Moonlight 16B, a memory-bound ceiling of 793 tenants against 822 without rotation at 2,048 tokens, a prefill gap of 14.3 ms and a decode gap of 0.74 to 1.35 ms per token, with every isolation gate passed. Second, one H100 holds 4,000 registered Moonlight tenants at a 2-second service target, with or without rotation, by parking idle tenants off the card and reviving them in 35.3 ms against 32.8 ms without rotation. The ceiling is host memory for the parked tenants, not the GPU. Third, a shared-prefix tier that holds a common system prompt once halves a new tenant’s first-request time (24.9 against 48.0 ms) and makes a parked tenant about 17 times smaller (4.0 against 67.7 MB). On the key path, fusion and CUDA graphs bring rotation’s decode cost to 0.8 to 2.2 ms per token, and an FP8 cache doubles the tenant ceiling under rotation.  The overhead targets we registered failed in every boot, and the Moonlight density figures rest on 64 keys. Both are reported in full. What rotation does not hide is measured in the companion Paper B.  Companion paper: “What Per-Tenant Rotation Leaks” (doi:10.5281/zenodo.23452292).
ER  -

Abstract

Related

Cite