Efficient Clustering with Quality Guardrails for LLM-based Recommender Systems at Industry Scale
Abstract
LLMs can be prohibitively expensive and slow to run at scale, especially for applications that invoke an LLM per sample over millions of inputs. A natural way to scale is to cluster the inputs, run the LLM only on cluster representatives, and propagate the outputs to other cluster members. However, the outputs a member receives are only as good as its match to the representative. Off-the-shelf clustering methods optimize an aggregate objective, targeting average-case quality without per-sample guardrails. As a result, members can be assigned to poorly-matched representatives, and the inherited outputs -- though appropriate for the representative -- may be irrelevant or even unsafe for the member. For example, a parent of a toddler grouped with parents of older children could receive age-inappropriate recommendations. Most clustering methods also scale poorly to millions of inputs in runtime and memory, limiting their use at industry scale. We propose a scalable two-stage clustering algorithm with provable per-sample guardrails: every sample is guaranteed to share a user-specified minimal embedding similarity and exact attribute match with its representative. The algorithm first generates initial clusters with Mini-batch K-Means, then greedily selects representatives within each to satisfy the guardrails. We provide theoretical guarantees, complexity analysis, and benchmarks against common methods on internal and public datasets. Our method delivers per-sample guardrails while running substantially faster and scaling to data sizes where most standard methods become intractable. We demonstrate its impact in a real-world deployment clustering 38 million customers, reducing downstream LLM cost and runtime by 50-fold while preserving personalization. This unblocked the launch of a persona-based recommender system that delivers significant gains in revenue and engagement in an A/B test.
Related
- Efficient Discrete Multi-View Clustering with Tensorized Label Learning
- Data Quality Audit and Bivariate Temperature-Humidity Analysis of a Long-Term LoRaWAN Network: Anomaly Detection, PCA, Clustering and Machine Learning
- Where should the null live? Surrogate tests for clustering of threshold-derived air-quality indices, with application to PM2.5 exceedances in Chile
- Toward Well-Posed Problems in the Social Sciences: Hadamard's Criteria as Epistemic Guardrails
- Ensemble-based support vector machine classifiers as an efficient tool for quality assessment of beef fillets from electronic nose data
BibTeX
@misc{wang2026efficient,
title = {{Efficient Clustering with Quality Guardrails for LLM-based Recommender Systems at Industry Scale}},
author = {Longshaokan Wang and Wai Tsang Keung and Punit Ghodasara and Roman Wang and Ali Dashti and Francesc Moreno-Noguer},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2607.19704},
eprint = {2607.19704},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.19704},
}
RIS
TY - UNPB
TI - Efficient Clustering with Quality Guardrails for LLM-based Recommender Systems at Industry Scale
AU - Longshaokan Wang
AU - Wai Tsang Keung
AU - Punit Ghodasara
AU - Roman Wang
AU - Ali Dashti
AU - Francesc Moreno-Noguer
PY - 2026
DA - 2026/09/03
PB - arXiv
DO - 10.48550/arxiv.2607.19704
UR - https://arxiv.org/abs/2607.19704
AB - LLMs can be prohibitively expensive and slow to run at scale, especially for applications that invoke an LLM per sample over millions of inputs. A natural way to scale is to cluster the inputs, run the LLM only on cluster representatives, and propagate the outputs to other cluster members. However, the outputs a member receives are only as good as its match to the representative. Off-the-shelf clustering methods optimize an aggregate objective, targeting average-case quality without per-sample guardrails. As a result, members can be assigned to poorly-matched representatives, and the inherited outputs -- though appropriate for the representative -- may be irrelevant or even unsafe for the member. For example, a parent of a toddler grouped with parents of older children could receive age-inappropriate recommendations. Most clustering methods also scale poorly to millions of inputs in runtime and memory, limiting their use at industry scale. We propose a scalable two-stage clustering algorithm with provable per-sample guardrails: every sample is guaranteed to share a user-specified minimal embedding similarity and exact attribute match with its representative. The algorithm first generates initial clusters with Mini-batch K-Means, then greedily selects representatives within each to satisfy the guardrails. We provide theoretical guarantees, complexity analysis, and benchmarks against common methods on internal and public datasets. Our method delivers per-sample guardrails while running substantially faster and scaling to data sizes where most standard methods become intractable. We demonstrate its impact in a real-world deployment clustering 38 million customers, reducing downstream LLM cost and runtime by 50-fold while preserving personalization. This unblocked the launch of a persona-based recommender system that delivers significant gains in revenue and engagement in an A/B test.
ER -