Causal Representation Learning for Generalisable Recommendation
Abstract
Predictive models trained on observational data often fail to generalise when deployed, especially when the training data is a product of the system being optimised. Recommender systems are a canonical example: they are trained on interaction logs confounded by the deployed policy, past user behaviour, and platform filtering. As a result, the training distribution differs from the candidate distribution scored at serving time, a gap that makes offline metrics unreliable predictors of online performance. We address the distribution shift problem with a method motivated by causal representation learning (CRL). We propose an information-theoretic disentanglement criterion and prove that its optimum excludes confounder-dependent non-causal information from the treatment representation. We derive a tractable variational lower bound making the criterion optimisable from finite observational data. The scope of our method is narrower than that of much of the CRL literature, in that we target better generalisation under distribution shift, not full identification of latent causal factors. This narrower target makes the method practical, requiring only existing confounded logs, applying to any standard supervised model, and adding no inference-time cost. Our headline evaluation is an A/B test with millions of users on Spotify, applied to a production ranker for personalised playlist generation. A CRL variant matched in capacity to the production baseline performed on par offline but delivered substantial online gains in listener engagement. Complementary evidence on a synthetic dataset with known causal structure and a semi-synthetic variant of the public KuaiRand recommendation dataset shows similar in-distribution performance to the baseline but gains under distribution shift. Across all three settings, adding our causal disentanglement objective yields better out-of-distribution generalisation.
Related
- Identifying ODEs from Unstructured Data with Causal Representation Learning
- HyCoSeq: Contextual Hyperbolic Representation Learning for Genomic Sequences
- FOCUS: Fixed-Confidence Online Causal Learning Using Sequential Adaptive Interventions
- Representation Learning with Quantum Signal Processing
- PaECTER: Patent-level representation learning using citation-informed transformers
BibTeX
@misc{felekis2026causal,
title = {{Causal Representation Learning for Generalisable Recommendation}},
author = {Yorgos Felekis and Michael O'Riordan and Oriol Corcoll and Ciarán M. Gilligan-Lee},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.27043},
eprint = {2605.27043},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.27043},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Causal Representation Learning for Generalisable Recommendation
AU - Yorgos Felekis
AU - Michael O'Riordan
AU - Oriol Corcoll
AU - Ciarán M. Gilligan-Lee
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2605.27043
UR - https://arxiv.org/abs/2605.27043
AB - Predictive models trained on observational data often fail to generalise when deployed, especially when the training data is a product of the system being optimised. Recommender systems are a canonical example: they are trained on interaction logs confounded by the deployed policy, past user behaviour, and platform filtering. As a result, the training distribution differs from the candidate distribution scored at serving time, a gap that makes offline metrics unreliable predictors of online performance. We address the distribution shift problem with a method motivated by causal representation learning (CRL). We propose an information-theoretic disentanglement criterion and prove that its optimum excludes confounder-dependent non-causal information from the treatment representation. We derive a tractable variational lower bound making the criterion optimisable from finite observational data. The scope of our method is narrower than that of much of the CRL literature, in that we target better generalisation under distribution shift, not full identification of latent causal factors. This narrower target makes the method practical, requiring only existing confounded logs, applying to any standard supervised model, and adding no inference-time cost. Our headline evaluation is an A/B test with millions of users on Spotify, applied to a production ranker for personalised playlist generation. A CRL variant matched in capacity to the production baseline performed on par offline but delivered substantial online gains in listener engagement. Complementary evidence on a synthetic dataset with known causal structure and a semi-synthetic variant of the public KuaiRand recommendation dataset shows similar in-distribution performance to the baseline but gains under distribution shift. Across all three settings, adding our causal disentanglement objective yields better out-of-distribution generalisation.
ER -