How Risky Is Your Data-Driven Policy?
Abstract
Data-driven algorithms are ubiquitous across industries, and there is an extensive literature on analyzing their \emph{average} performance. The risk profile of such algorithms, while critical, is much less understood. This paper quantifies the risk of data-driven policies and isolates its sources. We evaluate a policy through the conditional value-at-risk (CVaR) of its loss over the training sample, any policy randomization, and the future outcome, at a fixed unknown distribution. Regret against an oracle that knows the distribution decomposes into expected post-sample regret (PS), the average outcome-only CVaR gap of the selected action, and a sample-risk premium (SR), the additional tail risk omitted by averaging those evaluations. The analogous premium vanishes under expected loss. We characterize when the premium vanishes and how large it is relative to post-sample regret. For losses with a continuous density near the tail cutoff, we identify conditions under which both components decay at the same rate, yet the limiting ratio of the premium to post-sample regret can reach the thousands. When a probability mass at the cutoff moves with the action, the premium can decay more slowly and dominate post-sample regret. We establish conditions under which sample average approximation learns the oracle action at rate n−1/2 in both settings, but its combined-risk regret is of order n−1 in the first and n−1/2 in the second. In finite-sample numerical examples of SAA in both settings, the premium accounts for approximately 74%--97% of regret.
Related
- Reduction of the integro-differential equation to an ordinary differential equation in ruin models with investment in a risky asset
- Build Your Own Neural Network From Scratch
- A Branching Process with Geometric Emigration and the "Have Your Cookie and Eat It" Random Walk
- How protomodular is your favourite category?
- Choose your own adventure: How data source selection shapes urban accessibility analyses
BibTeX
@misc{besbes2026risky,
title = {{How Risky Is Your Data-Driven Policy?}},
author = {Omar Besbes and Alon Rieger},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.04292},
eprint = {2611.04292},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.04292},
}
RIS
TY - UNPB
TI - How Risky Is Your Data-Driven Policy?
AU - Omar Besbes
AU - Alon Rieger
PY - 2026
DA - 2026/11/03
PB - arXiv
DO - 10.48550/arxiv.2611.04292
UR - https://arxiv.org/abs/2611.04292
AB - Data-driven algorithms are ubiquitous across industries, and there is an extensive literature on analyzing their \emph{average} performance. The risk profile of such algorithms, while critical, is much less understood. This paper quantifies the risk of data-driven policies and isolates its sources. We evaluate a policy through the conditional value-at-risk (CVaR) of its loss over the training sample, any policy randomization, and the future outcome, at a fixed unknown distribution. Regret against an oracle that knows the distribution decomposes into expected post-sample regret (PS), the average outcome-only CVaR gap of the selected action, and a sample-risk premium (SR), the additional tail risk omitted by averaging those evaluations. The analogous premium vanishes under expected loss. We characterize when the premium vanishes and how large it is relative to post-sample regret. For losses with a continuous density near the tail cutoff, we identify conditions under which both components decay at the same rate, yet the limiting ratio of the premium to post-sample regret can reach the thousands. When a probability mass at the cutoff moves with the action, the premium can decay more slowly and dominate post-sample regret. We establish conditions under which sample average approximation learns the oracle action at rate n^−1/2 in both settings, but its combined-risk regret is of order n^−1 in the first and n^−1/2 in the second. In finite-sample numerical examples of SAA in both settings, the premium accounts for approximately 74%--97% of regret.
ER -