Parallel gradient boosting for flexible estimation of conditional distributions
Abstract
Boosting is one of the most successful learning techniques for standard classification and regression tasks. Its extension to multi-output prediction problems has found an increasing number of applications in recent years. Among them is the prediction of entire conditional distributions rather than single functionals, which can often be framed as a multi-output regression problem, for example multiple quantile regression. Addressing such problems with classical implementations of boosting is computationally challenging, because usually one base model is trained for each target at every iteration. More efficient variants of boosting have been proposed to speed up training, but they tend to be tied to specific loss functions and classes of base learners, usually decision trees. In this work, we study a modification of the gradient boosting algorithm, which we call parallel gradient boosting, designed to circumvent all these limitations. The core idea is to use a common descent direction for all training observations. By doing so, only one base model is needed at each iteration, regardless of the number of targets, which allows for considerable performance gains. We establish sufficient conditions for the convergence of the algorithm, whose practical use is introduced via the multiple quantile regression setting. We show that in such a setting, it provides predictions of similar quality to state-of-the-art boosting libraries such as XGBoost, while being faster by several orders of magnitude. Then, we evaluate the properties of the resulting conditional distribution estimator, which is shown empirically to outperform other nonparametric and semiparametric estimators, especially in high-dimensional settings and in the presence of mixed and/or missing covariates.
Related
- Nonparametric autoregressive estimation of probability distributions with parallel gradient boosting
- Conditional Weight of Evidence for Gradient-Boosted Credit Models: Estimation, Additivity and Its Relation to Adverse Action Reasons
- A functional central limit theorem for kernel gradient flow and infinitesimal gradient boosting
- Gradient boosting machine for survival prediction in studies with general failure-time outcome-dependent sampling design.
- Gradient boosting models for gallstone disease classification: A comparative study using demographic and clinical data
BibTeX
@misc{chapelle2026parallel,
title = {{Parallel gradient boosting for flexible estimation of conditional distributions}},
author = {Rémy Chapelle and Nicolas Vayatis and Bruno Falissard and Mohammed Sedki},
howpublished = {HAL},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2607.13550},
eprint = {2607.13550},
archivePrefix = {arXiv},
url = {https://hal.science/hal-05691531v2},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - Parallel gradient boosting for flexible estimation of conditional distributions
AU - Rémy Chapelle
AU - Nicolas Vayatis
AU - Bruno Falissard
AU - Mohammed Sedki
PY - 2026
DA - 2026/10/22
PB - HAL
DO - 10.48550/arxiv.2607.13550
UR - https://hal.science/hal-05691531v2
AB - Boosting is one of the most successful learning techniques for standard classification and regression tasks. Its extension to multi-output prediction problems has found an increasing number of applications in recent years. Among them is the prediction of entire conditional distributions rather than single functionals, which can often be framed as a multi-output regression problem, for example multiple quantile regression. Addressing such problems with classical implementations of boosting is computationally challenging, because usually one base model is trained for each target at every iteration. More efficient variants of boosting have been proposed to speed up training, but they tend to be tied to specific loss functions and classes of base learners, usually decision trees. In this work, we study a modification of the gradient boosting algorithm, which we call parallel gradient boosting, designed to circumvent all these limitations. The core idea is to use a common descent direction for all training observations. By doing so, only one base model is needed at each iteration, regardless of the number of targets, which allows for considerable performance gains. We establish sufficient conditions for the convergence of the algorithm, whose practical use is introduced via the multiple quantile regression setting. We show that in such a setting, it provides predictions of similar quality to state-of-the-art boosting libraries such as XGBoost, while being faster by several orders of magnitude. Then, we evaluate the properties of the resulting conditional distribution estimator, which is shown empirically to outperform other nonparametric and semiparametric estimators, especially in high-dimensional settings and in the presence of mixed and/or missing covariates.
ER -