A Distributional Optimisation Perspective on Combining Models in Deep Learning
Abstract
Combining predictions from different models can improve performance at machine learning tasks, but the training of the individual models and the rule used to combine them are typically chosen separately, and by ad hoc means. Recent advances in distributional optimisation (i.e. where the optimisation occurs over the set of probability distributions) offer an opportunity for principled joint training, viewing the collection of models as a discrete distribution whose support points are to be optimised, but the potential of these methods is not well-understood. In this paper we (1) cast two standard combination strategies - ensembles and low-rank adapter averaging - as entropy-regularised distributional optimisation, observing that the resulting objective is convex in the ensemble case but not in the adapter-averaging case, so that existing convergence guarantees for mean field Langevin dynamics transfer only to the former; (2) assess existing and novel algorithms for this task, including a functional variant of variational gradient descent; and (3) report an empirical study spanning synthetic classification tasks and fine-tuning of large language models on a commonsense reasoning benchmark.
Related
- Online Inference in Distributional Temporal-Difference Learning
- Distributional Balancing with Machine Learning for Clinical Trial Augmentation Using Real-World Data
- Online Inference for Quantile Distributional Reinforcement Learning
- Turtles All the Way Down: Parameter Sensitivity in Metaheuristic Hyperparameter Optimisation for Machine Learning Models
- Conditional Tensor Diffusion: Distributional Counterfactual Learning and Inference
BibTeX
@misc{wang2026distributional,
title = {{A Distributional Optimisation Perspective on Combining Models in Deep Learning}},
author = {Congye Wang and Yan Lin and Zheyang Shen and Matthew A. Fisher and Chris. J. Oates},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.24328},
eprint = {2609.24328},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.24328},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Distributional Optimisation Perspective on Combining Models in Deep Learning
AU - Congye Wang
AU - Yan Lin
AU - Zheyang Shen
AU - Matthew A. Fisher
AU - Chris. J. Oates
PY - 2026
DA - 2026/09/21
PB - arXiv
DO - 10.48550/arxiv.2609.24328
UR - https://arxiv.org/abs/2609.24328
AB - Combining predictions from different models can improve performance at machine learning tasks, but the training of the individual models and the rule used to combine them are typically chosen separately, and by ad hoc means. Recent advances in distributional optimisation (i.e. where the optimisation occurs over the set of probability distributions) offer an opportunity for principled joint training, viewing the collection of models as a discrete distribution whose support points are to be optimised, but the potential of these methods is not well-understood. In this paper we (1) cast two standard combination strategies - ensembles and low-rank adapter averaging - as entropy-regularised distributional optimisation, observing that the resulting objective is convex in the ensemble case but not in the adapter-averaging case, so that existing convergence guarantees for mean field Langevin dynamics transfer only to the former; (2) assess existing and novel algorithms for this task, including a functional variant of variational gradient descent; and (3) report an empirical study spanning synthetic classification tasks and fine-tuning of large language models on a commonsense reasoning benchmark.
ER -