Proximal Empirical Bayes for Sparse Regression with Posterior Decision Support
Abstract
In this work, we revisit estimation and variable selection in sparse Gaussian regression, possibly under affine constraints. We adopt a middle-ground approach between optimisation and fully hierarchical Bayesian methods, preserving much of the computational efficiency of the former while adding posterior uncertainty quantification. In particular, we develop an empirical Bayes framework that separates shrinkage calibration from posterior-informed selection while using the same log-concave model throughout. The resulting workflow is modular: different methods can be used for noise-variance estimation, shrinkage calibration and posterior sampling. At the same time, these tasks are linked, and numerical choices made at one stage can propagate through the workflow and ultimately affect the decision about which variables to retain. We investigate in detail one instance of the framework based on a Laplace prior, with the amount of shrinkage determined by stochastic approximation proximal gradient. The proximal machinery used for this calibration also supports posterior simulation. We further show how the empirical Bayes calibration should be modified when homogeneous constraints reduce the number of free parameters. Nonhomogeneous affine information is instead incorporated softly; when this information is strong, posterior sampling can become poorly conditioned, and we develop a preconditioning strategy to improve mixing and posterior exploration. Synthetic experiments examine how scale calibration, posterior run length and sampler conditioning affect estimation, uncertainty quantification and the final sparse decision. On a diabetes dataset, posterior uncertainty agrees closely with established Bayesian analyses, while the final decision provides a sparser practical summary.
Related
- On The Linear Convergence of Bregman Proximal Gradient Methods with Applications to Kullback--Leibler regression
- Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study
- Flow Annealing Posterior Sampling for Function-Space Regression and Inverse Problems
- Comparative Performance Analysis of Optimization Solvers for Linear Support Vector Machines: Impact of Preconditioning and Dimensionality An Empirical Study Using the Benchopt Framework
- Bias-Corrected Empirical-Likelihood Inference for Single-Index Expectile Regression Under Missing-at-Random Responses
BibTeX
@misc{roxanas2026proximal,
title = {{Proximal Empirical Bayes for Sparse Regression with Posterior Decision Support}},
author = {Dimitrios Roxanas},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.39983},
eprint = {2609.39983},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.39983},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Proximal Empirical Bayes for Sparse Regression with Posterior Decision Support
AU - Dimitrios Roxanas
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2609.39983
UR - https://arxiv.org/abs/2609.39983
AB - In this work, we revisit estimation and variable selection in sparse Gaussian regression, possibly under affine constraints. We adopt a middle-ground approach between optimisation and fully hierarchical Bayesian methods, preserving much of the computational efficiency of the former while adding posterior uncertainty quantification. In particular, we develop an empirical Bayes framework that separates shrinkage calibration from posterior-informed selection while using the same log-concave model throughout. The resulting workflow is modular: different methods can be used for noise-variance estimation, shrinkage calibration and posterior sampling. At the same time, these tasks are linked, and numerical choices made at one stage can propagate through the workflow and ultimately affect the decision about which variables to retain. We investigate in detail one instance of the framework based on a Laplace prior, with the amount of shrinkage determined by stochastic approximation proximal gradient. The proximal machinery used for this calibration also supports posterior simulation. We further show how the empirical Bayes calibration should be modified when homogeneous constraints reduce the number of free parameters. Nonhomogeneous affine information is instead incorporated softly; when this information is strong, posterior sampling can become poorly conditioned, and we develop a preconditioning strategy to improve mixing and posterior exploration. Synthetic experiments examine how scale calibration, posterior run length and sampler conditioning affect estimation, uncertainty quantification and the final sparse decision. On a diabetes dataset, posterior uncertainty agrees closely with established Bayesian analyses, while the final decision provides a sparser practical summary.
ER -