The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation.
- 1School of Engineering, University of Edinburgh, Edinburgh, UK.
- 2Institute for Analytics and Data Science, University of Essex, Colchester, UK.
- 3Institute for Social and Economic Research, University of Essex, Colchester, UK.
Abstract
ML is playing an increasingly crucial role in estimating causal effects of treatments on outcomes from observational data. Many ML methods ('causal estimators') have been proposed for this task. All of these methods, as with any ML approach, require extensive hyperparameter tuning. For non-causal predictive tasks, there is a consensus on the choice of tuning metrics (e.g. mean squared error), making it simple to compare models. However, for causal inference tasks, such a consensus is yet to be reached, making a any comparison of causal models difficult. On top of that, there is no ideal metric on which to tune causal estimators, so one must rely on proxies. Furthermore, the fact that model selection in causal inference involves multiple components (causal estimator, ML regressor, hyperparameters, metric), complicates the issue even further. In order to evaluate the importance of each component, we perform an extensive empirical study on their combination. Our experimental setup involves many commonly used causal estimators, regressors ('base learners' henceforth) and metrics applied to four well-known causal inference benchmark datasets. Our results show that hyperparameter tuning increased the probability of reaching state-of-the-art performance in average ([Formula: see text]) and individualised ([Formula: see text]) effect estimation with only commonly used estimators. We also show that the performance of standard metrics can be inconsistent across different scenarios. Our findings highlight the need for further research to establish whether metrics uniformly capable of state-of-the-art performance in causal model evaluation can be found.
Related
- Distributed Grid Search for Hyperparameter Tuning
- Statistically Valid Post-Training Hyperparameter Selection: From Tuning to Guarantees
- HyperMC: Multi-Fidelity Hyperparameter Tuning for Stochastic Gradient MCMC
- Accurate Density of States Estimation: A Comparative Study
- Keeping Score: Adaptive, Tuning-Free Loss Weighting for Score-Augmented Neural Ratio Estimation
BibTeX
@article{machlanski2026challenges,
title = {{The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation.}},
author = {Damian Machlanski and Spyridon Samothrakis and Paul Clarke},
journal = {Machine learning},
year = {2026},
month = oct,
doi = {10.1007/s10994-026-07140-9},
url = {https://europepmc.org/article/MED/42892864},
note = {CC BY},
}
RIS
TY - JOUR
TI - The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation.
AU - Damian Machlanski
AU - Spyridon Samothrakis
AU - Paul Clarke
PY - 2026
DA - 2026/10/16
JO - Machine learning
DO - 10.1007/s10994-026-07140-9
UR - https://europepmc.org/article/MED/42892864
AB - ML is playing an increasingly crucial role in estimating causal effects of treatments on outcomes from observational data. Many ML methods ('causal estimators') have been proposed for this task. All of these methods, as with any ML approach, require extensive hyperparameter tuning. For non-causal predictive tasks, there is a consensus on the choice of tuning metrics (e.g. mean squared error), making it simple to compare models. However, for causal inference tasks, such a consensus is yet to be reached, making a any comparison of causal models difficult. On top of that, there is no ideal metric on which to tune causal estimators, so one must rely on proxies. Furthermore, the fact that model selection in causal inference involves multiple components (causal estimator, ML regressor, hyperparameters, metric), complicates the issue even further. In order to evaluate the importance of each component, we perform an extensive empirical study on their combination. Our experimental setup involves many commonly used causal estimators, regressors ('base learners' henceforth) and metrics applied to four well-known causal inference benchmark datasets. Our results show that hyperparameter tuning increased the probability of reaching state-of-the-art performance in average ([Formula: see text]) and individualised ([Formula: see text]) effect estimation with only commonly used estimators. We also show that the performance of standard metrics can be inconsistent across different scenarios. Our findings highlight the need for further research to establish whether metrics uniformly capable of state-of-the-art performance in causal model evaluation can be found.
SN - 0885-6125
SN - 1573-0565
ER -