When Prediction Error Is Not Enough: Evaluating Nuisance-Function Prediction for Causal Estimation

Abstract

Prediction error is widely used to evaluate nuisance-function estimators in causal inference, but its relationship with the performance of the resulting causal estimator is not well understood. We studied this question in a partially linear model using Monte Carlo simulations, comparing ordinary least squares (OLS), generalized additive models (GAMs), XGBoost, and Double Machine Learning with XGBoost (DML-XGBoost) with independent and with clustered observations. We evaluated nuisance-function prediction error, bias, RMSE, and 95% confidence interval coverage, and derived the second-order remainder through which errors in the exposure and outcome nuisance functions jointly determine the bias of the residualized estimator. When all methods estimated the causal effect from the same training sample, DML-XGBoost had the lowest RMSE and the highest coverage among the non-oracle methods, although XGBoost fitted to the full training sample had lower out-of-sample prediction error for both nuisance functions. GAMs had the lowest prediction error of all methods but were biased, with coverage of 0.55 with independent observations. The remainder reproduced the average bias of OLS, GAMs, and DML-XGBoost, but only half to three-quarters of the bias of XGBoost, whose residuals were formed on the data used to fit it. A joint-error measure based on the absolute cross-product of the two nuisance errors was only weakly associated with causal bias. Prediction error is useful for assessing nuisance-function estimation, but it should not be treated as a direct measure of the quality of the resulting causal estimator.

Optimal Hamiltonian Parameter Estimation in the Presence of Nuisance Parameters
Zhiyao Hu, Haidong Yuan, Liang Jiang +1
arXiv11 Sep 2026PreprintarXiv licenceFull textGeneral mathematics Quantum physics & quantum informationarXiv
Failure Prevention Based on Parameters Estimation and Prediction
Oleg Abramov
Reliability: Theory & Applications1 Sep 2026Peer-reviewedCC BY-NCFull textGeneral mathematicsDOAJ
Prediction-Powered Data Fusion for Treatment Effect Estimation
Yonghan Jung, Shu Yang
arXiv23 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Information, Estimation, and Prediction for Discrete Memoryless Channels via Poissonization
Abhiram Gorle, Tsachy Weissman
arXiv21 Oct 2026PreprintCC BY 4.0Full textGeneral mathematicsarXiv
A novel Weighted Exponential-Lindley distribution with Bayesian estimation under progressive Type-II censoring and a Bayesian-Calibrated Variational Autoencoder for survival prediction.
P Furqan Ahmed, V Sujatha
Frontiers in artificial intelligence7 Oct 2026Peer-reviewedCC BYFull textStatisticsEurope PMC

Cite

BibTeX RIS

BibTeX

@misc{cao2026when,
  title = {{When Prediction Error Is Not Enough: Evaluating Nuisance-Function Prediction for Causal Estimation}},
  author = {Cong Cao},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2609.00071},
  eprint = {2609.00071},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.00071},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - When Prediction Error Is Not Enough: Evaluating Nuisance-Function Prediction for Causal Estimation
AU  - Cong Cao
PY  - 2026
DA  - 2026/10/21
PB  - arXiv
DO  - 10.48550/arxiv.2609.00071
UR  - https://arxiv.org/abs/2609.00071
AB  - Prediction error is widely used to evaluate nuisance-function estimators in causal inference, but its relationship with the performance of the resulting causal estimator is not well understood. We studied this question in a partially linear model using Monte Carlo simulations, comparing ordinary least squares (OLS), generalized additive models (GAMs), XGBoost, and Double Machine Learning with XGBoost (DML-XGBoost) with independent and with clustered observations. We evaluated nuisance-function prediction error, bias, RMSE, and 95% confidence interval coverage, and derived the second-order remainder through which errors in the exposure and outcome nuisance functions jointly determine the bias of the residualized estimator. When all methods estimated the causal effect from the same training sample, DML-XGBoost had the lowest RMSE and the highest coverage among the non-oracle methods, although XGBoost fitted to the full training sample had lower out-of-sample prediction error for both nuisance functions. GAMs had the lowest prediction error of all methods but were biased, with coverage of 0.55 with independent observations. The remainder reproduced the average bias of OLS, GAMs, and DML-XGBoost, but only half to three-quarters of the bias of XGBoost, whose residuals were formed on the data used to fit it. A joint-error measure based on the absolute cross-product of the two nuisance errors was only weakly associated with causal bias. Prediction error is useful for assessing nuisance-function estimation, but it should not be treated as a direct measure of the quality of the resulting causal estimator.
ER  -

Abstract

Related

Cite