PreprintCC BY 4.0arXiv
Just add noise: Debiasing tree-based variable importance in mixed data
Abstract
Variable importance scores from tree-based methods such as random forests favor continuous predictors over categorical ones. We present a theoretical analysis of this bias and propose a simple remedy: add a small amount of noise to each categorical predictor. The correction is demonstrated on a variety of simulated and real-world datasets and combined with integrated path stability selection to perform variable selection with false discovery control for mixed data.
Related
- Measuring Variable Importance Using Proportional Inclusive Marginal Effects
- Curvature-Dependent Path Concentration in Stochastic Fast-Slow Systems with Noise on the Slow Variable
- lepidoptera: A Domain-Agnostic Engine for Causal Simulation with Mixed Variable Types, Vine Copula Dependence, and Individual-Level Counterfactual Propagation
- NomadBBO: A direct-search framework for constrained mixed-variable blackbox optimization in Python
- Multiplicity of weak solutions for mixed variable exponent triharmonic problems with hardy potentials
BibTeX
@misc{li2026just,
title = {{Just add noise: Debiasing tree-based variable importance in mixed data}},
author = {Jiahe Li and Omar Melikechi},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.14083},
eprint = {2609.14083},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.14083},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Just add noise: Debiasing tree-based variable importance in mixed data
AU - Jiahe Li
AU - Omar Melikechi
PY - 2026
DA - 2026/09/12
PB - arXiv
DO - 10.48550/arxiv.2609.14083
UR - https://arxiv.org/abs/2609.14083
AB - Variable importance scores from tree-based methods such as random forests favor continuous predictors over categorical ones. We present a theoretical analysis of this bias and propose a simple remedy: add a small amount of noise to each categorical predictor. The correction is demonstrated on a variety of simulated and real-world datasets and combined with integrated path stability selection to perform variable selection with false discovery control for mixed data.
ER -