Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again
Abstract
Recent work has shown that well-optimized individual decision trees can match complex black box models in some settings, primarily in noisy domains. For the remaining settings, however, complex ensembled compositions of trees often achieve higher accuracy at the cost of interpretability, leaving practitioners with difficult modeling decisions along an accuracy-interpretability tradeoff. Ideally, we would like to classify as much of the data as possible with one or a small number of trees, achieving interpretability for most samples while maintaining state-of-the-art accuracy. We introduce Multistage Defer Trees: a sequence of sparse decision trees that each make predictions for most samples, while deferring a small proportion to the next tree in the sequence or, ultimately, to a black box. We demonstrate that we can train this model class to match the performance of complex tree-based ensembles while routing most samples through only one or a small number of sparse decision trees. We discuss a range of techniques for training these models while maintaining simplicity. Our method expands the accuracy--interpretability frontier in settings where single-tree methods remain insufficient, demonstrating that even when complex models are necessary, they need not be fully opaque.
Related
- Scentree: a framework for generating scenario trees for multistage stochastic programming
- Residual Deep Learning for Realized Volatility Using a Hybrid HAR-LSTM Model with ARIMA Features and SHAP Interpretability
- Online Learning-to-Defer with Varying Experts
- Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents
- The real option value of multistage agricultural research for development
BibTeX
@misc{heile2026multistage,
title = {{Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again}},
author = {Zakk Heile and Hayden McTavish and Margo Seltzer and Cynthia Rudin},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.30995},
eprint = {2606.30995},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.30995},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again
AU - Zakk Heile
AU - Hayden McTavish
AU - Margo Seltzer
AU - Cynthia Rudin
PY - 2026
DA - 2026/09/08
PB - arXiv
DO - 10.48550/arxiv.2606.30995
UR - https://arxiv.org/abs/2606.30995
AB - Recent work has shown that well-optimized individual decision trees can match complex black box models in some settings, primarily in noisy domains. For the remaining settings, however, complex ensembled compositions of trees often achieve higher accuracy at the cost of interpretability, leaving practitioners with difficult modeling decisions along an accuracy-interpretability tradeoff. Ideally, we would like to classify as much of the data as possible with one or a small number of trees, achieving interpretability for most samples while maintaining state-of-the-art accuracy. We introduce Multistage Defer Trees: a sequence of sparse decision trees that each make predictions for most samples, while deferring a small proportion to the next tree in the sequence or, ultimately, to a black box. We demonstrate that we can train this model class to match the performance of complex tree-based ensembles while routing most samples through only one or a small number of sparse decision trees. We discuss a range of techniques for training these models while maintaining simplicity. Our method expands the accuracy--interpretability frontier in settings where single-tree methods remain insufficient, demonstrating that even when complex models are necessary, they need not be fully opaque.
ER -