REALM: Regime-Switching, Explainable, and Activation-Induced Linear Models
Abstract
Deep ReLU networks are piecewise-affine mappings that partition the input space into cells, each characterized by a distinct activation pattern. This structure motivates fitting a local linear model within each cell to preserve predictive accuracy while improving interpretability. The challenge is to identify regimes that are stable, data-adaptive, and easy to explain. We propose REALM, a mixture of linear models whose regimes are induced by neural activation patterns. Because the number of activation cells in a deep neural network (DNN) can grow rapidly with depth, we first distill a deep teacher into a wide, shallow student network (WSSN), then binarize and cluster its hidden-layer activations to define the regimes and fit a linear model within each regime. Since the regimes are discovered from internal structure, the router does not carry the predictive burden. To make regime assignment interpretable, we train a multiclass logistic regression, the explanatory gate, to reproduce the regime assignments. The two-level structure is interpretable at both stages in terms of raw tabular or learned convolutional features: the gate identifies features that determine regime assignments, while the linear models identify features that drive predictions within each regime. We analyze an idealized setting that illustrates a trade-off between partition complexity and stability: as the number of regimes grows, finer partitions can improve approximation but may reduce regime-assignment stability. Experiments on tabular and image datasets show that REALM achieves competitive predictive performance relative to other DNN-guided mixture surrogates and inherently interpretable models while producing stable regime-level explanations.
Related
- Mathematical Classification of Switching Types and Hierarchical Structures: Three Switching Types, Six Hierarchical Structures, Coefficient of Variation Lower Bound, and Discretization Plateau Effect in the k-Regime Linear Switching Model
- Stochastic linear-quadratic control problem for regime-switching jump-diffusion system and its application to finance
- Exponential Convergence of the Sinkhorn Algorithm for the Schrödinger Bridge with Regime Switching
- A note on functional non-central limit theorem for periodic regime-switching stable-like processes
- Invariant probability measures for path-dependent regime-switching processes
BibTeX
@misc{cheng2026realm,
title = {{REALM: Regime-Switching, Explainable, and Activation-Induced Linear Models}},
author = {Xiaoran Cheng and Sen Na and Jia Li},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.32141},
eprint = {2609.32141},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.32141},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - REALM: Regime-Switching, Explainable, and Activation-Induced Linear Models
AU - Xiaoran Cheng
AU - Sen Na
AU - Jia Li
PY - 2026
DA - 2026/09/26
PB - arXiv
DO - 10.48550/arxiv.2609.32141
UR - https://arxiv.org/abs/2609.32141
AB - Deep ReLU networks are piecewise-affine mappings that partition the input space into cells, each characterized by a distinct activation pattern. This structure motivates fitting a local linear model within each cell to preserve predictive accuracy while improving interpretability. The challenge is to identify regimes that are stable, data-adaptive, and easy to explain. We propose REALM, a mixture of linear models whose regimes are induced by neural activation patterns. Because the number of activation cells in a deep neural network (DNN) can grow rapidly with depth, we first distill a deep teacher into a wide, shallow student network (WSSN), then binarize and cluster its hidden-layer activations to define the regimes and fit a linear model within each regime. Since the regimes are discovered from internal structure, the router does not carry the predictive burden. To make regime assignment interpretable, we train a multiclass logistic regression, the explanatory gate, to reproduce the regime assignments. The two-level structure is interpretable at both stages in terms of raw tabular or learned convolutional features: the gate identifies features that determine regime assignments, while the linear models identify features that drive predictions within each regime. We analyze an idealized setting that illustrates a trade-off between partition complexity and stability: as the number of regimes grows, finer partitions can improve approximation but may reduce regime-assignment stability. Experiments on tabular and image datasets show that REALM achieves competitive predictive performance relative to other DNN-guided mixture surrogates and inherently interpretable models while producing stable regime-level explanations.
ER -