Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale
Abstract
Simulation-based inference (SBI) provides a powerful way to fit mechanistic models to data, but offers little insight into which information in the data shapes the inferred parameter values. It could therefore benefit from robust and inexpensive interpretability methods. Learned weighting mechanisms such as feature gates provide a potential route to low-cost interpretability, although their interpretation assumes that predictive training reveals which inputs drive performance. We examined this assumption in a computational neuroscience setting, where neural posterior estimation (NPE) is used to infer four physiological parameters of a dynamical model of interacting neuronal populations (Jansen-Rit). We tested whether the learned gate weights captured feature importance by comparing them with a retrain-based ablation benchmark, in which each feature was removed in turn and the NPE retrained. We considered two jointly trained gates: independent sigmoid weights and competing softmax weights. If these reliably reflect feature importance, they should align with the ablation-based estimates. Across nine independent training seeds, however, gate-based and ablation-based importance were negatively correlated (Spearman's rho approximately -0.61 for sigmoid gating and -0.56 for softmax gating). The feature that ablation identified as most important, total signal power, was among the features the gate down-weighted most, despite gating costing negligible predictive accuracy. These results extend a pattern previously documented for attention mechanisms in natural language processing and for explainability methods in electroencephalography (EEG) deep learning to the simulation-based inference setting, and suggest caution when interpreting learned gate weights as measures of causal feature importance in scientific machine-learning pipelines.
Related
- Scalable phylodynamic inference of mutational and epistatic fitness effects with multi-feature birth-death models
- Compensation of hyperexcitability with simulation-based inference
- abcneuralnet: Simulation-based Inference with Bayesian Neural Networks and Explainable AI for Population Genomics Inference
- Deep learning for mass extinction detection on fossilized phylogenies: power, limitations and lessons for simulation-based birth–death inference
- Embodiment-aware control by inference over the operator: a simulation study
BibTeX
@misc{deshmukh2026gatebased,
title = {{Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale}},
author = {Shrivatsa Deshmukh and Sasha Brenner and Thomas R. Knösche and Nico Scherf},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.30364},
eprint = {2610.30364},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.30364},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Gate-Based Feature Importance in Simulation-Based Inference for EEG: A Cautionary Tale
AU - Shrivatsa Deshmukh
AU - Sasha Brenner
AU - Thomas R. Knösche
AU - Nico Scherf
PY - 2026
DA - 2026/10/21
PB - arXiv
DO - 10.48550/arxiv.2610.30364
UR - https://arxiv.org/abs/2610.30364
AB - Simulation-based inference (SBI) provides a powerful way to fit mechanistic models to data, but offers little insight into which information in the data shapes the inferred parameter values. It could therefore benefit from robust and inexpensive interpretability methods. Learned weighting mechanisms such as feature gates provide a potential route to low-cost interpretability, although their interpretation assumes that predictive training reveals which inputs drive performance. We examined this assumption in a computational neuroscience setting, where neural posterior estimation (NPE) is used to infer four physiological parameters of a dynamical model of interacting neuronal populations (Jansen-Rit). We tested whether the learned gate weights captured feature importance by comparing them with a retrain-based ablation benchmark, in which each feature was removed in turn and the NPE retrained. We considered two jointly trained gates: independent sigmoid weights and competing softmax weights. If these reliably reflect feature importance, they should align with the ablation-based estimates. Across nine independent training seeds, however, gate-based and ablation-based importance were negatively correlated (Spearman's rho approximately -0.61 for sigmoid gating and -0.56 for softmax gating). The feature that ablation identified as most important, total signal power, was among the features the gate down-weighted most, despite gating costing negligible predictive accuracy. These results extend a pattern previously documented for attention mechanisms in natural language processing and for explainability methods in electroencephalography (EEG) deep learning to the simulation-based inference setting, and suggest caution when interpreting learned gate weights as measures of causal feature importance in scientific machine-learning pipelines.
ER -