When Are Concept Bottleneck Model Explanations Faithful and Compact?
Abstract
Concept bottleneck models (CBMs) are neural classifiers that allow to explain their decisions via high-level concepts, potentially enabling understanding, steering and debugging. However, their explanations are often derived heuristically. Building on formal explainability, we argue they should also be faithful, i.e., not misreport which concepts actually matter. We show that, for widespread CBM architectures, including recent VLM-based variants, faithful explanations must include all concepts in the bottleneck, compromising interpretability when this is large. This result applies to both heuristic and faithful-by-construction formal explanations. To encourage the existence of compact faithful explanations, we suggest i) modeling concepts probabilistically as binary or categorical random variables (rather than logits), and ii) employing per-concept training-time sparsification via group lasso (rather than regular elastic net). We also extend algorithms from formal explainability to CBMs, and show they outperform natural heuristics in terms of guarantees and explanation size. Overall, our work warns against naive interpretability claims and provides formal conditions and practical strategies for ensuring CBMs are as interpretable as advertised.
Related
- Hoeffding Concept Bottleneck Models with Applications to Overhead Images
- A Faithful Action of the Mapping Class Group Mod(Sg) on the (2g−1)-Sphere
- Explanations, Prompts, and Formalizations: Arguments for New Norms in LLM-Enabled Mathematical Research
- The Burau representation of the braid group is faithful for n = 4
- A fully faithful p-adic Riemann--Hilbert functor for filtered coherent D-modules
BibTeX
@misc{teso2026when,
title = {{When Are Concept Bottleneck Model Explanations Faithful and Compact?}},
author = {Stefano Teso and Emanuele Marconato and Steve Azzolin and Antonio Vergari},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.06285},
eprint = {2610.06285},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.06285},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - When Are Concept Bottleneck Model Explanations Faithful and Compact?
AU - Stefano Teso
AU - Emanuele Marconato
AU - Steve Azzolin
AU - Antonio Vergari
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.06285
UR - https://arxiv.org/abs/2610.06285
AB - Concept bottleneck models (CBMs) are neural classifiers that allow to explain their decisions via high-level concepts, potentially enabling understanding, steering and debugging. However, their explanations are often derived heuristically. Building on formal explainability, we argue they should also be faithful, i.e., not misreport which concepts actually matter. We show that, for widespread CBM architectures, including recent VLM-based variants, faithful explanations must include all concepts in the bottleneck, compromising interpretability when this is large. This result applies to both heuristic and faithful-by-construction formal explanations. To encourage the existence of compact faithful explanations, we suggest i) modeling concepts probabilistically as binary or categorical random variables (rather than logits), and ii) employing per-concept training-time sparsification via group lasso (rather than regular elastic net). We also extend algorithms from formal explainability to CBMs, and show they outperform natural heuristics in terms of guarantees and explanation size. Overall, our work warns against naive interpretability claims and provides formal conditions and practical strategies for ensuring CBMs are as interpretable as advertised.
ER -