Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models
- 1Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America.
- 2Microsoft Research, Redmond, Washington, United States of America.
Abstract
Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs’ alignment with the brain can be mostly accounted for by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be accounted for by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features. Author summary: Modern AI systems trained on large amounts of speech, known as speech foundation models, are remarkably good at predicting how the human brain responds when people listen to spoken language. This has led many researchers to treat them as computational models of speech perception. But because their internal workings are opaque, it has been unclear which aspects of speech they actually capture, and why they align so well with brain activity. In this study, we used intracranial brain recordings from people listening to a podcast to ask a simple question: how much of what these AI models share with the brain can be explained by familiar, interpretable features of speech, such as its acoustic shape, its phonemes, its grammar, and its meaning? We found that most of the brain-model alignment can in fact be traced back to these simple features. We also found that as information moves through the layers of these models, low-level acoustic detail is gradually traded for higher-level meaning, and that the models’ grasp of brain-relevant meaning grows with both model size and context length. Combining the AI models with interpretable features yielded better and more transparent models of how the brain processes speech.
Related
- Dual-graph deconvolution of enzyme fitness: an interpretable framework to identify and explain catalytic modulators
- Surprisal contributes little beyond contextual embeddings in high-gamma ECoG encoding
- Setting the SCENE for Interpretable Cell-Gene Embeddings in Single-Cell RNA-seq
- Img2EEG: A Scalable and Interpretable Encoding Framework for Simulating Human EEG Responses to Visual Inputs
- BIRDNet: Mining and Encoding Boolean Implication Knowledge Graphs as Interpretable Deep Neural Networks
BibTeX
@article{shimizu2026interpretable,
title = {{Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models}},
author = {Riki Shimizu and Richard J Antonello and Chandan Singh and Nima Mesgarani},
journal = {PLOS Computational Biology},
year = {2026},
month = oct,
doi = {10.1371/journal.pcbi.1014906},
url = {https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1014906},
note = {CC BY},
}
RIS
TY - JOUR
TI - Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models
AU - Riki Shimizu
AU - Richard J Antonello
AU - Chandan Singh
AU - Nima Mesgarani
PY - 2026
DA - 2026/10/01
JO - PLOS Computational Biology
DO - 10.1371/journal.pcbi.1014906
UR - https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1014906
AB - Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs’ alignment with the brain can be mostly accounted for by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be accounted for by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features. Author summary: Modern AI systems trained on large amounts of speech, known as speech foundation models, are remarkably good at predicting how the human brain responds when people listen to spoken language. This has led many researchers to treat them as computational models of speech perception. But because their internal workings are opaque, it has been unclear which aspects of speech they actually capture, and why they align so well with brain activity. In this study, we used intracranial brain recordings from people listening to a podcast to ask a simple question: how much of what these AI models share with the brain can be explained by familiar, interpretable features of speech, such as its acoustic shape, its phonemes, its grammar, and its meaning? We found that most of the brain-model alignment can in fact be traced back to these simple features. We also found that as information moves through the layers of these models, low-level acoustic detail is gradually traded for higher-level meaning, and that the models’ grasp of brain-relevant meaning grows with both model size and context length. Combining the AI models with interpretable features yielded better and more transparent models of how the brain processes speech.
SN - 1553-7358
SN - 1553-734X
ER -