Surprisal contributes little beyond contextual embeddings in high-gamma ECoG encoding
- 1Soka University
Abstract
Surprisal and contextual embeddings are both derived from large language models and are widely used to predict neural responses during language comprehension, but it is unclear whether surprisal adds information beyond embeddings. We test this directly: does word surprisal improve out-of-sample prediction of high-gamma ECoG responses after GPT-2 XL contextual embeddings are included? Using public ECoG recordings from natural speech, we fit word-aligned ridge encoding models with baseline stimulus features, GPT-2 XL embeddings, and GPT-2 XL surprisal. Adding one surprisal predictor left held-out correlation essentially unchanged at the primary response window, and the increment stayed within an equivalence margin across alternative lags, residualized surprisal, and frequency-band targets; it did not uniformly meet the same criterion under banded ridge regularization or within labeled temporal subsets. This small increment suggests that surprisal does not act as an independent predictor. It is better understood as a compressed readout of the same broader predictive state that the embeddings already capture.
Related
- Electrophysiological correlates of contextual retrieval following virtual reality and desktop-based encoding.
- Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models
- Semantic Update as a Predictor of Reading Time: Moving Beyond Word-Level Surprisal
- Temporal Progression of Prior Precision, Surprisal, Model Updating and Tone Adaptation in Human Auditory Cortices
- Lexical surprisal drives pupil dilation during continuous story listening
BibTeX
@misc{sakuma2026surprisal,
title = {{Surprisal contributes little beyond contextual embeddings in high-gamma ECoG encoding}},
author = {T. Sakuma},
howpublished = {bioRxiv},
year = {2026},
month = jul,
doi = {10.64898/2026.07.10.737662},
url = {https://www.biorxiv.org/content/10.64898/2026.07.10.737662v2},
note = {CC BY},
}
RIS
TY - UNPB
TI - Surprisal contributes little beyond contextual embeddings in high-gamma ECoG encoding
AU - T. Sakuma
PY - 2026
DA - 2026/07/10
PB - bioRxiv
DO - 10.64898/2026.07.10.737662
UR - https://www.biorxiv.org/content/10.64898/2026.07.10.737662v2
AB - Surprisal and contextual embeddings are both derived from large language models and are widely used to predict neural responses during language comprehension, but it is unclear whether surprisal adds information beyond embeddings. We test this directly: does word surprisal improve out-of-sample prediction of high-gamma ECoG responses after GPT-2 XL contextual embeddings are included? Using public ECoG recordings from natural speech, we fit word-aligned ridge encoding models with baseline stimulus features, GPT-2 XL embeddings, and GPT-2 XL surprisal. Adding one surprisal predictor left held-out correlation essentially unchanged at the primary response window, and the increment stayed within an equivalence margin across alternative lags, residualized surprisal, and frequency-band targets; it did not uniformly meet the same criterion under banded ridge regularization or within labeled temporal subsets. This small increment suggests that surprisal does not act as an independent predictor. It is better understood as a compressed readout of the same broader predictive state that the embeddings already capture.
ER -