Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding
Abstract
We tested whether auditory-evoked EEG supports subject-independent five-vowel perception decoding when trial identity, model identity, prediction provenance, and participant-level inference are controlled within a single benchmark. We reconstructed Study 2 event tables from OpenNeuro ds006104 version 1.0.1 and analysed the consonant-vowel pair task. One-to-one marker-stimulus pairing yielded 3,840 independent trials; control-condition selection and artifact rejection retained 1,094 epochs from 16 participants and 61 EEG channels. Thirteen unique implementations were evaluated using leave-one-subject-out testing, with participant metrics reconstructed from 36,102 trial predictions across 33 complete prediction replicas. Random Forest was numerically highest at 21.474% balanced accuracy (95% participant-bootstrap interval, 19.526-23.482%; chance, 20%), but neither its participant-level tests nor any implementation survived correction across the 13-model family. Deep-model performance was close to chance, and several architectures showed substantial seed-dependent variation and low trial-label agreement. An exploratory MDM analysis comprising 9,616 genuine refits across training cohorts of 3-15 participants showed no monotonic performance gain. Within this dataset and protocol, evidence for reliable cross-subject five-vowel decoding is limited. The benchmark provides a reproducible chain from source rows to retained epochs, predictions, participant-level metrics, multiplicity-adjusted inference, and bounded diagnostic analyses.
Related
- AVID: A 5T fMRI Dataset for Benchmarking Auditory-induced Visual Mental Imagery Decoding
- Leakage-controlled benchmarking reveals generalization limits of deep learning for protein-ligand binding affinity prediction
- Leakage-Controlled Benchmarking of Variant Effect Predictors and Development of Gradient-Boosting Consensus Model
- Protify: A Benchmarking Platform Revealing Task Dependence and Homology Leakage in Protein Language Model Evaluation
- Moving Heads, Moving Sounds: Systematic Errors in Auditory Motion Perception
BibTeX
@misc{li2026leakageaudited,
title = {{Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding}},
author = {Xiaoyang Li and Zeyan Tao},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.00865},
eprint = {2605.00865},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.00865},
}
RIS
TY - UNPB
TI - Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding
AU - Xiaoyang Li
AU - Zeyan Tao
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2605.00865
UR - https://arxiv.org/abs/2605.00865
AB - We tested whether auditory-evoked EEG supports subject-independent five-vowel perception decoding when trial identity, model identity, prediction provenance, and participant-level inference are controlled within a single benchmark. We reconstructed Study 2 event tables from OpenNeuro ds006104 version 1.0.1 and analysed the consonant-vowel pair task. One-to-one marker-stimulus pairing yielded 3,840 independent trials; control-condition selection and artifact rejection retained 1,094 epochs from 16 participants and 61 EEG channels. Thirteen unique implementations were evaluated using leave-one-subject-out testing, with participant metrics reconstructed from 36,102 trial predictions across 33 complete prediction replicas. Random Forest was numerically highest at 21.474% balanced accuracy (95% participant-bootstrap interval, 19.526-23.482%; chance, 20%), but neither its participant-level tests nor any implementation survived correction across the 13-model family. Deep-model performance was close to chance, and several architectures showed substantial seed-dependent variation and low trial-label agreement. An exploratory MDM analysis comprising 9,616 genuine refits across training cohorts of 3-15 participants showed no monotonic performance gain. Within this dataset and protocol, evidence for reliable cross-subject five-vowel decoding is limited. The benchmark provides a reproducible chain from source rows to retained epochs, predictions, participant-level metrics, multiplicity-adjusted inference, and bounded diagnostic analyses.
ER -