Same pipelines, different answers: how evaluation design shapes EEG seizure prediction
- 1Auckland university of technology
Abstract
Seizure-prediction studies on scalp electroencephalography (EEG) often report accuracies that would be clinically useful, yet the score a model receives also depends on how its test data were separated from its training data. We kept five published pipelines fixed, from a spectral baseline (power spectral density with linear discriminant analysis, PSD+LDA) to a transformer, and changed only how they were evaluated on the CHB-MIT database. When whole subjects were held out in 20 repeated partitions, every pipeline scored close to the 0.5 reference (mean area under the ROC curve, AUC, 0.527-0.553), and the five pipelines differed less from one another than each varied between partitions. Within patients, PSD+LDA reached an AUC of 0.815 when windows were allocated at random but 0.572 when each preictal block was kept intact and later seizures were held out for testing. Tracing every window back to its recording showed that an index-based within-patient split had let raw EEG samples cross the train-test boundary in all ten eligible cases. Thresholds tuned to allow 1.67% false positives on validation data did not hold that rate on new subjects, and at those thresholds only 2-6% of preictal windows were detected. A zero-shot test of the baseline on the Siena database gave an AUC of 0.480. In this benchmark, the choice of evaluation changed the answer far more than the choice of architecture. We release the window manifests, partitions and predictions so that new models can be tested on the same terms. All results concern retrospective window classification, not continuous alarms.
Related
- Epilepsy domain adapted large language models enhance epileptogenic zone prediction from seizure semiology
- Comparing ChatGPT and physicians’ answers to endometriosis questions on Reddit: A blind expert evaluation
- Fahr Syndrome Identified During Evaluation of Recurrent Seizure-Like Activity in a Middle-Aged Woman.
- Response Quality of AI-Generated Answers to Orthodontic Patient Concerns Across Empathy, Accuracy, Comprehensiveness, Personalisation, Safety, and Patient-Centeredness: A Cross-Model, Bilingual Evaluation of Claude, GPT-4o, and Gemini.
- Countering Neural Activity Drift: Sustained Long-term Seizure Prediction Using an Evolutionary Machine-Learning Framework on Continuous Intracranial EEG
BibTeX
@misc{song2026same,
title = {{Same pipelines, different answers: how evaluation design shapes EEG seizure prediction}},
author = {z. song and M. Pedersen and D. Henare},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.28.26366763},
url = {https://www.medrxiv.org/content/10.64898/2026.10.28.26366763v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Same pipelines, different answers: how evaluation design shapes EEG seizure prediction
AU - z. song
AU - M. Pedersen
AU - D. Henare
PY - 2026
DA - 2026/10/28
PB - medRxiv
DO - 10.64898/2026.10.28.26366763
UR - https://www.medrxiv.org/content/10.64898/2026.10.28.26366763v1
AB - Seizure-prediction studies on scalp electroencephalography (EEG) often report accuracies that would be clinically useful, yet the score a model receives also depends on how its test data were separated from its training data. We kept five published pipelines fixed, from a spectral baseline (power spectral density with linear discriminant analysis, PSD+LDA) to a transformer, and changed only how they were evaluated on the CHB-MIT database. When whole subjects were held out in 20 repeated partitions, every pipeline scored close to the 0.5 reference (mean area under the ROC curve, AUC, 0.527-0.553), and the five pipelines differed less from one another than each varied between partitions. Within patients, PSD+LDA reached an AUC of 0.815 when windows were allocated at random but 0.572 when each preictal block was kept intact and later seizures were held out for testing. Tracing every window back to its recording showed that an index-based within-patient split had let raw EEG samples cross the train-test boundary in all ten eligible cases. Thresholds tuned to allow 1.67% false positives on validation data did not hold that rate on new subjects, and at those thresholds only 2-6% of preictal windows were detected. A zero-shot test of the baseline on the Siena database gave an AUC of 0.480. In this benchmark, the choice of evaluation changed the answer far more than the choice of architecture. We release the window manifests, partitions and predictions so that new models can be tested on the same terms. All results concern retrospective window classification, not continuous alarms.
ER -