A pre-registered prospective study of large language models predicting late-breaking cardiovascular trial results at ESC Congress 2026
- 1Seoul National University Bundang Hospital
Abstract
AimsIt is unknown whether large language models (LLMs) can predict a trials result at the design stage, from the information available when it is registered. We tested three LLMs prospectively on the Late-Breaking Science programme of the European Society of Cardiology (ESC) Congress 2026.
Methods and resultsOf 139 trials, 50 were selected by pre-specified criteria, and predictions were locked and registered on the Open Science Framework before the congress. Claude Fable 5, GPT-5.5 and Gemini 3.5 Flash, without web access, received each title and an investigator-compiled summary of the registered design. They gave the probability that the primary endpoint would be met, and a point estimate with an 80% prediction interval for a locked effect measure. Results were adjudicated from publications (19) or presenters slides (31). Of 48 scorable trials, 28 (58%) were positive. The area under the receiver operating characteristic curve was 0.80 (95% CI 0.66-0.92) for Claude, 0.79 (0.66-0.91) for GPT and 0.74 (0.59-0.87) for Gemini, with accuracies of 73%, 71% and 67%. Among 26 trials whose effect was reported exactly as locked, the direction was correct in 88% of predictions and the median multiplicative error of ratio estimates was about 15%. The 80% prediction intervals contained the observed effect in 73.1% (Claude), 84.6% (GPT) and 57.7% (Gemini).
ConclusionFrom the registered design alone, LLMs showed statistically significant discrimination between positive and negative trials and usually predicted the direction and approximate size of the effect. Such predictions might help trial design, and further studies are needed.
Related
- The cholesterol, high-density lipoprotein, and glucose index and its derivatives related to obesity and inflammation as novel metabolic markers for predicting new-onset cardiovascular-liver-metabolic multimorbidity: A prospective cohort study.
- Predicting cardiovascular disease in peritoneal dialysis patients using glycated proteomics and machine learning: A prospective multicentre study
- Development and validation of a novel metabolic-inflammatory health-related nomogram for predicting in-hospital mortality in older patients with type 2 diabetes mellitus combined with cardiovascular disease
- Prototyping OSA: A New Framework for Predicting Cardiovascular Outcomes and CPAP Benefit
- Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models
BibTeX
@misc{jeon2026preregistered,
title = {{A pre-registered prospective study of large language models predicting late-breaking cardiovascular trial results at ESC Congress 2026}},
author = {K.-H. Jeon and J.-S. Kwun and H.-W. Cho},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.23.26363773},
url = {https://www.medrxiv.org/content/10.64898/2026.09.23.26363773v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - A pre-registered prospective study of large language models predicting late-breaking cardiovascular trial results at ESC Congress 2026
AU - K.-H. Jeon
AU - J.-S. Kwun
AU - H.-W. Cho
PY - 2026
DA - 2026/09/23
PB - medRxiv
DO - 10.64898/2026.09.23.26363773
UR - https://www.medrxiv.org/content/10.64898/2026.09.23.26363773v1
AB - AimsIt is unknown whether large language models (LLMs) can predict a trials result at the design stage, from the information available when it is registered. We tested three LLMs prospectively on the Late-Breaking Science programme of the European Society of Cardiology (ESC) Congress 2026. Methods and resultsOf 139 trials, 50 were selected by pre-specified criteria, and predictions were locked and registered on the Open Science Framework before the congress. Claude Fable 5, GPT-5.5 and Gemini 3.5 Flash, without web access, received each title and an investigator-compiled summary of the registered design. They gave the probability that the primary endpoint would be met, and a point estimate with an 80% prediction interval for a locked effect measure. Results were adjudicated from publications (19) or presenters slides (31). Of 48 scorable trials, 28 (58%) were positive. The area under the receiver operating characteristic curve was 0.80 (95% CI 0.66-0.92) for Claude, 0.79 (0.66-0.91) for GPT and 0.74 (0.59-0.87) for Gemini, with accuracies of 73%, 71% and 67%. Among 26 trials whose effect was reported exactly as locked, the direction was correct in 88% of predictions and the median multiplicative error of ratio estimates was about 15%. The 80% prediction intervals contained the observed effect in 73.1% (Claude), 84.6% (GPT) and 57.7% (Gemini). ConclusionFrom the registered design alone, LLMs showed statistically significant discrimination between positive and negative trials and usually predicted the direction and approximate size of the effect. Such predictions might help trial design, and further studies are needed.
ER -