Exploratory Evaluation of the Statistical Fidelity of Large Language Model-Generated Synthetic Rehabilitation Data for Patients with Hip Fracture
- 1Department of Rehabilitation, Mie University Hospital, 2-174 Edobashi, Tsu 514-8507, Japan
Abstract
Background/Objectives: Synthetic data generated from aggregate statistics may support preliminary clinical research when patient-level data access is restricted. However, statistical similarity does not establish clinical validity, analytical utility, or privacy protection. We evaluated three datasets independently generated by large language models (LLMs) for patients with hip fractures. Methods: Of 26,637 hospitalization records, 12,130 complete eligible records were identified, and 1000 were reproducibly sampled. Aggregate statistics for 12 variables were provided to three LLMs, each generating one 1000-record dataset. We assessed fidelity to prompted marginal targets, preservation of unsupplied distributional and correlation structures, logical plausibility, and performance in an exploratory regression task predicting discharge Barthel Index scores. Results: Mean absolute standardized mean differences (SMDs) were 0.164, 0.027, and 0.001 for GPT, Claude, and Gemini, respectively. Mean absolute differences across 36 correlations were 0.090, 0.113, and 0.105, respectively; maximum correlation discrepancies were 0.411, 0.713, and 0.262. In train-on-synthetic/test-on-real evaluations, R2 values were 0.167, −0.080, and 0.146, respectively, compared with 0.262 for five-fold cross-validation on real data. Conclusions: The three individual LLM-generated datasets reproduced prompted marginal targets with varying fidelity but did not consistently preserve unsupplied multivariable structures or achieve comparable task-specific performance. These exploratory findings do not establish definitive model rankings, demonstrate clinical validity or privacy protection, or support replacing real patient-level data with synthetic data.
Related
- Composite inflammatory indices and incident lung cancer risk in an exploratory statistical mediation analysis
- Integrating large language model-generated culturally adapted verbal suggestions into music therapy for university students: A pilot randomized exploratory study
- Citation Fabrication and Accuracy in AI-Generated Psychiatric Research: A Comparative Evaluation of Four Contemporary Large Language Models
- Nutrient Calculation Accuracy, Expert-Rated Clinical Appropriateness and Safety Indicators, and Reproducibility of Large Language Model-Generated Diet Plans for Polyendocrine Metabolic Ovarian Syndrome: A Real-Case-Based Evaluation
- Application of spectroscopic and multispectral imaging technologies on the assessment of ready-to-eat pineapple quality: A performance evaluation study of machine learning models generated from two commercial data analytics tools
BibTeX
@article{momosaki2026exploratory,
title = {{Exploratory Evaluation of the Statistical Fidelity of Large Language Model-Generated Synthetic Rehabilitation Data for Patients with Hip Fracture}},
author = {Ryo Momosaki and Kenta Ushida and Yoshinori Yamamoto and Akio Shimizu},
journal = {Journal of Clinical Medicine},
year = {2026},
month = nov,
doi = {10.3390/jcm15218700},
url = {https://www.mdpi.com/2077-0383/15/21/8700},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Exploratory Evaluation of the Statistical Fidelity of Large Language Model-Generated Synthetic Rehabilitation Data for Patients with Hip Fracture
AU - Ryo Momosaki
AU - Kenta Ushida
AU - Yoshinori Yamamoto
AU - Akio Shimizu
PY - 2026
DA - 2026/11/07
JO - Journal of Clinical Medicine
DO - 10.3390/jcm15218700
UR - https://www.mdpi.com/2077-0383/15/21/8700
AB - Background/Objectives: Synthetic data generated from aggregate statistics may support preliminary clinical research when patient-level data access is restricted. However, statistical similarity does not establish clinical validity, analytical utility, or privacy protection. We evaluated three datasets independently generated by large language models (LLMs) for patients with hip fractures. Methods: Of 26,637 hospitalization records, 12,130 complete eligible records were identified, and 1000 were reproducibly sampled. Aggregate statistics for 12 variables were provided to three LLMs, each generating one 1000-record dataset. We assessed fidelity to prompted marginal targets, preservation of unsupplied distributional and correlation structures, logical plausibility, and performance in an exploratory regression task predicting discharge Barthel Index scores. Results: Mean absolute standardized mean differences (SMDs) were 0.164, 0.027, and 0.001 for GPT, Claude, and Gemini, respectively. Mean absolute differences across 36 correlations were 0.090, 0.113, and 0.105, respectively; maximum correlation discrepancies were 0.411, 0.713, and 0.262. In train-on-synthetic/test-on-real evaluations, R2 values were 0.167, −0.080, and 0.146, respectively, compared with 0.262 for five-fold cross-validation on real data. Conclusions: The three individual LLM-generated datasets reproduced prompted marginal targets with varying fidelity but did not consistently preserve unsupplied multivariable structures or achieve comparable task-specific performance. These exploratory findings do not establish definitive model rankings, demonstrate clinical validity or privacy protection, or support replacing real patient-level data with synthetic data.
SN - 2077-0383
ER -