Quality assessment of synthetic data in healthcare: a critical appraisal with a focus on subgroup bias.
7 affiliations
- 1Biomed Stadius, Department of Electrical Engineering, KU Leuven, Leuven, Belgium. [email protected].
- 2Department of Research & Development, Integraal Kankercentrum Nederland (IKNL), Utrecht, Netherlands.
- 3Biomed Stadius, Department of Electrical Engineering, KU Leuven, Leuven, Belgium.
- 4IT EMEA Departmental Janssen Pharmaceutica, Beerse, Belgium.
- 5Cancer Registry of Norway, Norwegian Institute of Public Health, Oslo, Norway.
- 6Janssen Global Services, LLC, Raritan, USA.
- 7Department of Development and Regeneration, KU Leuven, Leuven, Belgium.
Abstract
Background
The creation of synthetic data in healthcare research has emerged as a compelling solution to address challenges related to data privacy, in case of data-sharing and data scarceness for training machine learning models.Objective
The current paper aims to provide a critical landscape analysis of synthetic data quality metrics used in healthcare, mapping the state of the field, examining how current fidelity and utility scores often fail to capture weaknesses that appear in specific patient subgroups, and illustrating these blind spots with a lung-cancer case study. We specifically set out to highlight the need for assessing synthetic datasets not only at the aggregate level but also across clinically meaningful subgroups, to ensure reliability and fairness in downstream applications.Methods
By examining the landscape of synthetic data adoption in healthcare, we highlight the methodological and utility considerations that shape its integration into the research ecosystem. We elucidate how synthetic data can expedite research initiatives, support data-driven decision-making, and facilitate innovative methodologies while safeguarding sensitive patient information. Conversely, while many papers focus solely on the advantages of synthetic data, we aim to highlight also the potential constraints of using synthetic data in a real-world case involving lung cancer patient level data. Our analysis centers on identifying situations where synthetic data might introduce biases or inaccuracies that hinder the generation of meaningful clinical insights.Results
We demonstrate the limitations of using synthetic data in a lung cancer case study, especially when dealing with small subgroups from the original dataset. While high values for fidelity and utility metrics are achieved when considering the entire synthetic dataset, significant discrepancies arise in cross-classification performance when examining subsets. Additionally, visual inspection reveals spurious correlations in the synthetic data that are not present in the real data.Conclusion
Global realism scores can give a false sense of security: in our case study, metrics that rate the full dataset as "high quality" overlook errors which appear once we zoom in on specific subgroups. We argue that future work must (i) design subgroup-aware fidelity and utility metrics, (ii) favor conditional generators that model rare strata explicitly, and (iii) report metric panels alongside qualitative, clinical sanity checks. Until such standards mature, our analysis highlights that researchers and regulators should treat synthetic data with caution.
Related
- From Fragmentation to Standardization: A Systematic Review Update and Critical Appraisal of Genetic and Genomic Testing Assessment Frameworks and Their Alignment With the HTA Core Model.
- AI-Driven Hyperspectral Imaging in Clinical Medicine: A Critical Appraisal of Translational Readiness
- Ultrasound-guided percutaneous fine-needle aspiration in parapharyngeal space lesions: a critical appraisal with illustrative cases
- Prediction models for compassion fatigue in nurses: A protocol for systematic review and critical appraisal
- Reimagining Literature Evaluation Pedagogy Through AI-Generated Podcasts: Pharmacy Students’ Perspectives and Critical Appraisal
BibTeX
@article{chatzichristos2026quality,
title = {{Quality assessment of synthetic data in healthcare: a critical appraisal with a focus on subgroup bias.}},
author = {Christos Chatzichristos and Dimitris Katsimpokis and Congting Lai and Lore Van Santvliet and Flavio Camarrone and Daan Knoors and Gijs Geleijnse and Michel Van Speybroek and Martine Lewi and Bart Vannieuwenhuyse and Maarten De Vos},
journal = {BMC medical informatics and decision making},
year = {2026},
month = jul,
doi = {10.1186/s12911-026-03719-3},
url = {https://europepmc.org/article/MED/42907838},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Quality assessment of synthetic data in healthcare: a critical appraisal with a focus on subgroup bias.
AU - Christos Chatzichristos
AU - Dimitris Katsimpokis
AU - Congting Lai
AU - Lore Van Santvliet
AU - Flavio Camarrone
AU - Daan Knoors
AU - Gijs Geleijnse
AU - Michel Van Speybroek
AU - Martine Lewi
AU - Bart Vannieuwenhuyse
AU - Maarten De Vos
PY - 2026
DA - 2026/07/01
JO - BMC medical informatics and decision making
DO - 10.1186/s12911-026-03719-3
UR - https://europepmc.org/article/MED/42907838
AB - Background The creation of synthetic data in healthcare research has emerged as a compelling solution to address challenges related to data privacy, in case of data-sharing and data scarceness for training machine learning models.Objective The current paper aims to provide a critical landscape analysis of synthetic data quality metrics used in healthcare, mapping the state of the field, examining how current fidelity and utility scores often fail to capture weaknesses that appear in specific patient subgroups, and illustrating these blind spots with a lung-cancer case study. We specifically set out to highlight the need for assessing synthetic datasets not only at the aggregate level but also across clinically meaningful subgroups, to ensure reliability and fairness in downstream applications.Methods By examining the landscape of synthetic data adoption in healthcare, we highlight the methodological and utility considerations that shape its integration into the research ecosystem. We elucidate how synthetic data can expedite research initiatives, support data-driven decision-making, and facilitate innovative methodologies while safeguarding sensitive patient information. Conversely, while many papers focus solely on the advantages of synthetic data, we aim to highlight also the potential constraints of using synthetic data in a real-world case involving lung cancer patient level data. Our analysis centers on identifying situations where synthetic data might introduce biases or inaccuracies that hinder the generation of meaningful clinical insights.Results We demonstrate the limitations of using synthetic data in a lung cancer case study, especially when dealing with small subgroups from the original dataset. While high values for fidelity and utility metrics are achieved when considering the entire synthetic dataset, significant discrepancies arise in cross-classification performance when examining subsets. Additionally, visual inspection reveals spurious correlations in the synthetic data that are not present in the real data.Conclusion Global realism scores can give a false sense of security: in our case study, metrics that rate the full dataset as "high quality" overlook errors which appear once we zoom in on specific subgroups. We argue that future work must (i) design subgroup-aware fidelity and utility metrics, (ii) favor conditional generators that model rare strata explicitly, and (iii) report metric panels alongside qualitative, clinical sanity checks. Until such standards mature, our analysis highlights that researchers and regulators should treat synthetic data with caution.
SN - 1472-6947
ER -