Quality, readability, and information fidelity of large language model–generated versus physician-authored informed-consent content for targeted therapies: a blinded, paired comparative study
Abstract
ObjectiveTo compare the content quality of basic information, risks, benefits, and alternative options in informed consent forms for targeted therapy generated by large language models (LLMs) versus those drafted by physicians, and to evaluate informational fidelity and reading level.MethodsThis study used a single-center, retrospective, paired-case, blinded multi-evaluator design. We enrolled 70 adult patients who received targeted therapy in the Department of Rheumatology at Beijing Tongren Hospital during 2025. LLM texts were generated using DeepSeek-V4-pro based on identical de-identified case data and standardized prompts, and corresponding physician texts were extracted from electronic records. Three rheumatologists performed blinded scoring using a scale comprising 8 domains and 13 items. Reading level and informational errors were assessed using CRIE 3.0 and source tracing.ResultsA total of 140 texts were analyzed. The LLM arm achieved a higher median overall quality score than the physician arm (60 vs. 50; adjusted mean difference 9.95, 95% CI 9.36 to 10.54; p < 0.01), with higher scores for risks, benefits, and alternatives (all adjusted p < 0.01). The LLM arm also exhibited higher reading level scores (7.87 vs. 7.10; p < 0.01) and a higher rate of hallucinations (27.1% vs. 0; p < 0.01). No statistically significant between-arm differences were found for inaccuracies or omissions.ConclusionIn this exploratory study, according to a locally developed assessment framework, LLM-generated texts received higher composite evaluator ratings and more consistently covered predefined informed consent content domains. However, they more frequently contained unsupported patient-specific inferences and imposed a greater reading burden. LLMs may be useful for generating structured drafts and checking coverage of predefined content, but they should not be used to produce final informed consent documents without physician review and item-by-item verification. These findings require further validation using patient-centered outcomes.
Related
- AI-generated high-intensity interval training information: a cross-sectional comparative evaluation of quality and readability
- Quality and readability of responses generated by popular large language models to frequently asked questions about menopause: A comparative study
- Readability, quality, and reliability of AI-generated ınformation on myofascial pain syndrome: A comparative analysis of ChatGPT, Gemini, and Perplexity
- Evaluating the reliability, quality, and readability of AI-generated patient education on hallux valgus: a comparative study of large language models
- Quality, Readability, and Content Deficiencies of Three Large Language Models in Bee Sting Health Information: A Comparative Evaluation with Patient Perspectives
BibTeX
@article{xie2026quality,
title = {{Quality, readability, and information fidelity of large language model–generated versus physician-authored informed-consent content for targeted therapies: a blinded, paired comparative study}},
author = {Yue Xie and Yuan Gao and Jiajing Peng and Yiling Tao and Shanshan Xu and Zhongwen Cui and Boya Zhou and Ente Wang and Li Cui},
journal = {Frontiers in Pharmacology},
year = {2026},
month = nov,
doi = {10.3389/fphar.2026.1958826},
url = {https://www.frontiersin.org/articles/10.3389/fphar.2026.1958826},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Quality, readability, and information fidelity of large language model–generated versus physician-authored informed-consent content for targeted therapies: a blinded, paired comparative study
AU - Yue Xie
AU - Yuan Gao
AU - Jiajing Peng
AU - Yiling Tao
AU - Shanshan Xu
AU - Zhongwen Cui
AU - Boya Zhou
AU - Ente Wang
AU - Li Cui
PY - 2026
DA - 2026/11/04
JO - Frontiers in Pharmacology
DO - 10.3389/fphar.2026.1958826
UR - https://www.frontiersin.org/articles/10.3389/fphar.2026.1958826
AB - ObjectiveTo compare the content quality of basic information, risks, benefits, and alternative options in informed consent forms for targeted therapy generated by large language models (LLMs) versus those drafted by physicians, and to evaluate informational fidelity and reading level.MethodsThis study used a single-center, retrospective, paired-case, blinded multi-evaluator design. We enrolled 70 adult patients who received targeted therapy in the Department of Rheumatology at Beijing Tongren Hospital during 2025. LLM texts were generated using DeepSeek-V4-pro based on identical de-identified case data and standardized prompts, and corresponding physician texts were extracted from electronic records. Three rheumatologists performed blinded scoring using a scale comprising 8 domains and 13 items. Reading level and informational errors were assessed using CRIE 3.0 and source tracing.ResultsA total of 140 texts were analyzed. The LLM arm achieved a higher median overall quality score than the physician arm (60 vs. 50; adjusted mean difference 9.95, 95% CI 9.36 to 10.54; p < 0.01), with higher scores for risks, benefits, and alternatives (all adjusted p < 0.01). The LLM arm also exhibited higher reading level scores (7.87 vs. 7.10; p < 0.01) and a higher rate of hallucinations (27.1% vs. 0; p < 0.01). No statistically significant between-arm differences were found for inaccuracies or omissions.ConclusionIn this exploratory study, according to a locally developed assessment framework, LLM-generated texts received higher composite evaluator ratings and more consistently covered predefined informed consent content domains. However, they more frequently contained unsupported patient-specific inferences and imposed a greater reading burden. LLMs may be useful for generating structured drafts and checking coverage of predefined content, but they should not be used to produce final informed consent documents without physician review and item-by-item verification. These findings require further validation using patient-centered outcomes.
SN - 1663-9812
ER -