An LLM-Based Auditing Framework for Targeted Quality Assurance in Cancer Registries
- 1University of Missouri
Abstract
Cancer registries are essential for population-level surveillance, epidemiologic research, and public health planning, but their accuracy depends on human review of key variables documented in unstructured clinical text. This is time-consuming, and current quality assurance (QA) typically relies on reviewing fewer than 10% of records at random, which can miss facility- or time-specific errors.
We developed a large language model (LLM)-based framework for targeted cancer registry QA. The framework uses LLMs to extract biomarker values from clinical text, compares them with registry entries, and flags disagreements for expert review. Six open-source LLMs were benchmarked locally in a privacy-preserving environment, and the best model per site and report type was applied to breast and prostate cancer, two of the most common cancers in the United States. For the four breast cancer biomarkers ER, PR, HER2, and Ki-67, the disagreement rate against quality-assured registry values in 1,852 breast cancer cases was lowest for ER at 1.67% and highest for Ki-67 at 9.18%. In 920 prostate cancer cases, the disagreement rate with the deployed models stayed below 8% for every categorical Gleason variable, while the exact-value PSA metric reached 17.28%. PSA therefore generated the largest volume of candidate discrepancies, though individual PSA flags require more corroboration than categorical flags. Applied to a full year of records (11,740 breast and 6,792 prostate cases), the framework flagged disagreements and assigned each a probable cause, grounded in NAACCR coding rules, to prioritize expert review. Extracted values are linked to their supporting text, and model, prompt, and pipeline versions are logged for reproducibility.
By directing review toward records most likely to contain discrepancies, the framework offers an efficient, auditable alternative to random QA sampling for cancer registries.
Related
- Data Auditing and Quality Assurance in a Federated Learning Consortium; Getting the Best of Both Worlds from Cross-Institutional and In-House Data Quality Inspection
- A framework for assessing quality indicators in animal cancer registries: a case study
- Prediction of patient-specific quality assurance for volumetric modulated arc therapy using machine log files
- Data-efficient prostate cancer classification from T2-weighted MRI with mechanistic auditing and a dual-agent quality-control workflow
- Quality Assurance in Pharmacy Education in the East African Community: National Accreditation, Regional Harmonization, and Continuous Improvement
BibTeX
@misc{seifaddini2026llmbased,
title = {{An LLM-Based Auditing Framework for Targeted Quality Assurance in Cancer Registries}},
author = {M. Seifaddini and M. Beheshti and J. Steffens and D. Carnagey and M. Esebua and M. Wakefield and I. Zachary},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.22.26363595},
url = {https://www.medrxiv.org/content/10.64898/2026.09.22.26363595v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - An LLM-Based Auditing Framework for Targeted Quality Assurance in Cancer Registries
AU - M. Seifaddini
AU - M. Beheshti
AU - J. Steffens
AU - D. Carnagey
AU - M. Esebua
AU - M. Wakefield
AU - I. Zachary
PY - 2026
DA - 2026/09/22
PB - medRxiv
DO - 10.64898/2026.09.22.26363595
UR - https://www.medrxiv.org/content/10.64898/2026.09.22.26363595v1
AB - Cancer registries are essential for population-level surveillance, epidemiologic research, and public health planning, but their accuracy depends on human review of key variables documented in unstructured clinical text. This is time-consuming, and current quality assurance (QA) typically relies on reviewing fewer than 10% of records at random, which can miss facility- or time-specific errors. We developed a large language model (LLM)-based framework for targeted cancer registry QA. The framework uses LLMs to extract biomarker values from clinical text, compares them with registry entries, and flags disagreements for expert review. Six open-source LLMs were benchmarked locally in a privacy-preserving environment, and the best model per site and report type was applied to breast and prostate cancer, two of the most common cancers in the United States. For the four breast cancer biomarkers ER, PR, HER2, and Ki-67, the disagreement rate against quality-assured registry values in 1,852 breast cancer cases was lowest for ER at 1.67% and highest for Ki-67 at 9.18%. In 920 prostate cancer cases, the disagreement rate with the deployed models stayed below 8% for every categorical Gleason variable, while the exact-value PSA metric reached 17.28%. PSA therefore generated the largest volume of candidate discrepancies, though individual PSA flags require more corroboration than categorical flags. Applied to a full year of records (11,740 breast and 6,792 prostate cases), the framework flagged disagreements and assigned each a probable cause, grounded in NAACCR coding rules, to prioritize expert review. Extracted values are linked to their supporting text, and model, prompt, and pipeline versions are logged for reproducibility. By directing review toward records most likely to contain discrepancies, the framework offers an efficient, auditable alternative to random QA sampling for cancer registries.
ER -