Privacy-Preserving Large Language Model Deployment for Oncology Registry Abstraction: Structure-Aware Evaluation in a Real-World Clinical Setting
- 1Bond University
Abstract
ObjectivesStructuring oncology clinical notes into registry-grade variables is essential for research and care but remains labour-intensive and error-prone. This study developed and evaluated a privacy-preserving large language model pipeline for oncology registry abstraction in a real-world clinical setting.
MethodsAn open-source Meta Llama 3.3 70B-based pipeline, prompted without task-specific fine-tuning, was deployed to extract over 50 variables from 6,700 oncology notes at a cancer centre in Singapore. Data were de-identified locally using a Hide-In-Plain-Sight approach, ensuring no identifiable data left hospital infrastructure. Performance was assessed on 200 randomly sampled notes with adjudicated ground truth, using a structure-aware framework that classified outputs as correct, missing, spurious, or incorrect.
ResultsF1 scores were high across variables, including diagnosis (97.2%), histology (95.8%), stage (92.6%), biomarkers (91.4%), and treatments (88.1%). Preliminary transferability testing on 50 external notes showed strong performance for core diagnostic variables, with reduced performance for list-valued variables.
DiscussionThe structure-aware evaluation approach separated omissions from fabrications, providing more clinically actionable insight than aggregate metrics alone.
ConclusionPrivacy-preserving large language models achieved high extraction accuracy under a clinician-verified reference standard, supporting their feasibility for oncology registry abstraction in real-world clinical settings.
What is already known on this topicStructuring oncology clinical notes into registry-grade variables is essential for cancer research and care but remains labour-intensive, costly, and inconsistent from one reviewer to the next when performed manually. Large language models offer potential to automate this task, but their use in real-world registries has been limited by concerns over patient data privacy, reliance on task-specific fine-tuning, and evaluation methods that obscure the difference between missing and fabricated information.
What this study addsThis study shows that an open-source, privacy-preserving large language model pipeline, deployed entirely within hospital infrastructure without task-specific fine-tuning, can extract over 50 registry variables from oncology notes with high accuracy (F1 scores 88-97% across key variables), validated against ground truth checked by two clinicians. Using a structure-aware evaluation framework, we demonstrate that errors can be meaningfully separated into missing, spurious, and incorrect categories, offering clearer insight into model behaviour than aggregate accuracy metrics alone, and show preliminary evidence of transferability to notes from an external setting.
How this study might affect research, practice or policyThese findings support the feasibility of deploying privacy-preserving, locally hosted large language models for oncology registry abstraction without requiring identifiable data to leave hospital systems or extensive model customisation. The structure-aware evaluation framework is also shared as a reusable template, enabling other institutions to validate similar AI extraction tools in a consistent, clinically meaningful way before wider implementation.
Related
- A Reproducible Decomposed Large Language Model Pipeline for Structured Abstraction of Multimodal Neuroimaging Reports: Development and Independent Validation in 1,669 Emergency Department Encounters for Dizziness and Vertigo
- Context presentation and native conversation structure in large language model responses to escalating suicide risk: a controlled multimodel evaluation
- Toward Standards for Responsible AI in Health Communication: A Dual Method Pre-Deployment Risk Evaluation Framework and Case Study
- Feasibility of Privacy-preserving Locally Hosted Language Models for Assessing Medical Student Reflective Writing.
- Multi-Version Evaluation of Deep Learning Architectures for Tibial Plateau Fracture Detection and Deployment in a Web-Based Clinical Support System.
BibTeX
@misc{moldovan2026privacypreserving,
title = {{Privacy-Preserving Large Language Model Deployment for Oncology Registry Abstraction: Structure-Aware Evaluation in a Real-World Clinical Setting}},
author = {M. Moldovan and R. Enikeev and M. Chu and A. Amalraj and P. P. Koli and S. S. Abdul and H. Sivaraj and U. Iqbal and C. K. Toh},
howpublished = {medRxiv},
year = {2026},
month = may,
doi = {10.64898/2026.05.18.26353541},
url = {https://www.medrxiv.org/content/10.64898/2026.05.18.26353541v2},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Privacy-Preserving Large Language Model Deployment for Oncology Registry Abstraction: Structure-Aware Evaluation in a Real-World Clinical Setting
AU - M. Moldovan
AU - R. Enikeev
AU - M. Chu
AU - A. Amalraj
AU - P. P. Koli
AU - S. S. Abdul
AU - H. Sivaraj
AU - U. Iqbal
AU - C. K. Toh
PY - 2026
DA - 2026/05/18
PB - medRxiv
DO - 10.64898/2026.05.18.26353541
UR - https://www.medrxiv.org/content/10.64898/2026.05.18.26353541v2
AB - ObjectivesStructuring oncology clinical notes into registry-grade variables is essential for research and care but remains labour-intensive and error-prone. This study developed and evaluated a privacy-preserving large language model pipeline for oncology registry abstraction in a real-world clinical setting. MethodsAn open-source Meta Llama 3.3 70B-based pipeline, prompted without task-specific fine-tuning, was deployed to extract over 50 variables from 6,700 oncology notes at a cancer centre in Singapore. Data were de-identified locally using a Hide-In-Plain-Sight approach, ensuring no identifiable data left hospital infrastructure. Performance was assessed on 200 randomly sampled notes with adjudicated ground truth, using a structure-aware framework that classified outputs as correct, missing, spurious, or incorrect. ResultsF1 scores were high across variables, including diagnosis (97.2%), histology (95.8%), stage (92.6%), biomarkers (91.4%), and treatments (88.1%). Preliminary transferability testing on 50 external notes showed strong performance for core diagnostic variables, with reduced performance for list-valued variables. DiscussionThe structure-aware evaluation approach separated omissions from fabrications, providing more clinically actionable insight than aggregate metrics alone. ConclusionPrivacy-preserving large language models achieved high extraction accuracy under a clinician-verified reference standard, supporting their feasibility for oncology registry abstraction in real-world clinical settings. What is already known on this topicStructuring oncology clinical notes into registry-grade variables is essential for cancer research and care but remains labour-intensive, costly, and inconsistent from one reviewer to the next when performed manually. Large language models offer potential to automate this task, but their use in real-world registries has been limited by concerns over patient data privacy, reliance on task-specific fine-tuning, and evaluation methods that obscure the difference between missing and fabricated information. What this study addsThis study shows that an open-source, privacy-preserving large language model pipeline, deployed entirely within hospital infrastructure without task-specific fine-tuning, can extract over 50 registry variables from oncology notes with high accuracy (F1 scores 88-97% across key variables), validated against ground truth checked by two clinicians. Using a structure-aware evaluation framework, we demonstrate that errors can be meaningfully separated into missing, spurious, and incorrect categories, offering clearer insight into model behaviour than aggregate accuracy metrics alone, and show preliminary evidence of transferability to notes from an external setting. How this study might affect research, practice or policyThese findings support the feasibility of deploying privacy-preserving, locally hosted large language models for oncology registry abstraction without requiring identifiable data to leave hospital systems or extensive model customisation. The structure-aware evaluation framework is also shared as a reusable template, enabling other institutions to validate similar AI extraction tools in a consistent, clinically meaningful way before wider implementation.
ER -