Feasibility of Privacy-preserving Locally Hosted Language Models for Assessing Medical Student Reflective Writing.
- 1Cleveland Clinic Lerner College of Medicine at Case Western Reserve University, Cleveland, OH USA.
- 2Department of Medical Education, Tufts University School of Medicine, 136 Harrison Ave., Boston, MA USA.
- 3Department of Medicine, Indiana University School of Medicine, Indianapolis, IN USA.
Abstract
Reflective writing assessment in medical education is labor-intensive and raises data privacy concerns that preclude cloud-based AI solutions. We conducted a feasibility study using a locally-hosted, open-weight large language model (LLM) deployed on institutional servers to assess medical students' reflective writing. The model evaluated 10 de-identified student narratives using two published rubrics across three independent passes, generating rubric-aligned scores and formative feedback. Scores were consistent across passes and rubrics with no statistically significant differences. Expert faculty rated LLM-generated feedback as meeting most quality criteria (mean 5.48/7). These findings support feasibility of a privacy-preserving, locally-hosted LLM for scalable reflective writing assessment.Supplementary information
The online version contains supplementary material available at https://doi.org/10.1007/s40670-026-02865-7.
Related
- Privacy-Preserving Large Language Model Deployment for Oncology Registry Abstraction: Structure-Aware Evaluation in a Real-World Clinical Setting
- Federated learning for privacy-preserving opportunistic screening of lumbar-spine DXA-defined osteoporotic-level bone mineral density using chest radiographs: A multi-center independent validation study
- Privacy-preserving clustered federated learning via differential privacy and homomorphically encrypted prototypes
- Montage-agnostic federated learning for privacy-preserving Alzheimer’s disease classification from heterogeneous multi-site EEG
- Privacy-Aware Distillation of Large Language Models for Enhanced Multimorbidity Scoring
BibTeX
@article{rejeleene2026feasibility,
title = {{Feasibility of Privacy-preserving Locally Hosted Language Models for Assessing Medical Student Reflective Writing.}},
author = {Rick Rejeleene and Jessica N Byram and Richard M Frankel and J Harry Isaacson and Neil B Mehta},
journal = {Medical science educator},
year = {2026},
month = sep,
doi = {10.1007/s40670-026-02865-7},
url = {https://europepmc.org/article/MED/42968798},
note = {CC BY},
}
RIS
TY - JOUR
TI - Feasibility of Privacy-preserving Locally Hosted Language Models for Assessing Medical Student Reflective Writing.
AU - Rick Rejeleene
AU - Jessica N Byram
AU - Richard M Frankel
AU - J Harry Isaacson
AU - Neil B Mehta
PY - 2026
DA - 2026/09/01
JO - Medical science educator
DO - 10.1007/s40670-026-02865-7
UR - https://europepmc.org/article/MED/42968798
AB - Reflective writing assessment in medical education is labor-intensive and raises data privacy concerns that preclude cloud-based AI solutions. We conducted a feasibility study using a locally-hosted, open-weight large language model (LLM) deployed on institutional servers to assess medical students' reflective writing. The model evaluated 10 de-identified student narratives using two published rubrics across three independent passes, generating rubric-aligned scores and formative feedback. Scores were consistent across passes and rubrics with no statistically significant differences. Expert faculty rated LLM-generated feedback as meeting most quality criteria (mean 5.48/7). These findings support feasibility of a privacy-preserving, locally-hosted LLM for scalable reflective writing assessment.Supplementary information The online version contains supplementary material available at https://doi.org/10.1007/s40670-026-02865-7.
SN - 2156-8650
ER -