Non-generative readout with few-label calibration versus LoRA fine-tuning of a large language model for ASA physical status classification: accuracy, speed, cost and label efficiency in three data sources
- 1People's Hospital of Xinjiang Uygur Autonomous Region
Abstract
BackgroundThe American Society of Anesthesiologists Physical Status (ASA-PS) class underpins perioperative risk communication, but its assignment is subjective. Generative large language models (LLMs) are slow and costly, and LoRA fine-tuning needs site-specific labels and a separately maintained adapter for every site. Non-generative decision models return option probabilities in one forward pass, but their comparison with LoRA in anesthesia, and the number of local labels they need, is unstudied.
ObjectiveTo compare, on one open base model, generative zero-shot prompting, a non-generative readout, the readout with cutpoint calibration on a few local labels, and task LoRA for four-class ASA-PS (I, II, III, IV-V) in accuracy, speed and label requirement.
MethodsRetrospective study, not preregistered. MOVER (57,473 operations) was the development source, the local hospitals 2025-2026 operations (n = 2,753) the temporally external validation set, and VitalDB (n = 6,242) a second site. The base model was Qwen3.5-9B; input was preoperative structured variables (T1) or T1 plus diagnosis text (T2). The primary metric was quadratic-weighted {kappa} against each sites recorded ASA class, with patient-level cluster-bootstrap intervals; the non-inferiority margin was 0.05. Cost means tokens and GPU time.
ResultsIn the local validation set the readout reached {kappa} = 0.460 versus 0.227 for generative zero-shot prompting with a requested rationale (difference +0.234, 95
ConclusionsThe non-generative readout needs no training and ran about 64 times faster than generative prompting with a rationale. A few hundred local labels bring it close to LoRA in {kappa} without training and storing a per-site adapter, although {kappa} of 0.52-0.58 is not evidence of human-level performance. LoRA keeps a higher {kappa} once several hundred or more labels are available, and higher exact agreement. Non-inferiority holds for the prespecified analyses and sits at the margin in stricter sensitivity analyses, so it should be read as conditional.
Related
- Knowledge distillation-based lightweight generative large language models for standardized quality control of Chinese radiology reports
- Calibration, Uncertainty, and Data Efficiency of Deep Learning Chest Radiograph Classifiers Under Cross-Dataset Distribution Shift: A Multi-Dataset Benchmark
- Generative Artificial Intelligence in Periodontology: Application of OpenAI Models for the Classification of Periodontitis and Risk Assessment in Patients Undergoing Supportive Periodontal Care
- Accuracy and confidence calibration of large language models on neurology board-style questions
- Classification without Calibration: The Internal Logical Incoherence of Nova and Its Conflict with Dose-Dependent Biological Reasoning
BibTeX
@misc{shi2026nongenerative,
title = {{Non-generative readout with few-label calibration versus LoRA fine-tuning of a large language model for ASA physical status classification: accuracy, speed, cost and label efficiency in three data sources}},
author = {Y. Shi and Y. Wang and A. Simayi and L. Qu and G. Xu},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.20.26366028},
url = {https://www.medrxiv.org/content/10.64898/2026.10.20.26366028v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Non-generative readout with few-label calibration versus LoRA fine-tuning of a large language model for ASA physical status classification: accuracy, speed, cost and label efficiency in three data sources
AU - Y. Shi
AU - Y. Wang
AU - A. Simayi
AU - L. Qu
AU - G. Xu
PY - 2026
DA - 2026/10/20
PB - medRxiv
DO - 10.64898/2026.10.20.26366028
UR - https://www.medrxiv.org/content/10.64898/2026.10.20.26366028v1
AB - BackgroundThe American Society of Anesthesiologists Physical Status (ASA-PS) class underpins perioperative risk communication, but its assignment is subjective. Generative large language models (LLMs) are slow and costly, and LoRA fine-tuning needs site-specific labels and a separately maintained adapter for every site. Non-generative decision models return option probabilities in one forward pass, but their comparison with LoRA in anesthesia, and the number of local labels they need, is unstudied. ObjectiveTo compare, on one open base model, generative zero-shot prompting, a non-generative readout, the readout with cutpoint calibration on a few local labels, and task LoRA for four-class ASA-PS (I, II, III, IV-V) in accuracy, speed and label requirement. MethodsRetrospective study, not preregistered. MOVER (57,473 operations) was the development source, the local hospitals 2025-2026 operations (n = 2,753) the temporally external validation set, and VitalDB (n = 6,242) a second site. The base model was Qwen3.5-9B; input was preoperative structured variables (T1) or T1 plus diagnosis text (T2). The primary metric was quadratic-weighted {kappa} against each sites recorded ASA class, with patient-level cluster-bootstrap intervals; the non-inferiority margin was 0.05. Cost means tokens and GPU time. ResultsIn the local validation set the readout reached {kappa} = 0.460 versus 0.227 for generative zero-shot prompting with a requested rationale (difference +0.234, 95 ConclusionsThe non-generative readout needs no training and ran about 64 times faster than generative prompting with a rationale. A few hundred local labels bring it close to LoRA in {kappa} without training and storing a per-site adapter, although {kappa} of 0.52-0.58 is not evidence of human-level performance. LoRA keeps a higher {kappa} once several hundred or more labels are available, and higher exact agreement. Non-inferiority holds for the prespecified analyses and sits at the margin in stricter sensitivity analyses, so it should be read as conditional.
ER -