Multi-model LLM assessment of Quality Control Circle methodological quality: a designed-anchor reliability study
- 1An-Nan Hospital-China Medical University, Tainan, Taiwan
Abstract
BackgroundQuality Control Circle (QCC) reports are often reviewed qualitatively, but reviewer workload and inter-rater variability make large-scale assessment difficult. We evaluated whether multiple large language models (LLMs) could score QCC methodological quality reliably on a designed-anchor benchmark.
ObjectiveTo estimate inter-model reliability for QCC quality scoring and to assess alignment with designed anchors and descriptive comparability to a small set of public PMC QCC reports.
MethodsThirty synthetic QCC reports and eight public PMC QCC reports were evaluated across four primary LLM evaluators (GPT, Gemini, Grok, DeepSeek) and one sensitivity evaluator (Claude); Claude was excluded from the primary panel because a same-family model generated the synthetic cases. Evaluators received the report text only; the trap manifest and designed-anchor scores were withheld, and their absence was verified programmatically. Each case was scored on eight QCC quality dimensions in three runs per evaluator, summarized by median, and ICC(A,1) was estimated across the primary panel. Anchor calibration, trap detection from the evaluators structured defect output under a map fixed before scoring, sensitivity analyses, and a descriptive synthetic-versus-PMC check were also examined.
ResultsInter-model reliability on the primary k=4 panel was good: ICC(A,1) = 0.843 (95% CI 0.812 to 0.870; case-level bootstrap 0.782 to 0.874) on 239 pooled case-dimension rows from all 30 cases; per-dimension estimates ranged from 0.747 to 0.901. The pre-specified k=5 analysis including Claude gave 0.833, leave-one-out estimates ranged from 0.835 to 0.857, and the originally specified score parser gave 0.854. On the 52 trap-affected case-dimension cells the primary-panel consensus fell within {+/-}1 of the designed anchor in 50 (96.2%), with mean absolute error 0.54 and exact agreement 34.6%; 51.5% of individual primary-panel run scores equalled the anchor. A defect of the expected category was emitted in 87% of evaluator runs and by 94% of (trap, evaluator) pairs, versus 55% and 68% for a pre-specified keyword detector; because some categories are emitted at high base rates regardless of case, these are upper bounds. All eight dimensions met the descriptive synthetic-versus-PMC margin; synthetic cases tended to score higher, significantly so only for Action Alignment.
ConclusionsMulti-model LLM scoring of QCC methodological quality achieved good inter-model reliability and directionally consistent alignment with synthetic anchor scores, with planted weaknesses identified far more often in structured defect output than by keyword matching. Benchmark feasibility is established; expert-labeled validation in clinical QCC reports is the planned next step.
Related
- Evaluation of the Reliability and Quality of Nursing Care Plans Produced by Artificial Intelligence Models for Patients Experiencing Anaphylactic Shock: A Methodological and Exploratory Investigation.
- Clustering of patient-derived laboratory data: A privacy-preserving distributed approach to enhance clinical laboratory quality control and regional external quality assessment
- Different Analytical Modes on Automatic Hematology Analyzers for Testing Quality Control Materials and Commutability Assessment
- Methodological quality and clinical applicability of long COVID guidelines and scientific statements: a systematic review using AGREE II and AGREE-REX
- Spin and Methodological Quality in Systematic Reviews of Decompression Timing for Central Cord Syndrome
BibTeX
@misc{lin2026multimodel,
title = {{Multi-model LLM assessment of Quality Control Circle methodological quality: a designed-anchor reliability study}},
author = {H. LIn and J. Lyu},
howpublished = {medRxiv},
year = {2026},
month = aug,
doi = {10.64898/2026.08.12.26360276},
url = {https://www.medrxiv.org/content/10.64898/2026.08.12.26360276v2},
note = {CC BY},
}
RIS
TY - UNPB
TI - Multi-model LLM assessment of Quality Control Circle methodological quality: a designed-anchor reliability study
AU - H. LIn
AU - J. Lyu
PY - 2026
DA - 2026/08/12
PB - medRxiv
DO - 10.64898/2026.08.12.26360276
UR - https://www.medrxiv.org/content/10.64898/2026.08.12.26360276v2
AB - BackgroundQuality Control Circle (QCC) reports are often reviewed qualitatively, but reviewer workload and inter-rater variability make large-scale assessment difficult. We evaluated whether multiple large language models (LLMs) could score QCC methodological quality reliably on a designed-anchor benchmark. ObjectiveTo estimate inter-model reliability for QCC quality scoring and to assess alignment with designed anchors and descriptive comparability to a small set of public PMC QCC reports. MethodsThirty synthetic QCC reports and eight public PMC QCC reports were evaluated across four primary LLM evaluators (GPT, Gemini, Grok, DeepSeek) and one sensitivity evaluator (Claude); Claude was excluded from the primary panel because a same-family model generated the synthetic cases. Evaluators received the report text only; the trap manifest and designed-anchor scores were withheld, and their absence was verified programmatically. Each case was scored on eight QCC quality dimensions in three runs per evaluator, summarized by median, and ICC(A,1) was estimated across the primary panel. Anchor calibration, trap detection from the evaluators structured defect output under a map fixed before scoring, sensitivity analyses, and a descriptive synthetic-versus-PMC check were also examined. ResultsInter-model reliability on the primary k=4 panel was good: ICC(A,1) = 0.843 (95% CI 0.812 to 0.870; case-level bootstrap 0.782 to 0.874) on 239 pooled case-dimension rows from all 30 cases; per-dimension estimates ranged from 0.747 to 0.901. The pre-specified k=5 analysis including Claude gave 0.833, leave-one-out estimates ranged from 0.835 to 0.857, and the originally specified score parser gave 0.854. On the 52 trap-affected case-dimension cells the primary-panel consensus fell within {+/-}1 of the designed anchor in 50 (96.2%), with mean absolute error 0.54 and exact agreement 34.6%; 51.5% of individual primary-panel run scores equalled the anchor. A defect of the expected category was emitted in 87% of evaluator runs and by 94% of (trap, evaluator) pairs, versus 55% and 68% for a pre-specified keyword detector; because some categories are emitted at high base rates regardless of case, these are upper bounds. All eight dimensions met the descriptive synthetic-versus-PMC margin; synthetic cases tended to score higher, significantly so only for Action Alignment. ConclusionsMulti-model LLM scoring of QCC methodological quality achieved good inter-model reliability and directionally consistent alignment with synthetic anchor scores, with planted weaknesses identified far more often in structured defect output than by keyword matching. Benchmark feasibility is established; expert-labeled validation in clinical QCC reports is the planned next step.
ER -