Multimodal Large Language Models Rate Wetlab Suturing Repeatably but Fail to Detect Training-Related Improvement: A Pilot Study With Expert Assessment
- 1Ophthalmology, Moorfields Eye Hospital NHS Foundation Trust, London, GBR
- 2Ophthalmology, Western Eye Hospital - Imperial College Healthcare NHS Trust, London, GBR
Abstract
Background
Objective assessment of technical skill remains a bottleneck in surgical training. Expert rating is valid but consumes scarce faculty time and is susceptible to systematic rater effects, while existing automated methods rely on bespoke computer-vision models that are inaccessible to most educators. General-purpose multimodal large language models can score an image against a written rubric with no task-specific training, but their ability to judge surgical technical quality from a still photograph has not been established.
Materials and methods
Five ophthalmology residents (ST3 to ST6) attending a face-to-face oculoplastic wetlab course each closed a standardised 10 cm linear wound in ex vivo porcine tissue with a running continuous 5-0 polypropylene suture, once before and once after the course, within a fixed 10-minute limit. The 10 resulting closures were photographed under standardised conditions and cropped to the wound. A senior oculoplastic fellow, blinded to delegate identity and to pre- versus post-course status, scored every image on an eight-item anchored rubric (1 to 5 per item). The same images and the same locked rubric prompt were submitted to three multimodal large language models (Claude Opus 4.5, GPT-5.5, and Gemini 3.1 Flash-Lite), three times each at temperature 0, yielding 90 model ratings and 720 item scores from the same 10 photographs. The independent unit of analysis was the delegate (n = 5) for change and the image (n = 10) for agreement. The primary outcome was the pre- to post-course change in a seven-item composite score; secondary outcomes were agreement with the expert, repeatability across identical repeat calls, and completeness of model output.
Results
The expert recorded improvement in all five delegates (composite 2.90 ± 0.65 to 3.71 ± 0.49; change +0.82). Only Gemini 3.1 Flash-Lite reproduced a comparable effect (+0.77); Claude Opus 4.5 (+0.21) and GPT-5.5 (+0.19) did not. Claude Opus 4.5 assigned the scale midpoint of 3 to 72.3% of all item ratings and GPT-5.5 to 68.0%, compared with 39.2% for the expert; no model awarded a score of 5 anywhere in the 720 item scores, and all models rated systematically below the expert (mean item difference −0.45 to −0.73). Rank agreement with the expert for overall closure quality was moderate (Spearman rho 0.65 to 0.79), but item-level agreement was poor (exact agreement 33% to 42%; quadratic weighted kappa 0.20 to 0.24). Repeatability was high, with identical scores across all three repeat calls in 100% (Gemini 3.1 Flash-Lite), 91% (Claude Opus 4.5), and 70% (GPT-5.5) of image-item pairs. Knot security accounted for almost every rating returned as non-assessable.
Conclusions
Contemporary multimodal large language models rated wetlab suturing photographs highly repeatably but with severe central-tendency compression and systematic severity, and two of three failed to detect an improvement that both an expert and one model identified. Repeatability should not be mistaken for validity. With single-image rubric prompting, these models are not yet suitable for summative assessment of surgical technical skill, and methods that widen their effective scoring range are required before formative use can be recommended. These pilot findings, from five delegates and 10 images, are hypothesis-generating.
Related
- Image-Based Dietary Assessment Using Multimodal Large Language Models: Performance, Limitations, and Clinical Applicability in Diabetes
- Factors associated with the quality of urban geriatric healthcare services empowered by large language models and improvement pathways: a theoretical framework and cross-sectional study
- A Multimodal Quality Improvement Initiative to Reduce Non-ICU Telemetry Overutilization
- Aligning Antimicrobial Stewardship and Infection Prevention Through Repeated Assessment and Feedback: A Multisite Quality Improvement Initiative in Viet Nam
- Relative Brain Perfusion Differences and Cognitive Improvement During Multimodal Management in a Patient With Progressive Cognitive Symptoms: A Case Report.
BibTeX
@article{mushtaq2026multimodal,
title = {{Multimodal Large Language Models Rate Wetlab Suturing Repeatably but Fail to Detect Training-Related Improvement: A Pilot Study With Expert Assessment}},
author = {Jameel Mushtaq and Zaid Alsafi and Robert Brady},
journal = {Cureus},
year = {2026},
month = sep,
doi = {10.7759/cureus.117189},
url = {https://europepmc.org/article/PMC/PMC13720391},
note = {CC BY},
}
RIS
TY - JOUR
TI - Multimodal Large Language Models Rate Wetlab Suturing Repeatably but Fail to Detect Training-Related Improvement: A Pilot Study With Expert Assessment
AU - Jameel Mushtaq
AU - Zaid Alsafi
AU - Robert Brady
PY - 2026
DA - 2026/09/30
JO - Cureus
DO - 10.7759/cureus.117189
UR - https://europepmc.org/article/PMC/PMC13720391
AB - Background Objective assessment of technical skill remains a bottleneck in surgical training. Expert rating is valid but consumes scarce faculty time and is susceptible to systematic rater effects, while existing automated methods rely on bespoke computer-vision models that are inaccessible to most educators. General-purpose multimodal large language models can score an image against a written rubric with no task-specific training, but their ability to judge surgical technical quality from a still photograph has not been established. Materials and methods Five ophthalmology residents (ST3 to ST6) attending a face-to-face oculoplastic wetlab course each closed a standardised 10 cm linear wound in ex vivo porcine tissue with a running continuous 5-0 polypropylene suture, once before and once after the course, within a fixed 10-minute limit. The 10 resulting closures were photographed under standardised conditions and cropped to the wound. A senior oculoplastic fellow, blinded to delegate identity and to pre- versus post-course status, scored every image on an eight-item anchored rubric (1 to 5 per item). The same images and the same locked rubric prompt were submitted to three multimodal large language models (Claude Opus 4.5, GPT-5.5, and Gemini 3.1 Flash-Lite), three times each at temperature 0, yielding 90 model ratings and 720 item scores from the same 10 photographs. The independent unit of analysis was the delegate (n = 5) for change and the image (n = 10) for agreement. The primary outcome was the pre- to post-course change in a seven-item composite score; secondary outcomes were agreement with the expert, repeatability across identical repeat calls, and completeness of model output. Results The expert recorded improvement in all five delegates (composite 2.90 ± 0.65 to 3.71 ± 0.49; change +0.82). Only Gemini 3.1 Flash-Lite reproduced a comparable effect (+0.77); Claude Opus 4.5 (+0.21) and GPT-5.5 (+0.19) did not. Claude Opus 4.5 assigned the scale midpoint of 3 to 72.3% of all item ratings and GPT-5.5 to 68.0%, compared with 39.2% for the expert; no model awarded a score of 5 anywhere in the 720 item scores, and all models rated systematically below the expert (mean item difference −0.45 to −0.73). Rank agreement with the expert for overall closure quality was moderate (Spearman rho 0.65 to 0.79), but item-level agreement was poor (exact agreement 33% to 42%; quadratic weighted kappa 0.20 to 0.24). Repeatability was high, with identical scores across all three repeat calls in 100% (Gemini 3.1 Flash-Lite), 91% (Claude Opus 4.5), and 70% (GPT-5.5) of image-item pairs. Knot security accounted for almost every rating returned as non-assessable. Conclusions Contemporary multimodal large language models rated wetlab suturing photographs highly repeatably but with severe central-tendency compression and systematic severity, and two of three failed to detect an improvement that both an expert and one model identified. Repeatability should not be mistaken for validity. With single-image rubric prompting, these models are not yet suitable for summative assessment of surgical technical skill, and methods that widen their effective scoring range are required before formative use can be recommended. These pilot findings, from five delegates and 10 images, are hypothesis-generating.
SN - 2168-8184
ER -