Large language model-augmented implicit surgical video review

Abstract

Surgical video interpretation is a promising medical artificial intelligence application. However, no existing video annotation method preserves the spatiotemporal complexity of surgeon reasoning. Here we show that verbal reasoning and visual attention can be converted into structured, machine-actionable records of intraoperative behaviours. Our method decomposes transcribed verbal commentary into video-anchored semantic feedback chunks, which are classified via a large language model, with spatial grounding to surgical scenes via eyegaze or cursor tracking. We demonstrate method validity and scalability on structured and unstructured annotation tasks. For quality feedback on full-length colorectal procedures, the method reached near-human fidelity for chunking (cosine similarity: 0.95{+/-}0.01) and semantic classification across observations (Cohens {kappa}: 0.71{+/-}0.07) and evaluative triggers (Cohens {kappa}: 0.67{+/-}0.14), with excellent usability ratings. For structured critical view of safety assessment in laparoscopic cholecystectomy, implicit annotation yielded excellent agreement with explicit reviewer ratings (Cohens {kappa}: 0.83, 0.49 and 0.81 across three criteria). We anticipate this method will advance surgical data science by enabling scalable construction of meaningfully annotated surgical video datasets.

Calibrated Reliance on Large Language Models for Personalized Surgical Care: A Critical Narrative Review of Confidence, Clinical Reasoning and Decision Support
Mirko Barone, Federico Selvaggi, Valeria Pasquale +3
Journal of Clinical Medicine1 Oct 2026Peer-reviewedCC BY 4.0Full textEpidemiology & public healthCrossrefEurope PMC
Large Language Model-Assisted Review of Clinical Trial Informed Consent for Data Reuse: Retrieval-Augmented Generation Approach.
Lameck Mbangula Amugongo, Lena Schaller, Helene Wendt +5
JMIR medical informatics15 Oct 2026Peer-reviewedCC BYFull text1 citedEpidemiology & public healthEurope PMC
Implicit bias in surgical decision-making: a scoping review of recommendations, timing, and access to operative care.
Josue Merida, Alexsandra Hernandez, Phyllis Migdal
BMC surgery6 Aug 2026Peer-reviewedCC BY-NC-NDFull textSurgery & anaesthesiaEurope PMCPMC
AFP assistant: a retrieval-augmented generation and large language model-powered multilingual polio chatbot for low-resource language communities
Gelane Biru, Honey Gemechu, Firanol Teshome +8
npj Digital Medicine1 Jun 2026Peer-reviewedCC BYFull text1 citedEpidemiology & public healthDOAJ
Evidence-based clinical question answering using retrieval-augmented generation (RAG) and open-source large language models
Ayesha Qasim, Jesus Jaime Moreno Escobar, Christian E. Maldonado-Sifuentes +3
Scientific Reports13 Oct 2026Peer-reviewedCC BY 4.0Full textEpidemiology & public healthCrossref

Cite

BibTeX RIS

BibTeX

@misc{zhang2026large,
  title = {{Large language model-augmented implicit surgical video review}},
  author = {Z. Zhang and M. I. Qadir and R. Ramchand and M. Belwadi and R. P. Ball and K. Konstantinopoulos and E. M. Abbey and K. T. Ernsberger and M. J. Guzman and S. Hendren and B. K. Holcomb and B. W. Robb and T. Stankowski and J. A. Waters and D. Stefanidis and K. Y. Bilimoria and S. Mohanty and F. R. Kolbinger},
  howpublished = {medRxiv},
  year = {2026},
  month = aug,
  doi = {10.64898/2026.08.25.26361071},
  url = {https://www.medrxiv.org/content/10.64898/2026.08.25.26361071v1},
  note = {CC BY-NC-ND},
}

RIS

TY  - UNPB
TI  - Large language model-augmented implicit surgical video review
AU  - Z. Zhang
AU  - M. I. Qadir
AU  - R. Ramchand
AU  - M. Belwadi
AU  - R. P. Ball
AU  - K. Konstantinopoulos
AU  - E. M. Abbey
AU  - K. T. Ernsberger
AU  - M. J. Guzman
AU  - S. Hendren
AU  - B. K. Holcomb
AU  - B. W. Robb
AU  - T. Stankowski
AU  - J. A. Waters
AU  - D. Stefanidis
AU  - K. Y. Bilimoria
AU  - S. Mohanty
AU  - F. R. Kolbinger
PY  - 2026
DA  - 2026/08/25
PB  - medRxiv
DO  - 10.64898/2026.08.25.26361071
UR  - https://www.medrxiv.org/content/10.64898/2026.08.25.26361071v1
AB  - Surgical video interpretation is a promising medical artificial intelligence application. However, no existing video annotation method preserves the spatiotemporal complexity of surgeon reasoning. Here we show that verbal reasoning and visual attention can be converted into structured, machine-actionable records of intraoperative behaviours. Our method decomposes transcribed verbal commentary into video-anchored semantic feedback chunks, which are classified via a large language model, with spatial grounding to surgical scenes via eyegaze or cursor tracking. We demonstrate method validity and scalability on structured and unstructured annotation tasks. For quality feedback on full-length colorectal procedures, the method reached near-human fidelity for chunking (cosine similarity: 0.95{+/-}0.01) and semantic classification across observations (Cohens {kappa}: 0.71{+/-}0.07) and evaluative triggers (Cohens {kappa}: 0.67{+/-}0.14), with excellent usability ratings. For structured critical view of safety assessment in laparoscopic cholecystectomy, implicit annotation yielded excellent agreement with explicit reviewer ratings (Cohens {kappa}: 0.83, 0.49 and 0.81 across three criteria). We anticipate this method will advance surgical data science by enabling scalable construction of meaningfully annotated surgical video datasets.
ER  -

Abstract

Related

Cite