A Granular Interaction Framework for Surgical Robotic Digital Twins: Enabling Realistic Segment-Level Manipulation in Extended Reality
- 1Department of Thoracic Surgery, University Hospital Bonn, Bonn, Nordrhein-Westphalia Germany
- 2Bonn Surgical Technology Center (BOSTER), University Hospital Bonn, Bonn, Nordrhein-Westphalia Germany
- 3Department of Thoracic Surgery, Helios Hospital Bonn/Rhein-Sieg, Bonn, Nordrhein-Westphalia Germany
Abstract
Introduction: While Robot-Assisted Surgery (RAS) offers significant clinical advantages, adoption is often hindered by steep learning curves. Extended Reality (XR) and Digital Twins (DT) provide promising solutions for surgical training. However, existing integration pipelines require high-fidelity system-level simulations that address complex kinematic constraints. This paper proposes a modernized framework for integrating RAS into Extended Reality (XR) by transitioning from CAD models to scan-derived digital twins (DTs), as well as a force-based interaction paradigm for realistic simulation.
Methods: Performance profiling was conducted on the Apple Vision Pro (AVP) to establish empirical hardware benchmarks for polygon counts. Utilizing 3D scanning and manual retopology, we developed a digital surrogate of the Dexter Surgical Robotic System, prioritizing model fidelity within stand-alone XR constraints. Within Unity, we evaluated Rigidbody versus ArticulationBody physics for kinematic stability. This necessitated the development of a force-based interaction system supporting both linear feedback (PD) and geometric constraint-based controllers.
Results: The scan-derived model achieved a spatial accuracy of 1.6mm relative to the physical system. ArticulationBody physics demonstrated superior numerical stability compared to traditional Rigidbody components. While the linear feedback controller showed higher quantitative precision, it required extensive manual gain tuning; conversely, the geometric constraint-based controller provided high-speed stability without iterative calibration.
Conclusion: This framework modernizes RAS integration by providing a non-CAD-dependent workflow, a prescriptive testing protocol for mesh decimation, and a force-based interaction system that enhances kinematic realism for clinical training.
Related
- Surgical management of a rare bronchial granular cell tumor: a case report
- Extratemporal Facial Nerve: Surgical Landmarks, Risk Zones, and a Practical Framework for Safe Dissection
- Attention Is All You Need: A Universal Augmented Reality Surgical Navigation Framework
- Surgical Timing Considerations in osteosarcoma patients with autism spectrum disorder: A systematic review of indirect evidence and a conceptual framework
- Psychopathology-AI-interaction (PAII) framework: Disorder-specific dynamics in conversational AI
BibTeX
@article{vo2026granular,
title = {{A Granular Interaction Framework for Surgical Robotic Digital Twins: Enabling Realistic Segment-Level Manipulation in Extended Reality}},
author = {An-Nhien Vo and Dhanakumaresh Subramani and Joachim Schmidt and Jan Arensmeyer and Philipp Feodorovici},
journal = {International Journal of Computer Assisted Radiology and Surgery},
year = {2026},
month = may,
doi = {10.1007/s11548-026-03677-1},
url = {https://doi.org/10.1007/s11548-026-03677-1},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - A Granular Interaction Framework for Surgical Robotic Digital Twins: Enabling Realistic Segment-Level Manipulation in Extended Reality
AU - An-Nhien Vo
AU - Dhanakumaresh Subramani
AU - Joachim Schmidt
AU - Jan Arensmeyer
AU - Philipp Feodorovici
PY - 2026
DA - 2026/05/19
JO - International Journal of Computer Assisted Radiology and Surgery
DO - 10.1007/s11548-026-03677-1
UR - https://doi.org/10.1007/s11548-026-03677-1
AB - Introduction: While Robot-Assisted Surgery (RAS) offers significant clinical advantages, adoption is often hindered by steep learning curves. Extended Reality (XR) and Digital Twins (DT) provide promising solutions for surgical training. However, existing integration pipelines require high-fidelity system-level simulations that address complex kinematic constraints. This paper proposes a modernized framework for integrating RAS into Extended Reality (XR) by transitioning from CAD models to scan-derived digital twins (DTs), as well as a force-based interaction paradigm for realistic simulation. Methods: Performance profiling was conducted on the Apple Vision Pro (AVP) to establish empirical hardware benchmarks for polygon counts. Utilizing 3D scanning and manual retopology, we developed a digital surrogate of the Dexter Surgical Robotic System, prioritizing model fidelity within stand-alone XR constraints. Within Unity, we evaluated Rigidbody versus ArticulationBody physics for kinematic stability. This necessitated the development of a force-based interaction system supporting both linear feedback (PD) and geometric constraint-based controllers. Results: The scan-derived model achieved a spatial accuracy of 1.6mm relative to the physical system. ArticulationBody physics demonstrated superior numerical stability compared to traditional Rigidbody components. While the linear feedback controller showed higher quantitative precision, it required extensive manual gain tuning; conversely, the geometric constraint-based controller provided high-speed stability without iterative calibration. Conclusion: This framework modernizes RAS integration by providing a non-CAD-dependent workflow, a prescriptive testing protocol for mesh decimation, and a force-based interaction system that enhances kinematic realism for clinical training.
SN - 1861-6410
SN - 1861-6429
ER -