A multimodal computer-vision based pipeline for eye-hand coordination quantification during the Nine-Hole Peg Test
Abstract
The Nine-Hole Peg Test (9-HPT) is a widely used clinical measure of manual dexterity in multiple sclerosis (MS), but its primary outcome, task completion time, provides limited information about the visuomotor mechanisms contributing to impaired performance. Instrumented approaches can provide richer kinematic information but often require modifications to the task or physical instrumentation that may alter natural hand-object interaction. We developed a multimodal computer-vision pipeline that enables synchronized analysis of gaze, hand kinematics, and object state during an unmodified 9-HPT. Gaze was recorded using Pupil Labs Invisible glasses, three-dimensional hand landmarks were reconstructed using WiLoR, and a custom YOLO26x-seg model detected and tracked the pegboard, holes, pegs, and collection bin. Eye-tracking, hand, and object data were temporally synchronized at the video-frame level and spatially registered to a common pegboard-centered coordinate system using frame-specific projective transformations. The pipeline further integrated hole-state transitions, hand-zone classification, and fingertip kinematics to identify PICK and PLACE events and segment hand transport and stationary phases, while fixation and saccade labels were derived from the eye-tracking system. Automated detections were visually reviewed and corrected when necessary. The resulting framework provides synchronized, event-resolved measures of gaze behavior, hand movement, object interaction, and eye-hand coordination while preserving standardized 9-HPT administration. This approach extends the 9-HPT beyond completion time and provides a reproducible platform for investigating the visuomotor mechanisms underlying dexterity impairment in MS and other neurological disorders.
Related
- A modular open-source computer vision pipeline enables individual-level behavior analysis of group-housed pigs
- Multimodal computer vision and machine learning for predictive monitoring of planar-flow casting of amorphous steel foils
- Design and implementation of an automated Drosophila locomotor assay using computer vision tracking
- Computer vision-aided locomotor behavioral analysis identifies therapeutic motor signatures in a mouse model of Huntington disease
- Impact Of Honey Bee Waggle Dance Recording Framerate On Computer Vision Process
BibTeX
@misc{gadde2026multimodal,
title = {{A multimodal computer-vision based pipeline for eye-hand coordination quantification during the Nine-Hole Peg Test}},
author = {Rajvardhan Gadde* and Mahya Beheshti* and Anjali Rajkumar and Silvana L. Costa and John-Ross Rizzo and Todd E. Hudson},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.05553},
eprint = {2610.05553},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.05553},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - A multimodal computer-vision based pipeline for eye-hand coordination quantification during the Nine-Hole Peg Test
AU - Rajvardhan Gadde*
AU - Mahya Beheshti*
AU - Anjali Rajkumar
AU - Silvana L. Costa
AU - John-Ross Rizzo
AU - Todd E. Hudson
PY - 2026
DA - 2026/10/04
PB - arXiv
DO - 10.48550/arxiv.2610.05553
UR - https://arxiv.org/abs/2610.05553
AB - The Nine-Hole Peg Test (9-HPT) is a widely used clinical measure of manual dexterity in multiple sclerosis (MS), but its primary outcome, task completion time, provides limited information about the visuomotor mechanisms contributing to impaired performance. Instrumented approaches can provide richer kinematic information but often require modifications to the task or physical instrumentation that may alter natural hand-object interaction. We developed a multimodal computer-vision pipeline that enables synchronized analysis of gaze, hand kinematics, and object state during an unmodified 9-HPT. Gaze was recorded using Pupil Labs Invisible glasses, three-dimensional hand landmarks were reconstructed using WiLoR, and a custom YOLO26x-seg model detected and tracked the pegboard, holes, pegs, and collection bin. Eye-tracking, hand, and object data were temporally synchronized at the video-frame level and spatially registered to a common pegboard-centered coordinate system using frame-specific projective transformations. The pipeline further integrated hole-state transitions, hand-zone classification, and fingertip kinematics to identify PICK and PLACE events and segment hand transport and stationary phases, while fixation and saccade labels were derived from the eye-tracking system. Automated detections were visually reviewed and corrected when necessary. The resulting framework provides synchronized, event-resolved measures of gaze behavior, hand movement, object interaction, and eye-hand coordination while preserving standardized 9-HPT administration. This approach extends the 9-HPT beyond completion time and provides a reproducible platform for investigating the visuomotor mechanisms underlying dexterity impairment in MS and other neurological disorders.
ER -