Design of monocular vision human arm-robotic arm interaction system based on attention feature fusion and sequential graph convolution
Abstract
Aiming at the problems of insufficient spatial context modeling and limited understanding of temporal dynamics in monocular vision-based human-arm-robot interaction, this study proposes a monocular vision-based human-arm-robot interaction system based on attention feature fusion and temporal graph convolution. The system enhances spatial feature expression through the scaled dot-product self-attention mechanism guided by multi-scale feature pyramid. Moreover, it combines the video-based monocular three-dimensional pose estimation model to drive the spatial-temporal graph convolutional network to achieve intent recognition, and builds a two-level architecture of “feature enhancement-spatial and temporal evolution”. Experimental results reveals that under the optimal parameter configuration, the system achieves a precision rate of 96.7%, a recall rate of 94.1%, and an F1-Score of 95.4% in the visual perception task. Moreover, all indicators are significantly better than the comparison method ( p < 0.001). The model inference time is shortened to 20.8 ms, and the calculation amount is 58.7 GFLOPs. The trajectory tracking error is reduced to 1.18 cm, and the average multi-task grasping success rate reaches 93.2%. Meanwhile, CPU resource utilization is 58.7 GFLOPs, and energy consumption per task is controlled at 53.8 J. The proposed method effectively solves the problem of accurate perception of continuous human arm movement intentions and smooth interaction of robotic arms under monocular vision through the collaborative mechanism of multi-scale spatial feature adaptive enhancement and dynamic spatiotemporal map modeling. This provides key technical support for natural human-computer collaboration.
Related
- YOLO-RMF: A Real-Time Surface Defect Detection Model for Hot-Rolled Strip Steel with Reparameterized Convolution and Axial Attention
- A parallel convolution attention network for crop genomic prediction
- Asymmetric encoder-decoder U-net with hybrid attention and residual dilated convolution for brain tumor segmentation
- ACMNet: An attention-convolution mixed neural network for semantic segmentation of high-resolution remote sensing images
- A convolution–attention network predicts boiler slagging and warns of waterwall overheating from the furnace temperature field
BibTeX
@article{he2026design,
title = {{Design of monocular vision human arm-robotic arm interaction system based on attention feature fusion and sequential graph convolution}},
author = {Yaqin He and Fei Zhang},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-73045-1},
url = {https://www.nature.com/articles/s41598-026-73045-1},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Design of monocular vision human arm-robotic arm interaction system based on attention feature fusion and sequential graph convolution
AU - Yaqin He
AU - Fei Zhang
PY - 2026
DA - 2026/10/03
JO - Scientific Reports
DO - 10.1038/s41598-026-73045-1
UR - https://www.nature.com/articles/s41598-026-73045-1
AB - Aiming at the problems of insufficient spatial context modeling and limited understanding of temporal dynamics in monocular vision-based human-arm-robot interaction, this study proposes a monocular vision-based human-arm-robot interaction system based on attention feature fusion and temporal graph convolution. The system enhances spatial feature expression through the scaled dot-product self-attention mechanism guided by multi-scale feature pyramid. Moreover, it combines the video-based monocular three-dimensional pose estimation model to drive the spatial-temporal graph convolutional network to achieve intent recognition, and builds a two-level architecture of “feature enhancement-spatial and temporal evolution”. Experimental results reveals that under the optimal parameter configuration, the system achieves a precision rate of 96.7%, a recall rate of 94.1%, and an F1-Score of 95.4% in the visual perception task. Moreover, all indicators are significantly better than the comparison method ( p < 0.001). The model inference time is shortened to 20.8 ms, and the calculation amount is 58.7 GFLOPs. The trajectory tracking error is reduced to 1.18 cm, and the average multi-task grasping success rate reaches 93.2%. Meanwhile, CPU resource utilization is 58.7 GFLOPs, and energy consumption per task is controlled at 53.8 J. The proposed method effectively solves the problem of accurate perception of continuous human arm movement intentions and smooth interaction of robotic arms under monocular vision through the collaborative mechanism of multi-scale spatial feature adaptive enhancement and dynamic spatiotemporal map modeling. This provides key technical support for natural human-computer collaboration.
SN - 2045-2322
ER -