SET-HOI: Skeleton-Enhanced Transformer with Parallel Fusion for HOI Detection.
- 1School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China.
Abstract
Human-Object Interaction (HOI) detection seeks to identify the relationships between humans and objects in images that play a pivotal role in high-level vision tasks and biomimetic systems, such as biomimetic robotics, intelligent prosthetics, and embodied AI for intent understanding. However, existing approaches that rely solely on visual appearance features are susceptible to background interference, which adversely affects detection accuracy. Inspired by the biomimetic dual-stream mechanism of biological visual perception, we propose a Transformer-based HOI detection model with a dual-branch parallel fusion architecture, incorporating a skeleton topology branch alongside the standard image branch. The skeleton branch leverages a Graph Convolutional Network (GCN) to explicitly model spatial-kinematic relationships between human keypoints and objects, providing fine-grained topological priors to complement visual appearance features before joint decoding. Our approach achieves 51.6% Mean Average Precision (mAP) on the Verbs in Common Objects in Context (V-COCO) dataset, outperforming the Transformer-based HOI baseline by 2.7 mAP points, highlighting its effectiveness in enhancing perceptual robustness for biomimetic interactive applications.
Related
- Enhanced multimodal transformer framework for sarcasm detection with particle swarm optimization based feature fusion
- MedFuse: dual-stream fusion of convolutional and vision transformer-based features for enhanced medical image classification
- Parallel Local–Global Feature Extraction and Cross-Attention Gated Fusion for Network Intrusion Detection
- WaterMamba: A Hybrid Mamba-Enhanced Detection Transformer for Small River Obstacle Detection in Complex Hydrological Environments
- Edge context fusion transformer for accurate and efficient photovoltaic electroluminescence defect detection
BibTeX
@article{xiong2026sethoi,
title = {{SET-HOI: Skeleton-Enhanced Transformer with Parallel Fusion for HOI Detection.}},
author = {Rui Xiong and Bohong Wu and Qing Gao},
journal = {Biomimetics (Basel, Switzerland)},
year = {2026},
month = sep,
doi = {10.3390/biomimetics11100687},
url = {https://europepmc.org/article/MED/42930239},
note = {CC BY},
}
RIS
TY - JOUR
TI - SET-HOI: Skeleton-Enhanced Transformer with Parallel Fusion for HOI Detection.
AU - Rui Xiong
AU - Bohong Wu
AU - Qing Gao
PY - 2026
DA - 2026/09/23
JO - Biomimetics (Basel, Switzerland)
DO - 10.3390/biomimetics11100687
UR - https://europepmc.org/article/MED/42930239
AB - Human-Object Interaction (HOI) detection seeks to identify the relationships between humans and objects in images that play a pivotal role in high-level vision tasks and biomimetic systems, such as biomimetic robotics, intelligent prosthetics, and embodied AI for intent understanding. However, existing approaches that rely solely on visual appearance features are susceptible to background interference, which adversely affects detection accuracy. Inspired by the biomimetic dual-stream mechanism of biological visual perception, we propose a Transformer-based HOI detection model with a dual-branch parallel fusion architecture, incorporating a skeleton topology branch alongside the standard image branch. The skeleton branch leverages a Graph Convolutional Network (GCN) to explicitly model spatial-kinematic relationships between human keypoints and objects, providing fine-grained topological priors to complement visual appearance features before joint decoding. Our approach achieves 51.6% Mean Average Precision (mAP) on the Verbs in Common Objects in Context (V-COCO) dataset, outperforming the Transformer-based HOI baseline by 2.7 mAP points, highlighting its effectiveness in enhancing perceptual robustness for biomimetic interactive applications.
SN - 2313-7673
ER -