Foundation model-driven framework for human–object interaction prediction with segmentation mask integration

Abstract

In this work, we introduce Segmentation to Human–Object Interaction (Seg2HOI), a novel framework that integrates segmentation-based vision foundation models with the human–object interaction (HOI) task, distinguished from traditional detection-based HOI methods. Our approach enhances HOI detection by not only predicting the standard triplets but also introducing quadruplets, which extend HOI triplets by including segmentation masks for human–object pairs. More specifically, Seg2HOI inherits the properties of the vision foundation model (e.g., promptable and interactive mechanisms) and incorporates an implicit relation-learning decoder that applies these attributes to the HOI task. By freezing the vision foundation model, Seg2HOI preserves its original capabilities while using pixel-level mask guidance to support high-level relational reasoning. Despite being trained only for the HOI task, without additional training mechanisms for these properties, the framework demonstrates that such features still operate efficiently. Extensive experiments on two public benchmark datasets demonstrate that Seg2HOI achieves performance comparable to state-of-the-art methods, even in zero-shot scenarios. Lastly, we propose that Seg2HOI can generate HOI quadruplets and interactive HOI segmentation from novel text and visual prompts that were not used during training, making it versatile for a wide range of applications by leveraging this flexibility.

PRInter: an inductive multimodal heterogeneous network embedding framework with contrastive learning for lncRNA-protein interaction prediction
Guoqing Zhao, Baolu Shi, Pengpai Li +1
BMC Bioinformatics30 Jul 2026Peer-reviewedCC BY-NC-ND 4.0Full textOncologyPMC
MDM-DDI: a drug–drug interaction prediction framework integrating multi-dimensional molecular features and memory-augmented networks
Yiqun Wang, Shilong Song, Yufei Li
Frontiers in Pharmacology9 Nov 2026Peer-reviewedCC BY 4.0Pharmacology & toxicologyFrontiersCrossref
HAFI-DTA: an interpretable multimodal framework for drug-target affinity prediction based on hierarchical attention guidance and feature iterative interaction
Yiqun Wang, Shilong Song, Wei Tian +1
Scientific Reports23 Oct 2026Peer-reviewedCC BY-NC-ND 4.0Full textGeneral medicineCrossrefDOAJ
Automated thigh muscle segmentation using foundation models: validation and impact of segmentation quality
Katharina Ziegeler, Gabrielle Hoyer, Felix Liu +7
European radiology experimental13 Oct 2026Peer-reviewedCC BYFull textMedical imaging & radiologyEurope PMC
From Prediction to Interaction: A Behavioral Informatics Framework for AI in Preventive Health.
Anand Gavai, Jaap Heringa
Journal of medical Internet research1 Oct 2026Peer-reviewedCC BYFull textEpidemiology & public healthEurope PMCPMCDOAJ

Cite

BibTeX RIS

BibTeX

@article{park2026foundation,
  title = {{Foundation model-driven framework for human–object interaction prediction with segmentation mask integration}},
  author = {Juhan Park and Kyungjae Lee and Hyung Jin Chang and Jungchan Cho},
  journal = {Scientific Reports},
  year = {2026},
  month = oct,
  doi = {10.1038/s41598-026-67660-1},
  url = {https://www.nature.com/articles/s41598-026-67660-1},
  note = {CC BY 4.0},
}

RIS

TY  - JOUR
TI  - Foundation model-driven framework for human–object interaction prediction with segmentation mask integration
AU  - Juhan Park
AU  - Kyungjae Lee
AU  - Hyung Jin Chang
AU  - Jungchan Cho
PY  - 2026
DA  - 2026/10/23
JO  - Scientific Reports
DO  - 10.1038/s41598-026-67660-1
UR  - https://www.nature.com/articles/s41598-026-67660-1
AB  - In this work, we introduce Segmentation to Human–Object Interaction (Seg2HOI), a novel framework that integrates segmentation-based vision foundation models with the human–object interaction (HOI) task, distinguished from traditional detection-based HOI methods. Our approach enhances HOI detection by not only predicting the standard triplets but also introducing quadruplets, which extend HOI triplets by including segmentation masks for human–object pairs. More specifically, Seg2HOI inherits the properties of the vision foundation model (e.g., promptable and interactive mechanisms) and incorporates an implicit relation-learning decoder that applies these attributes to the HOI task. By freezing the vision foundation model, Seg2HOI preserves its original capabilities while using pixel-level mask guidance to support high-level relational reasoning. Despite being trained only for the HOI task, without additional training mechanisms for these properties, the framework demonstrates that such features still operate efficiently. Extensive experiments on two public benchmark datasets demonstrate that Seg2HOI achieves performance comparable to state-of-the-art methods, even in zero-shot scenarios. Lastly, we propose that Seg2HOI can generate HOI quadruplets and interactive HOI segmentation from novel text and visual prompts that were not used during training, making it versatile for a wide range of applications by leveraging this flexibility.
SN  - 2045-2322
ER  -

Abstract

Related

Cite