Journal of Computer Applications ›› 2026, Vol. 46 ›› Issue (7): 2277-2287.DOI: 10.11772/j.issn.1001-9081.2025060786

• Multimedia computing and computer simulation • Previous Articles    

Zero-shot human-object interaction detection method via multimodal collaborative prompt optimization

Yue MA1,2,3, Huicheng LAI1,2,3(), Di JIANG1,2,3, Liejun WANG1,2,3   

  1. 1.School of Computer Science and Technology,Xinjiang University,Urumqi Xinjiang 830046,China
    2.Xinjiang Key Laboratory of Signal Detection and Processing (Xinjiang University),Urumqi Xinjiang 830046,China
    3.Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing (Xinjiang University),Urumqi Xinjiang 830046,China
  • Received:2025-07-16 Revised:2025-09-16 Accepted:2025-09-25 Online:2025-10-13 Published:2026-07-10
  • Contact: Huicheng LAI
  • About author:MA Yue, born in 1998, M. S. candidate. His research interests include computer vision, deep learning.
    JIANG Di, born in 1997, Ph. D. candidate. His research interests include video anomaly detection, deep learning, object detection.
    WANG Liejun, born in 1975, Ph. D., professor. His research interests include video communication processing, image recognition and processing.
  • Supported by:
    Key Research and Development Program of Xinjiang Uygur Autonomous Region(2022B01008)

多模态协同提示优化下的零样本人物交互检测方法

马岳1,2,3, 赖惠成1,2,3(), 姜迪1,2,3, 汪烈军1,2,3   

  1. 1.新疆大学 计算机科学与技术学院,乌鲁木齐 830046
    2.新疆信号检测与处理重点实验室(新疆大学),乌鲁木齐 830046
    3.丝路多语言认知计算国际合作联合实验室(新疆大学),乌鲁木齐 830046
  • 通讯作者: 赖惠成
  • 作者简介:马岳(1998—),男,新疆乌鲁木齐人,硕士研究生,主要研究方向:计算机视觉、深度学习
    姜迪(1997—),男,山东济宁人,博士研究生,主要研究方向:视频异常检测、深度学习、目标检测
    汪烈军(1975—),男,四川眉山人,教授,博士,主要研究方向:视频通信处理、图像识别与处理。
  • 基金资助:
    新疆维吾尔自治区重点研发计划项目(2022B01008)

Abstract:

To address the challenge of identifying unseen categories in zero-shot Human-Object Interaction (HOI) detection, a zero-shot HOI detection method via multimodal collaborative prompt optimization, named MCPNet (Multimodal Cooperative Prompt Network), was proposed to enhance the model's generalization ability under zero-shot conditions. The method leverages text-driven prompt learning and visual feature generation mechanisms to improve the recognition performance of unseen interaction categories and the overall generalization performance from both semantic and visual modeling perspectives. First, in terms of semantic modeling, a prompt-guided module was developed by integrating human-designed prompt templates with learnable prompts, which were encoded by the Contrastive Language-Image Pre-training(CLIP)text encoder to produce fine-grained semantic representations. Then, a generation module conditioned on learnable prompts was introduced, in which a Variational AutoEncoder (VAE) was employed to model the latent semantic space and to synthesize discriminative image features for unseen categories, thus alleviating the long-tailed distribution problem. Finally, for visual modeling, a collaborative mechanism between local and global features was proposed. Local interaction region features were integrated with global image semantics as inputs to the interaction detection head, enhancing the recognition performance of HOIs in complex scenes. Experimental results in zero-shot detection tasks on the HICO-DET (Humans Interacting with Common Objects DETection) dataset show that the proposed method achieves significant gains in mean Average Precision (mAP) of unseen categories, surpassing CLIP4HOI (towards adapting CLIP for practical zero-shot HOI detection) method by 8.59% under the Unseen Combination (UC) setting, by 9.94% under the Rare-First UC (RF-UC) setting, by 8.24% under the Non-rare-First UC (NF-UC) setting, and by 9.37% under the Unseen Object (UO) setting. The conclusion verifies the effectiveness of the proposed method.

Key words: Human-Object Interaction (HOI) detection, zero-shot learning, Contrastive Language-Image Pre-training (CLIP), learnable prompt, feature generation

摘要:

针对零样本人物交互(HOI)检测任务中未见类别难以识别的问题,提出一种多模态协同提示优化下的零样本HOI检测方法MCPNet (Multimodal Cooperative Prompt Network),以提升模型在零样本条件下的泛化能力。该方法结合文本引导的提示学习与图像特征生成机制,从语义建模与视觉建模2个层面提升对未见交互类别的识别能力与整体泛化性能。首先,在语义建模方面,设计融合人工模板与可学习提示的文本引导模块,结合对比语言-图像预训练(CLIP)文本编码器生成细粒度的语义表征;其次,引入可学习提示为条件的生成模块,通过变分自编码器(VAE)建模潜在的语义空间,合成具有判别性的未见类别图像特征,从而缓解类别的长尾分布;最后,在视觉建模方面,提出局部与全局特征的协同机制,将局部交互区域特征与全局图像语义信息共同作为交互检测头的输入,提升复杂场景下的HOI检测效果。实验结果表明,在HICO-DET (Humans Interacting with Common Objects DETection)数据集上的零样本检测任务中,本文方法在多个评估设置下的未见类别的平均精度均值(mAP)均优于大多数现有的先进方法。具体而言,与CLIP4HOI(towards adapting CLIP for practical zero-shot HOI detection)方法相比,本文方法的mAP在未见组合(UC)设置下提升了8.59%,在稀有优先未见组(RF-UC)设置下提升了9.94%,在非稀有优先未见组合(NF-UC)设置下提升了8.24%,在未见物体(UO)设置下提升了9.37%,验证了该方法的有效性。

关键词: 人物交互检测, 零样本学习, 对比语言-图像预训练, 可学习提示, 特征生成

CLC Number: