关键词: 多模态协同, 文本语义提示, 视觉语义提示, 目标检测, 开放集识别

Abstract: To address the limited generalization of single-modal prompts and poor scenario adaptability of multimodal fixed-weight fusion in open-set object detection, a cross-domain object detection algorithm with multimodal collaborative semantic prompts (MMCS-Prompt) was proposed. First, a collaborative framework of image, text-prompt, visual-prompt encoders and a decoder was built, with hierarchical vision transformer using shifted windows(Swin Transformer) and 6-layer encoder extracting multi-scale visual features. Then, a text-visual contrastive alignment module was designed for bi-modal semantic correlation and knowledge complementation via mutual information maximization. Finally, a multi-factor dynamic weight fusion mechanism was proposed to adaptively allocate weights via semantic consistency, modal complementarity and scene uncertainty. Zero-shot experiments on COCO, LVIS, ODinW and Roboflow100 with 10 random seeds show that compared with Grounding DINO (Marrying DINO with Grounded Pre-Training), the Average Precision (AP) on common and cross-scene object detection improves by 0.9 and 1.1 percentage points (1.86% and 4.20% relative gains); compared with the hybrid and visual modes of T-Rex2 (Towards Generic Object Detection via Text-Visual Prompt Synergy), the AP on long-tailed and complex-scene improves by 2.7 and 0.5 percentage points (7.28% and 2.86% relative gains). It provides a new text-visual semantic collaboration and dynamic weight fusion method for open-set object detection, and an effective path for cross-domain applications like industrial part inspection, dense small object analysis and intelligent traffic.

Key words: multimodal collaboration, text semantic prompt, vision semantic prompt, object detection, open-set recognition