关键词: 目标检测, 多模态融合, 图文融合, 融合强度可调, 语义增强

Abstract: To address the limitations of single visual modalities in object detection tasks within complex scenarios, particularly their limited expressive power and insufficient semantic information, a multimodal object detection framework with adjustable fusion intensity was proposed. A textual modality was introduced as a semantic prior, performing global semantic modeling on text data to compensate for the shortcomings of visual modalities in category recognition and semantic association at the semantic level. Considering that textual data was essentially a semantic description of the entire dataset rather than a specific description of a single image, a semantic enhancement module based on category prediction soft mapping was proposed. This module was used to adaptively filter and enhance semantic information relevant to the current image content, effectively suppressing interference from irrelevant semantic noise. To address the uneven contribution of different modalities during multimodal fusion, an adjustable fusion strength mechanism was proposed to achieve optimal information fusion based on the quality of the input data and task requirements by dynamically adjusting the fusion weights of visual and semantic features. Experimental results on the PASCAL VOC and DUT Anti-UAV datasets show that the average accuracy (AP) of the proposed method is improved by 1.1 percentage points and 3.1 percentage points, respectively, compared with the optimal baseline method. This verifies the effectiveness and generalization ability of the proposed method in multimodal feature collaborative modeling and object detection in complex scenes.

Key words: object detection, multi-modal fusion, text-image fusion, adjustable fusion intensity, semantic enhancement