关键词: 跨模态检索, 视频文本检索, 多特征融合, 弱语义数据, 自适应

Abstract: Video-text Retrieval, as a basic task in visual language learning, aims to retrieve Video data with the same semantic information based on a given Text description, or to retrieve text description with the same semantic information based on a given video. How to accurately mine the potential semantic correspondence between video and text is the key difficulty to solve this task. However, the existing video text retrieval methods usually assume that there is a strong semantic association between text description and video, but ignore the widely existing video text pairs with weak semantic description in the data set. As a result, the model is good at recognizing common general concepts, but cannot fully tap the potential information of weak semantic description, and ignores the hidden detailed information interaction between different modes. Thus, the model retrieval performance is affected. To solve these problems, a video text retrieval model based on cross-modal information fusion is proposed in this paper. Specifically, the model utilizes relevant external knowledge in a cross-modal manner to improve model retrieval performance. First, two external knowledge retrieval modules are constructed, which are used to retrieve video and external knowledge and text and external knowledge respectively, so as to strengthen the feature representation of original video and text with the help of external knowledge. Secondly, an adaptive cross-modal information fusion module is designed to remove redundant information in video and text, and use the complementary information between different modes for feature fusion, so as to learn more discriminative feature representation. Finally, inter-modal and intra-modal similarity loss functions are introduced to ensure the integrity of information representation in the fusion feature space, video feature space and text feature space, and to achieve accurate retrieval between cross-modal data. Experimental results on three public datasets, MSR-VTT, DiDeMo and LSMDC, show that the proposed method is superior to existing video text retrieval methods.

Key words: Cross-modal retrieval, Video-text retrieval, Multi-feature fusion, Weak semantic data, adaptive