关键词: 跨模态检索, 张量融合网络, 图卷积网络, 语义相关性, 语义关系图

Abstract: Abstract: The key of cross-modal image-text retrieval is how to capture the semantic correlation between images and texts effectively. Most of the existing methods learn the global semantic correlation between image region features and text features or local semantic correlations between inter-modality objects, ignoring the intra-modality relationship and inter-modality correlation between the relationships of image regions and sentence words. To solve this problem, This method proposes a cross-modal tensor fusion network based on semantic relation graph (CMTFN-SRG) for image-text retrieval. The intra-modality relationships of image regions and sentence words are generated by Graph Convolutional Networks (GCN) and Bidirectional Gated Recurrent Unit (Bi-GRU) separately, which are used to explore the fine-grained semantic correlation between the intra-modality relationships of image regions and sentence words by the tensor fusion network. Moreover, this method uses GRU to learn the global features of the image, and match the global features of the image and the text to capture the inter-modality global semantic correlation. The proposed method is compared with the current relevant algorithms on two bench-mark datasets, Flickr30K and MS-COCO. The experimental results show that the proposed method can effectively improve the retrieval accuracy in the image-text retrieval.

Key words: Cross-modal retrieval, Tensor fusion network, Graph Convolutional Networks, Semantic correlation, Semantic graph