关键词: 多模态事实验证, 语义关联, 注意力机制, CLIP模型, 特征融合

Abstract: Multimodal fact verification aims to classify the veracity of claims by leveraging multimodal evidence, addressing the limitations of traditional unimodal methods in handling complex claims. However, existing approaches face challenges in bridging semantic gaps between multimodal evidence and between claims and evidence during feature fusion. To address this, the proposed method introduces a cross-modal semantic association approach, achieving cross-level semantic alignment and adaptive feature interaction through a cross-modal attention mechanism, effectively mitigating semantic discrepancies among multi-source information and improving classification performance for complex claims. In the evidence retrieval phase, textual evidence is retrieved based on the claim and used to filter semantically related image evidence, ensuring high relevance of multimodal evidence. During verification, the method employs the Contrastive Language–Image Pretraining (CLIP) model for semantic alignment between text and multimodal evidence and designs a claim-evidence joint attention module to enhance semantic associations among claim text, textual evidence, and image evidence. Experimental results on the MOCHEG and CEAD datasets show significant improvements in accuracy, recall, and F1 score, demonstrating the method's effectiveness in multimodal fact verification.

Key words: multimodal fact verification, semantic association, attention mechanism, Contrastive Language–Image Pretraining (CLIP), feature fusion