关键词: 跨模态, 自注意力机制, 对比学习, 多模态, 谣言检测方法

Abstract: Social media multi-modal rumor detection faces challenges such as weak cross-modal feature correlation and insufficient intrinsic representation of data. Therefore, a rumor detection method based on cross-modal attention mechanism and contrastive learning was proposed. Fine-grained features of text and vision were extracted by a multi-modal feature module in this method. Cross-modal co-attention mechanism and difference learning were utilized to enhance inter-modal relevance. Complex semantic contexts were captured by multi-head self-attention. A contrastive learning module was innovatively introduced to achieve feature optimization under machine supervision. Experimental results on public Twitter and Weibo datasets showed that the accuracy of the proposed method was improved by 5.47 and 4.44 percentage points respectively compared with the existing optimal model MMFN (Multi-modal fake news detection on social media via multi-grained information fusion), verifying the key roles of fine-grained feature mining and cross-modal similarity modeling in detection performance. It can be seen that deeply analyzing multi-modal content differences and strengthening cross-modal association mechanisms can effectively improve the recognition accuracy of social media rumors.

Key words: cross-modal, self-attention mechanism, contrastive learning, multi-modal, rumor detection method