Abstract: Social media multi-modal rumor detection faces challenges such as weak cross-modal feature correlation and insufficient intrinsic representation of data. Therefore, a rumor detection method based on cross-modal attention mechanism and contrastive learning was proposed. Fine-grained features of text and vision were extracted by a multi-modal feature module in this method. Cross-modal co-attention mechanism and difference learning were utilized to enhance inter-modal relevance. Complex semantic contexts were captured by multi-head self-attention. A contrastive learning module was innovatively introduced to achieve feature optimization under machine supervision. Experimental results on public Twitter and Weibo datasets showed that the accuracy of the proposed method was improved by 5.47 and 4.44 percentage points respectively compared with the existing optimal model MMFN (Multi-modal fake news detection on social media via multi-grained information fusion), verifying the key roles of fine-grained feature mining and cross-modal similarity modeling in detection performance. It can be seen that deeply analyzing multi-modal content differences and strengthening cross-modal association mechanisms can effectively improve the recognition accuracy of social media rumors.

Key words: cross-modal, self-attention mechanism, contrastive learning, multi-modal, rumor detection method

摘要： 社交媒体多模态谣言检测面临跨模态特征关联性弱、数据内在表征不足的挑战。因此，提出基于跨模态注意力机制与对比学习的谣言检测方法。该方法通过多模态特征模块提取文本与视觉的细粒度特征，利用跨模态共同注意力机制和差异性学习来增强模态间关联性，运用多头自注意力捕获复杂语义上下文，并创新性地引入对比学习模块实现机器监督下的特征优化。在Twitter和Weibo公开数据集上的实验结果表明，所提方法准确率较现有最优模型MMFN（Multi-modal fake news detection on social media via multi-grained information fusion）分别提升5.47和4.44个百分点，验证了细颗粒度特征挖掘与跨模态相似性建模对检测性能的关键作用。可见，深度解析多模态内容差异、强化跨模态关联机制能有效提升社交媒体谣言的识别精度。

关键词: 跨模态, 自注意力机制, 对比学习, 多模态, 谣言检测方法