Abstract: Comments on social media platforms sometimes express their attitudes towards events through sarcasm. Sarcasm detection can more accurately analyze user emotions and opinions. Traditional models based on vocabulary and syntactic structure ignore the role of text sentiment information in sarcasm detection and the problem of reduced detection performance due to data noise. A multimodal sarcasm detection model that integrates contrastive learning and sentiment analysis is proposed. First, BERT (Bidirectional Encoder Representation from Transformers) is used to extract text features, and ViT (Vision Transformer) is used to extract image features. Then, the contrastive loss in contrastive learning is used to train a shallow model, and the image and text features are aligned before fusion. Finally, the cross-modal features are combined with the sentiment features to make classification judgments, and the information between different modalities is maximized to achieve sarcasm detection. Experimental results on the open dataset of multimodal sarcasm detection show that the accuracy and F1 value of the proposed model are at least 1.85% and 1.99% higher than the baseline model based on decomposition and relation network (D&R Net), which verifies the effectiveness of using sentiment information and contrastive learning in multimodal sarcasm detection.

Key words: social media, sarcasm detection, sentiment analysis, contrastive learning, momentum distillation

摘要： 社交媒体平台上的评论有时会通过反讽来表达对事件的态度，通过反讽检测，可以更准确地分析用户情绪和观点。基于词汇和句法结构的传统模型忽略了文本情感信息对反讽检测的作用和由于数据噪声而造成检测性能降低的问题，提出一个融合对比学习和情感分析的多模态反讽检测模型。首先运用BERT(Bidirectional Encoder Representation from Transformers)提取文本特征，运用ViT(Vision Transformer)提取图像特征；再利用对比学习中的对比损失训练浅层模型，在融合之前对齐图像和文本特征；最后结合跨模态特征与情感特征融合后的结果做分类判断，最大限度地利用不同模态间信息实现反讽检测。在多模态反讽检测开放数据集上的实验结果表明，相较于基于分解和关系网络的基准模型(D&R Net)，所提模型的准确率和F1值至少提高了1.85%和1.99%，验证了在多模态反讽检测中利用情感信息和对比学习的有效性。

关键词: 社交媒体, 反讽检测, 情感分析, 对比学习, 动量蒸馏