Abstract:

Comments on social media platforms sometimes express their attitudes towards events through sarcasm. Sarcasm detection can more accurately analyze user sentiments and opinions. But traditional models based on vocabulary and syntactic structure ignore the role of text sentiment information in sarcasm detection and suffer from performance degradation due to data noise. To address these limitations， a Multimodal Sarcasm Detection model integrating Contrastive learning with Sentiment analysis （MSDCS） was proposed. Firstly， BERT （Bidirectional Encoder Representation from Transformers） was used to extract text features， and ViT （Vision Transformer） was used to extract image features. Then， the contrastive loss in contrastive learning was employed to train a shallow model， and the image and text features were aligned before fusion. Finally， the cross-modal features were combined with the sentiment features to make classification judgments， and the use of information between different modalities was maximized to achieve sarcasm detection. Experimental results on the open dataset of multimodal sarcasm detection show that the accuracy and F1 value of MSDCS are at least 1.85% and 1.99% higher than those of the baseline model Decomposition and Relation Network （D&R Net）， verifying the effectiveness of using sentiment information and contrastive learning in multimodal sarcasm detection.

Key words: social media, sarcasm detection, sentiment analysis, contrastive learning, momentum distillation

摘要：

社交媒体平台上的评论有时会通过反讽来表达对事件的态度，通过反讽检测，可以更准确地分析用户情绪和观点。针对基于词汇和句法结构的传统模型忽略了文本情感信息对反讽检测的作用和由于数据噪声造成的检测性能降低等问题，提出一个融合对比学习和情感分析的多模态反讽检测模型（MSDCS）。首先，利用BERT（Bidirectional Encoder Representation from Transformers）提取文本特征，并利用ViT（Vision Transformer）提取图像特征；其次，利用对比学习中的对比损失训练浅层模型，在融合之前对齐图像和文本特征；最后，结合跨模态特征与情感特征融合后的结果作分类判断，最大限度地利用不同模态间信息实现反讽检测。在多模态反讽检测开放数据集上的实验结果表明，相较于基于分解和关系网络（D&R Net）的基准模型，MSDCS的准确率和F1值至少提高了1.85%和1.99%，验证了在多模态反讽检测中利用情感信息和对比学习的有效性。

关键词: 社交媒体, 反讽检测, 情感分析, 对比学习, 动量蒸馏