关键词: 关键词: 多模态, 情感分析, 自监督, 注意力机制, 视觉语言预训练模型

Abstract: Aiming at the problems of incomplete intra-modal information, poor inter-modal interaction and difficult training in multimodal sentiment analysis tasks, a visual language pre-training model (VLP) is applied to the field of multimodal sentiment analysis, and a multimodal sentiment analysis network (Multimodal EmotionNet Fused Self- Supervised Learning and Multi-Layer Cross, MESM). The visual coder module is strengthened by self-supervised learning and multi-layer cross attention is added to better model textual and visual features, which makes the intra-modal information richer and more complete and the inter-modal information interaction more adequate, and solves the problem of high complexity of attention computation in the Transformer by the fast, memory-efficient Flash Attention with IO-awareness. high complexity problem in Transformer. Compared with the current mainstream TomBERT, CLIP, VILT, and ViLBERT, MESM has the best results in accuracy and recall on the processed MVSA dataset, reaching 71.3% and 69.2%, respectively, which proves that the method can simultaneously and efficiently improve the completeness of the multimodal information fusion while reducing the cost of computation.

Key words: Keywords: multimodal, sentiment analysis, self-supervision, attentional mechanism, visual language pretraining model