关键词: 情绪识别, 多模态, 模态缺失, 预训练, 深度学习

Abstract: Aiming at the problem of model compatibility caused by modality absence in real complex scenes, an emotion recognition method compatible with missing modal reasoning was proposed. Firstly, during the pre-training and fine-tuning stages, a modality-random-dropout training strategy to ensure model compatibility during inference was employed. Secondly, a spatio-temporal masking strategy and a feature fusion mechanism based on cross-modal attention mechanism were proposed separately, to reduce the risk of model over-fitting and enhance cross-modal feature fusion efficiency. Finally, to tackle the problem of inconsistent emotion labels across various modalities, an adaptive denoising strategy based on multi-prototype clustering was proposed. The strategy separately sets class centers for different modalities and removes noisy labels by comparing the consistency between the clustering categories of each modality’ features and their labels. Compared with the baseline, experimental results on a self-built dataset show that the proposed model improves the WAR index by 6.98 percentage points while full modality reasoning, 4.09 percentage points while video modality is absent, and 33.05 percentage points while audio modality is absent. Compared with existing methods on published video dataset DFEW, the proposed model also achieves the highest WAR, reaching 68.94%.

Key words: emotion recognition, multi-modality, modality absence, pre-training, deep learning