Journals
  Publication Years
  Keywords
Search within results Open Search
Please wait a minute...
For Selected: Toggle Thumbnails
Multimodal sentiment analysis model for missing modalities under shared semantic conditions
Shang LIU, Zhaosen TANG, Hongyue LIU, Linfang DONG, Jin ZHOU
Journal of Computer Applications    2026, 46 (9): 2761-2768.   DOI: 10.11772/j.issn.1001-9081.2025080959
Abstract2)   HTML0)    PDF (671KB)(12)       Save

Existing studies on multimodal sentiment analysis in modality missing scenarios often neglect inter-modal correlations when generating missing modalities, leading to semantic inconsistencies between restored and original data. Additionally, missing modality generation based on diffusion models brings large computational overhead. To address these issues, a Multimodal Sentiment Analysis Model for Missing Modalities under Shared Semantic Conditions (MM-SSC) was proposed. First, a shared latent space mapping module was designed to use the shared latent space of Vector Quantized Variational AutoEncoder (VQ-VAE) to capture multimodal distributions effectively, thereby ensuring cross-modal shared semantics. Second, a cross-modal consistency constraint method was proposed to learn mutual information within each modality's latent space, thereby promoting the refinement of semantic information between modalities and enhancing cross-modal consistency. Third, a missing modality reconstruction and alignment module was designed to reconstruct and refine missing modalities while reducing reconstruction computational overhead. Finally, a multimodal fusion and prediction module was introduced to fuse reconstructed and available modalities for consistent sentiment analysis. Experimental results demonstrate that under fixed modality missing conditions, compared with Incomplete Multimodality-Diffused emotion recognition (IMDer) model, the proposed model achieves average improvements of 0.4 and 0.7 percentage points in the F1 score and ACC7 on the CMU-MOSI dataset, respectively; on the CMU-MOSEI dataset, the F1 score and ACC7 are improved by 0.9 and 0.3 percentage points, respectively. Under random modality missing conditions, compared with IMDer model, the proposed model achieves an average improvement of 2.2 percentage points in both F1 score and ACC7 on the CMU-MOSI dataset; on the CMU-MOSEI dataset, the F1 score and ACC7 (accuracy for seven classes) are improved by 0.8 and 0.4 percentage points, respectively. It can be seen that MM-SSC can address multimodal sentiment analysis tasks under modality missing scenarios effectively.

Table and Figures | Reference | Related Articles | Metrics