Journals
  Publication Years
  Keywords
Search within results Open Search
Please wait a minute...
For Selected: Toggle Thumbnails
Multimodal sentiment analysis model with cross-modal text information enhancement
Yihan WANG, Chong LU, Zhongyuan CHEN
Journal of Computer Applications    2025, 45 (7): 2237-2244.   DOI: 10.11772/j.issn.1001-9081.2024060886
Abstract58)   HTML2)    PDF (1163KB)(21)       Save

Multimodal Sentiment Analysis (MSA) that utilize text, visual, and audio data to analyze speakers’ emotions in videos have garnered widespread attention. However, the contributions of different modalities to sentiment analysis vary significantly. Generally, the information contained in text is more intuitive, making it particularly important to seek a strategy for enhancing text in sentiment analysis. To address this issue, a Multimodal Sentiment Analysis Model with Cross-modal Text-information Enhancement (MSAM-CTE) was proposed. Firstly, the BERT (Bidirectional Encoder Representations from Transformers) pre-trained model was employed to extract text features, and the Bi-directional Long Short-Term Memory (Bi-LSTM) network was used to further process the pre-processed audio and video features. Then, a text based cross-attention mechanism was applied to integrate text information into emotion related nonverbal representations, thereby learning text oriented pairwise cross-modal mappings to obtain effective unified multimodal representations. Finally, the fused features were utilized for sentiment analysis. Experimental results show that compared to the optimal baseline model — Text Enhanced Transformer Fusion Network (TETFN), the proposed model achieved a 2.6% reduction in Mean Absolute Error (MAE) and a 0.1% increase in Pearson Correlation coefficient (Corr) on the CMU-MOSI (Carnegie Mellon University Multimodal Opinion Sentiment Intensity) dataset;on the CMU-MOSEI (Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity) dataset, the improvements are 3.8% for MAE and 1.7% for Corr, respectively, verifying the effectiveness of MSAM-CTE in sentiment analysis.

Table and Figures | Reference | Related Articles | Metrics