Abstract: The proliferation of multimodal harmful information on social media not only undermines public interests but also severely disrupts social order, highlighting the urgent need for effective detection methods. Existing approaches have predominantly relied on pre-trained models to extract and integrate multimodal features, often neglecting the limitations of general semantics in harmful information detection and the complex, dynamic combinations of harmful content. To address these issues, a multimodal harmful content detection framework based on weakly Supervised modality semantic enhancement (weak-S) was introduced. Weakly supervised modality information was utilized to facilitate the harmful semantic alignment of multimodal features, and a low-rank bilinear pooling-based multimodal gated integration mechanism was designed to differentiate the contributions of various information sources. Experimental results show that the proposed method achieves F1-score improvements of 2.2 and 3.2 percentage points on the HarmP and MultiOFF benchmark datasets, respectively, outperforming SOTA (State-Of-The-Art) models and validating the significance of weakly supervised modality semantics in multimodal harmful information detection. Additionally, the method delivers a 1-percentage-point improvement in generalization performance for multimodal exaggeration detection tasks.

Key words: unimodal weak supervision, contrastive learning, gated integration, multimodal, harmful content detection

摘要： 社交媒体上多模态有害信息的泛滥，不仅侵害公众利益，还严重扰乱社会秩序，亟需有效的检测方法。现有研究依赖预训练模型提取与融合多模态特征，忽视了通用语义在有害信息检测任务中的局限性，且未能充分考虑有害信息复杂多变的组合形式。为此，提出一种基于弱监督模态语义增强的多模态有害信息检测方法(weak-S)，所提方法通过引入弱监督模态信息辅助多模态特征的有害语义对齐，并设计一种低秩双线性池化的多模态门控集成机制，以区分不同信息的贡献度。实验结果表明，所提方法在HarmP和MuitiOFF等公开基准数据集上的F1值相较于SOTA(State-Of-The-Art)模型分别提高了2.2和3.2个百分点，验证了弱监督模态语义在多模态有害信息检测中的重要性。此外，所提方法还在多模态夸张检测任务上取得了1个百分点的泛化性能提升。

关键词: 单模态弱监督, 对比学习, 门控集成, 多模态, 有害信息检测