Multimodal harmful content detection method based on weakly supervised modality semantic enhancement

doi:10.11772/j.issn.1001-9081.2024101453

Journal of Computer Applications ›› 2025, Vol. 45 ›› Issue (10): 3146-3153.DOI: 10.11772/j.issn.1001-9081.2024101453

• Artificial intelligence • Previous Articles

Multimodal harmful content detection method based on weakly supervised modality semantic enhancement

Jinwen LIU¹^,²^,³, Lei WANG¹^,²^,³(), Bo MA¹^,²^,³, Rui DONG¹^,²^,³, Yating YANG¹^,²^,³, Ahtamjan Ahmat¹^,²^,³, Xinyue WANG⁴

^1.Xinjiang Technical Institute of Physics and Chemistry，Chinese Academy of Sciences，Urumqi Xinjiang 830011，China
^2.University of Chinese Academy of Sciences，Beijing 100049，China
^3.Xinjiang Laboratory of Minority Speech and Language Information Processing，Urumqi Xinjiang 830011，China
^4.Hohai University，Nanjing Jiangsu 210000，China

Received:2024-10-14 Revised:2024-12-05 Accepted:2024-12-09 Online:2024-12-23 Published:2025-10-10
Contact: Lei WANG
About author:LIU Jinwen， born in 1999， M. S. candidate. Her research interests include harmful information detection.
WANG Lei， born in 1974， Ph. D.， research fellow. His research interests include multilingual intelligent information processing.
MA Bo， born in 1984， Ph. D.， research fellow. His research interests include multilingual intelligent information processing.
DONG Rui， born in 1985， Ph. D.， associate research fellow. His research interests include natural language processing， metaphor detection.
YANG Yating， born in 1985， Ph. D.， research fellow. Her research interests include multilingual intelligent information processing.
Ahtamjan Ahmat， born in 1997， Ph. D. candidate. His research interests include multilingual and multimodal semantic modeling.
WANG Xinyue， born in 2003. Her research interests include intelligent science and technology.
Supported by:
Key Project of Xinjiang Uygur Autonomous Region Natural Science Foundation(2023D01D17);“Tianshan Talents” Scientific and Technological Innovation Leading Talent Project(2022TSYCLJ0046);Xinjiang Uygur Autonomous Region Key Research and Development Program(2023B03024);Sciences Youth Innovation Promotion Association of Chinese Academy of Sciences(Y2021112);Xinjiang Uygur Autonomous Region “Tianshan Talents” Training Program(2023TSYCCX0041)

基于弱监督模态语义增强的多模态有害信息检测方法

刘晋文¹^,²^,³, 王磊¹^,²^,³(), 马博¹^,²^,³, 董瑞¹^,²^,³, 杨雅婷¹^,²^,³, 艾合塔木江·艾合麦提¹^,²^,³, 王欣乐⁴

^1.中国科学院新疆理化技术研究所，乌鲁木齐 830011
^2.中国科学院大学，北京 100049
^3.新疆民族语音语言信息处理实验室，乌鲁木齐 830011
^4.河海大学，南京 210000

通讯作者: 王磊
作者简介:刘晋文（1999—），女，山西吕梁人，硕士研究生，主要研究方向：有害信息检测
王磊（1974—），男，河南南阳人，研究员，博士，主要研究方向：多语言智能信息处理 Email:wanglei@ms.xjb.ac.cn
马博（1984—），男，辽宁鞍山人，研究员，博士，CCF高级会员，主要研究方向：多语言智能信息处理
董瑞（1985—），男，山东威海人，副研究员，博士，CCF高级会员，主要研究方向：自然语言处理、隐喻检测
杨雅婷（1985—），女，新疆奇台人，研究员，博士，CCF高级会员，主要研究方向：多语言智能信息处理
艾合塔木江·艾合麦提（1997—），男（维吾尔族），新疆喀什人，博士研究生，主要研究方向：多语言多模态语义建模
王欣乐（2003—），女，河南南阳人，主要研究方向：智能科学与技术。
基金资助:
新疆维吾尔自治区自然科学基金重点项目(2023D01D17);“天山英才”科技创新领军人才项目(2022TSYCLJ0046);新疆维吾尔自治区重点研发计划项目(2023B03024);中国科学院青年创新促进会资助项目(Y2021112);新疆维吾尔自治区“天山英才”培养计划项目(2023TSYCCX0041)

Abstract

Abstract:

Proliferation of multimodal harmful content on social media harms public interests and disrupts social order severely at the same time， highlighting the urgent need for effective detection methods of this content. The existing researches rely on pre-trained models to extract and fuse multimodal features， often neglect the limitations of general semantics in harmful content detection tasks， and fail to consider complex， dynamic combinations of harmful content. Therefore， a multimodal harmful content detection method based on weakly Supervised modality semantic enhancement （weak-S） was proposed. In the proposed method， weakly supervised modality information was introduced to facilitate the harmful semantic alignment of multimodal features， and a low-rank bilinear pooling-based multimodal gated integration mechanism was designed to differentiate the contributions of various information. Experimental results show that the proposed method achieves the F1 value improvements of 2.2 and 3.2 percentage points， respectively， on Harm-P and MultiOFF datasets， outperforming SOTA （State-Of-The-Art） models and validating the significance of weakly supervised modality semantics in multimodal harmful content detection. Additionally， the proposed method has improvement in generalization performance for multimodal exaggeration detection tasks.

Key words: unimodal weak supervision, contrastive learning, gated integration, multimodal, harmful content detection

摘要：

社交媒体上多模态有害信息的泛滥不仅损害公众利益，还严重扰乱社会秩序，亟需有效的检测方法。现有研究依赖预训练模型提取与融合多模态特征，忽视了通用语义在有害信息检测任务中的局限性，且未能充分考虑有害信息复杂多变的组合形式。为此，提出一种基于弱监督模态语义增强的多模态有害信息检测方法（weak-S），所提方法通过引入弱监督模态信息辅助多模态特征的有害语义对齐，并设计一种低秩双线性池化的多模态门控集成机制，以区分不同信息的贡献度。实验结果表明，所提方法在Harm-P和MultiOFF数据集上的F1值相较于SOTA （State-Of-The-Art）模型分别提高了2.2和3.2个百分点，验证了弱监督模态语义在多模态有害信息检测中的重要性。此外，所提方法在多模态夸张检测任务上取得了泛化性能的提升。

关键词: 单模态弱监督, 对比学习, 门控集成, 多模态, 有害信息检测

CLC Number:

TP391.7

Jinwen LIU, Lei WANG, Bo MA, Rui DONG, Yating YANG, Ahtamjan Ahmat, Xinyue WANG. Multimodal harmful content detection method based on weakly supervised modality semantic enhancement[J]. Journal of Computer Applications, 2025, 45(10): 3146-3153.

刘晋文, 王磊, 马博, 董瑞, 杨雅婷, 艾合塔木江·艾合麦提, 王欣乐. 基于弱监督模态语义增强的多模态有害信息检测方法[J]. 《计算机应用》唯一官方网站, 2025, 45(10): 3146-3153.

Figures/Tables 8

Fig. 1 Weakly supervised modality semantic enhancement model

Tab. 1 Data distribution and partitioning

数据集	有害性	样本数
数据集	有害性	训练集	测试集	验证集
Harm-P	Hate	1 486	173	86
	Not-hate	1 534	182	91
	合计	3 020	355	177
Harm-C	Hate	1 064	124	61
	Not-hate	1 949	230	116
	合计	3 013	354	177
MultiOFF	Offensive	187	59	59
	Non-offensive	258	90	90
	合计	445	149	149

Tab. 2 Comparison of detection effects of different models on MultiOFF dataset

模型	MultiOFF
模型	Acc↑	Pre↑	Rec↑	F1↑
Stacked LSTM+VGG16	—	0.400	0.660	0.500
BiLSTM+VGG16	—	0.400	0.440	0.410
CNNText+VGG16	—	0.380	0.670	0.480
DisMultiHate	—	0.645	0.651	0.646
MeBERT	—	0.670	0.671	0.671
MemeFier	0.685	—	—	0.625
weak-S	0.711	0.706	0.711	0.703

Tab. 3 Comparison of detection effects of different models on Harm-P and Harm-C datasets

模型	Harm-P			Harm-C
模型	Acc↑	F1↑	MMAE↓	Acc↑	F1↑	MMAE↓
Late Fusion	0.783	0.785	0.167	0.732	0.703	0.293
MMBT	0.825	0.802	0.141	0.735	0.671	0.326
Visual BERT COCO	0.868	0.861	0.132	0.814	0.801	0.186
CLIP	0.879	0.879	0.121	0.825	0.816	0.165
MOMENTA	0.898	0.883	0.131	0.838	0.828	0.174
PromptHate	0.882	0.871	—	0.845	0.815	—
ISSUES	0.881	0.864	0.164	0.848	0.778	0.174
Pro-Cap	—	—	—	0.851	0.839	—
MR.HARM	0.896	0.896	—	0.861	0.854	—
weak-S	0.918	0.918	0.082	0.867	0.853	0.149

Fig. 2 ROC curves of experimental results

Tab. 4 Ablation experimental results

方法	Harm-C			Harm-P			MultiOFF
方法	Acc↑	F1↑	MMAE↓	Acc↑	F1↑	MMAE↓	Acc↑	F1↑	Pre↑	Rec↑
weak-S	0.867	0.853	0.149	0.918	0.918	0.082	0.711	0.703	0.706	0.711
w/o $L u n i$	0.841	0.819	0.191	0.886	0.886	0.113	0.672	0.654	0.665	0.671
w/o $L m c$	0.846	0.828	0.179	0.889	0.889	0.110	0.664	0.666	0.668	0.664
w/o MGI	0.841	0.822	0.184	0.897	0.896	0.102	0.664	0.613	0.682	0.664

Tab. 4 Ablation experimental results

方法	Harm-C			Harm-P			MultiOFF
方法	Acc↑	F1↑	MMAE↓	Acc↑	F1↑	MMAE↓	Acc↑	F1↑	Pre↑	Rec↑
weak-S	0.867	0.853	0.149	0.918	0.918	0.082	0.711	0.703	0.706	0.711
w/o $L u n i$	0.841	0.819	0.191	0.886	0.886	0.113	0.672	0.654	0.665	0.671
w/o $L m c$	0.846	0.828	0.179	0.889	0.889	0.110	0.664	0.666	0.668	0.664
w/o MGI	0.841	0.822	0.184	0.897	0.896	0.102	0.664	0.613	0.682	0.664

Tab. 5 Comparison of multimodal exaggeration detection effects

模型	Acc↑	F1↑	模型	Acc↑	F1↑
CLIP+prompt	0.642	0.632	BriVL+concat	0.667	0.665
CLIP+concat	0.644	0.584	BriVL+gate	0.637	0.644
CLIP+gate	0.642	0.580	weak-S	0.684	0.675
BriVL+prompt	0.628	0.587

Fig. 3 Case analysis for multimodal harmful content detection

References 30

[1]	PRAMANICK S， SHARMA S， DIMITROV D， et al. MOMENTA： a multimodal framework for detecting harmful memes and their targets［C］// Findings of the Association for Computational Linguistics： EMNLP 2021. Stroudsburg： ACL， 2021： 4439-4455.
[2]	KIELA D， FIROOZ H， MOHAN A， et al. The hateful memes challenge： detecting hate speech in multimodal memes［C］// Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2020： 2611-2624.
[3]	SHARMA S， ALAM F， AKHTAR M S， et al. Detecting and understanding harmful memes： a survey［C］// Proceedings of the 31st International Joint Conference on Artificial Intelligence. California： ijcai.org， 2022： 5597-5606.
[4]	KOUTLIS C， SCHINAS M， PAPADOPOULOS S. MemeFier： dual-stage modality fusion for image meme classification［C］// Proceedings of the 2023 ACM International Conference on Multimedia Retrieval. New York： ACM， 2023： 586-591.
[5]	孟杰，王莉，杨延杰，等. 基于多模态深度融合的虚假信息检测［J］. 计算机应用， 2022， 42（2）：419-425.
	MENG J， WANG L， YANG Y J， et al. Multi-modal deep fusion for false information detection［J］. Journal of Computer Applications， 2022， 42（2）：419-425.
[6]	LIN H， LUO Z， MA J， et al. Beneath the surface： unveiling harmful memes with multimodal reasoning distilled from large language models［C］// Findings of the Association for Computational Linguistics： EMNLP 2023. Stroudsburg： ACL， 2023： 9114-9128.
[7]	CAO R， LEE R K W， CHONG W H， et al. Prompting for multimodal hateful meme classification［C］// Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Stroudsburg： ACL， 2022： 321-332.
[8]	CAO R， HEE M S， KUEK A， et al. Pro-Cap： leveraging a frozen vision-language model for hateful meme detection［C］// Proceedings of the 31st ACM International Conference on Multimedia. New York： ACM， 2023： 5244-5252.
[9]	WU F， GAO B， PAN X， et al. Fuser： an enhanced multimodal fusion framework with congruent reinforced perceptron for hateful memes detection［J］. Information Processing and Management， 2024， 61（4）： No.103772.
[10]	LEE R K W， CAO R， FAN Z， et al. Disentangling hate in online memes［C］// Proceedings of the 29th ACM International Conference on Multimedia. New York： ACM， 2021： 5138-5147.
[11]	杜鹏飞. 多模态内容安全识别关键技术研究［D］. 北京：北京邮电大学， 2023.
	DU P F. Research on key technologies of multimodal content security identification［D］. Beijing： Beijing University of Posts and Telecommunications， 2023.
[12]	ZHONG Q， WANG Q， LIU J. Combining knowledge and multi-modal fusion for meme classification［C］// Proceedings of the 2022 International Conference on Multimedia Modeling， LNCS 13141. Cham： Springer， 2022： 599-611.
[13]	GOMEZ R， GIBERT J， GOMEZ L， et al. Exploring hate speech detection in multimodal publications［C］// Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2020： 1459-1467.
[14]	SURYAWANSHI S， CHAKRAVARTHI B R， ARCAN M， et al. Multi modal meme dataset （MultiOFF） for identifying offensive content in image and text［C］// Proceedings of the 2nd Workshop on Trolling， Aggression and Cyberbullying. Stroudsburg： ACL， 2020： 32-41.
[15]	HOSSAIN E， SHARIF O， HOQUE M M， et al. Deciphering hate： identifying hateful memes and their targets［C］// Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg： ACL， 2024： 8347-8359.
[16]	KUMAR G K， NANDAKUMAR K. Hate-CLIPper： multimodal hateful meme classification based on cross-modal interaction of CLIP features［C］// Proceedings of the 2nd Workshop on NLP for Positive Impact. Stroudsburg： ACL， 2022： 171-183.
[17]	BURBI G， BALDRATI A， AGNOLUCCI L， et al. Mapping memes to words for multimodal hateful meme classification［C］// Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway： IEEE， 2023： 2824-2828.
[18]	RADFORD A， KIM J W， HALLACY C， et al. Learning transferable visual models from natural language supervision［C］// Proceedings of the 38th International Conference on Machine Learning. New York： JMLR.org， 2021： 8748-8763.
[19]	MEI J， CHEN J， LIN W， et al. Improving hateful meme detection through retrieval-guided contrastive learning［C］// Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg： ACL， 2024： 5333-5347.
[20]	ZHANG Y， ZHANG H， ZHAN L M， et al. New intent discovery with pre-training and contrastive learning［C］// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics （Volume 1： Long Papers）. Stroudsburg： ACL， 2022： 256-269.
[21]	ZOU H， SHEN M， CHEN C， et al. UniS-MMC： multimodal classification via unimodality-supervised multimodal contrastive learning［C］// Findings of the Association for Computational Linguistics： ACL 2023. Stroudsburg： ACL， 2023： 659-672.
[22]	QU Y， HE X， PIERSON S， et al. On the evolution of （hateful） memes by means of multimodal contrastive learning［C］// Proceedings of the 2023 IEEE Symposium on Security and Privacy. Piscataway： IEEE， 2023： 293-310.
[23]	DEVLIN J， CHANG M W， LEE K， et al. BERT： pre-training of deep bidirectional Transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies， Volume 1 （Long and Short Papers）. Stroudsburg： ACL， 2019： 4171-4186.
[24]	DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An image is worth 16x16 words： Transformers for image recognition at scale［EB/OL］. ［2024-12-04］. .
[25]	HE K， ZHANG X， REN S， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778.
[26]	KIELA D， BHOOSHAN S， FIROOZ H， et al. Supervised multimodal bitransformers for classifying images and text［EB/OL］. ［2024-12-04］. .
[27]	LU J， BATRA D， PARIKH D， et al. ViLBERT： pretraining task-agnostic visiolinguistic representations for vision-and-language tasks［C］// Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2019： 13-23.
[28]	VAN DEN OORD A， LI Y， VINYALS O. Representation learning with contrastive predictive coding［EB/OL］. ［2024-12-04］. .
[29]	ZHANG H， WAN X. Image matters： a new dataset and empirical study for multimodal hyperbole detection［C］// Proceedings of the 2024 Joint International Conference on Computational Linguistics， Language Resources and Evaluation. ［S.l.］： ELRA and ICCL， 2024： 8652-8661.
[30]	HUO Y， ZHANG M， LIU G， et al. WenLan： bridging vision and language by large-scale multi-modal pre-training［EB/OL］. ［2024-12-04］. .

Multimodal harmful content detection method based on weakly supervised modality semantic enhancement

基于弱监督模态语义增强的多模态有害信息检测方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 30

Related Articles 15

Recommended Articles

Metrics

[1]	Chao LIU, Yanhua YU. Knowledge-aware recommendation model combining denoising strategy and multi-view contrastive learning [J]. Journal of Computer Applications, 2025, 45(9): 2827-2837.
[2]	Jinyang HUANG, Fengqi CUI, Changxiu MA, Wendong FAN, Meng LI, Jingyu LI, Xiao SUN, Linsheng HUANG, Zhi LIU. Sleep apnea detection based on universal wristband [J]. Journal of Computer Applications, 2025, 45(9): 3045-3056.
[3]	Zhixiong XU, Bo LI, Xiaoyong BIAN, Qiren HU. Adversarial sample embedded attention U-Net for 3D medical image segmentation [J]. Journal of Computer Applications, 2025, 45(9): 3011-3016.
[4]	Zhiyuan WANG, Tao PENG, Jie YANG. Integrating internal and external data for out-of-distribution detection training and testing [J]. Journal of Computer Applications, 2025, 45(8): 2497-2506.
[5]	Yihan WANG, Chong LU, Zhongyuan CHEN. Multimodal sentiment analysis model with cross-modal text information enhancement [J]. Journal of Computer Applications, 2025, 45(7): 2237-2244.
[6]	Jin XIE, Surong CHU, Yan QIANG, Juanjuan ZHAO, Hua ZHANG, Yong GAO. Dual-branch distribution consistency contrastive learning model for hard negative sample identification in chest X-rays [J]. Journal of Computer Applications, 2025, 45(7): 2369-2377.
[7]	Zhenzhou WANG, Fangfang GUO, Jingfang SU, He SU, Jianchao WANG. Robustness optimization method of visual model for intelligent inspection [J]. Journal of Computer Applications, 2025, 45(7): 2361-2368.
[8]	Jiaqi CHEN, Yulin HE, Yingchao CHENG, Zhexue HUANG. Semi-EM algorithm for solving Gamma mixture model of multimodal probability distribution [J]. Journal of Computer Applications, 2025, 45(7): 2153-2161.
[9]	Wenjing YAN, Ruidong WANG, Min ZUO, Qingchuan ZHANG. Recipe recommendation model based on hierarchical learning of flavor embedding heterogeneous graph [J]. Journal of Computer Applications, 2025, 45(6): 1869-1878.
[10]	Zonghang WU, Dong ZHANG, Guanyu LI. Multimodal fusion recommendation algorithm based on joint self-supervised learning [J]. Journal of Computer Applications, 2025, 45(6): 1858-1868.
[11]	Mingfeng YU, Yongbin QIN, Ruizhang HUANG, Yanping CHEN, Chuan LIN. Multi-label text classification method based on contrastive learning enhanced dual-attention mechanism [J]. Journal of Computer Applications, 2025, 45(6): 1732-1740.
[12]	Chaoying JIANG, Qian LI, Ning LIU, Lei LIU, Lizhen CUI. Readmission prediction model based on graph contrastive learning [J]. Journal of Computer Applications, 2025, 45(6): 1784-1792.
[13]	Qing ZHANG, Fan YANG, Yuhan FANG. Chinese spelling correction algorithm based on multi-modal information fusion [J]. Journal of Computer Applications, 2025, 45(5): 1528-1534.
[14]	Wenbin HU, Tianxiang CAI, Tianle HAN, Zhaoman ZHONG, Changxia MA. Multimodal sarcasm detection model integrating contrastive learning with sentiment analysis [J]. Journal of Computer Applications, 2025, 45(5): 1432-1438.
[15]	Yufei LONG, Yuchen MOU, Ye LIU. Multi-source data representation learning model based on tensorized graph convolutional network and contrastive learning [J]. Journal of Computer Applications, 2025, 45(5): 1372-1378.