Multi-modal deep fusion for false information detection

doi:10.11772/j.issn.1001-9081.2021071184

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (2): 419-425.DOI: 10.11772/j.issn.1001-9081.2021071184

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Multi-modal deep fusion for false information detection

Jie MENG¹, Li WANG¹(), Yanjie YANG¹, Biao LIAN²

^1.College of Data Science，Taiyuan University of Technology，Taiyuan Shanxi 030600，China
^2.North Automatic Control Technology Institute，Taiyuan Shanxi 030006，China

Received:2021-07-09 Revised:2021-07-18 Accepted:2021-07-21 Online:2022-02-11 Published:2022-02-10
Contact: Li WANG
About author:MENG Jie， born in 1994， M. S. candidate. His research interests include natural language processing， false information detection.
WANG Li， born in 1971， Ph. D， professor. Her research interests include big data computing and analysis， data mining.
YANG Yanjie， born in 1995， M. S. candidate. His research interests include natural language processing， data mining.
LIAN Biao， born in 1987， M. S. His research interests include software development， data mining.
Supported by:
National Natural Science Foundation of China(61872260)

基于多模态深度融合的虚假信息检测

孟杰¹, 王莉¹(), 杨延杰¹, 廉飚²

^1.太原理工大学大数据学院，太原 030600
^2.北方自动控制技术研究所，太原 030006

通讯作者: 王莉
作者简介:孟杰（1994—），男，山西长治人，硕士研究生，主要研究方向：自然语言处理、虚假信息检测；
王莉（1971—），女，山西太原人，教授，博士，CCF高级会员，主要研究方向：大数据计算与分析、数据挖掘；
杨延杰（1995—），男，山西原平人，硕士研究生，主要研究方向：自然语言处理、数据挖掘；
廉飚（1987—），男，山西太原人，硕士，主要研究方向：软件开发、数据挖掘。
基金资助:
国家自然科学基金资助项目(61872260)

Abstract

Abstract:

Concerning the problem of insufficient image feature extraction and ignorance of single-modal internal relations and the interactions between single-modal and multi-modal， a text and image information based Multi-Modal Deep Fusion （MMDF） model was proposed. Firstly， the Bi-Gated Recurrent Unit （Bi-GRU） was used to extract the rich semantic features of the text， and the multi-branch Convolutional-Recurrent Neural Network （CNN-RNN） was used to extract the multi-level features of the image. Then the inter-modal and intra-modal attention mechanisms were established to capture the high-level interaction between the fields of language and vision， and the multi-modal joint representation was obtained. Finally， the original representation of each modal and the fused multi-modal joint representation were re-fused according to their attention weights to strengthen the role of the original information. Compared with the Multimodal Variational AutoEncoder （MVAE） model， the proposed model has the accuracy improved by 1.9 percentage points and 2.4 percentage points on the China Computer Federation （CCF） competition and the Weibo datasets respectively. Experimental results show that the proposed model can fully fuse multi-modal information and effectively improve the accuracy of false information detection.

Key words: false information detection, multi-modal fusion, Bi-directional Gated Recurrent Unit (Bi-GRU), attention mechanism, joint representation

摘要：

针对虚假信息检测中图片特征提取不充分，以及忽视了单模内关系以及单模与多模之间交互作用的问题，提出一种基于文本和图片信息的多模态深度融合（MMDF）模型。首先，用双向门控循环单元（Bi-GRU）提取文本的丰富语义特征，用多分支卷积-循环神经网络（CNN-RNN）提取图片的多层次特征；然后，建立模间和模内的注意力机制以捕获语言和视觉领域之间的高层交互，并得到多模态的联合表征；最后，将各模态原表征与融合后的多模态联合表征依据注意力权重进行再融合，以加强原信息的作用。该模型与多模态变分自动编码器（MVAE）模型相比，在中国计算机学会（CCF）竞赛和微博数据集上的准确率分别提升了1.9个百分点和2.4个百分点。实验结果表明，所提模型能够充分融合多模态信息，有效提高虚假信息检测的准确率。

关键词: 虚假信息检测, 多模态融合, 双向门控循环单元, 注意力机制, 联合表征

CLC Number:

TP391

Jie MENG, Li WANG, Yanjie YANG, Biao LIAN. Multi-modal deep fusion for false information detection[J]. Journal of Computer Applications, 2022, 42(2): 419-425.

孟杰, 王莉, 杨延杰, 廉飚. 基于多模态深度融合的虚假信息检测[J]. 《计算机应用》唯一官方网站, 2022, 42(2): 419-425.

Figures/Tables 7

References 23

1	KOUZY R， JAOUDE J ABI， KRAITEM A， et al. Coronavirus goes viral： quantifying the COVID-19 misinformation epidemic on Twitter［J］. Cureus， 2020， 12（3）： No.e7255. 10.7759/cureus.7255
2	RASHKIN H， CHOI E， JANG J Y， et al. Truth of varying shades： analyzing language in fake news and political fact-checking ［C］// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg， PA： Association for Computational Linguistics， 2017： 2931-2937. 10.18653/v1/d17-1317
3	POPAT K， MUKHERJEE S， STRÖTGEN J， et al. Credibility assessment of textual claims on the Web ［C］// Proceedings of the 25th ACM International Conference on Information and Knowledge Management. New York： ACM， 2016： 2173-2178. 10.1145/2983323.2983661
4	MA J， GAO W， MITRA P， et al. Detecting rumors from microblogs with recurrent neural networks ［C］// Proceedings of the 25th International Joint Conference on Artificial Intelligence. ［S.l.］： IJCAI Organization， 2016： 3818-3824.
5	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［EB/OL］. （2015-04-10）［2021-03-10］. . 10.5244/c.28.6
6	QI P， CAO J， YANG T Y， et al. Exploiting multi-domain visual information for fake news detection ［C］// Proceedings of the 2019 IEEE International Conference on Data Mining. Piscataway： IEEE， 2019： 518-527. 10.1109/icdm.2019.00062
7	RUCHANSKY N， SEO S， LIU Y. CSI： a hybrid deep model for fake news detection ［C］// Proceedings of the 2017 ACM Conference on Information and Knowledge Management. New York： ACM， 2017： 797-806. 10.1145/3132847.3132877
8	SHU K， SLIVA A， WANG S H， et al. Fake news detection on social media： a data mining［J］. ACM SIGKDD Explorations Newsletter， 2017， 19（1）： 22-36. 10.1145/3137597.3137600
9	毛二松，陈刚，刘欣，等.基于深层特征和集成分类器的微博谣言检测研究［J］.计算机应用研究， 2016， 33（11）： 3369-3373. 10.3969/j.issn.1001--3695.2016.11.037
	MAO E S， CHEN G， LIU X， et al. Research on detecting micro-blog rumors based on deep features and ensemble classifier［J］. Application Research of Computers， 2016， 33（11）： 3369-3373. 10.3969/j.issn.1001--3695.2016.11.037
10	刘政，卫志华，张韧弦.基于卷积神经网络的谣言检测［J］.计算机应用， 2017， 37（11）： 3053-3056， 3100. 10.11772/j.issn.1001-9081.2017.11.3053
	LIU Z， WEI Z H， ZHANG R X. Rumor detection based on convolutional neural network［J］. Journal of Computer Applications， 2017， 37（11）： 3053-3056， 3100. 10.11772/j.issn.1001-9081.2017.11.3053
11	MA J， GAO W， WONG K F. Detect rumors on Twitter by promoting information campaigns with generative adversarial learning ［C］// Proceedings of the 2019 World Wide Web Conference. New York： ACM， 2019： 3049-3055. 10.1145/3308558.3313741
12	JIN Z W， CAO J， ZHANG Y D， et al. Novel visual and statistical image features for microblogs news verification［J］. IEEE Transactions on Multimedia， 2017， 19（3）： 598-608. 10.1109/tmm.2016.2617078
13	SINGHAL S， SHAH R R， CHAKRABORTY T， et al. SpotFake： a multi-modal framework for fake news detection ［C］// Proceedings of the IEEE 5th International Conference on Multimedia Big Data. Piscataway： IEEE， 2019： 39-47. 10.1109/bigmm.2019.00-44
14	JIN Z W， CAO J， GUO H， et al. Multimodal fusion with recurrent neural networks for rumor detection on microblogs ［C］// Proceedings of the 25th ACM International Conference on Multimedia. New York： ACM， 2017： 795-816. 10.1145/3123266.3123454
15	SONG C G， NING N W， ZHANG Y L， et al. A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks［J］. Information Processing and Management， 2021， 58（1）： No.102437. 10.1016/j.ipm.2020.102437
16	WANG Y Q， MA F L， JIN Z W， et al. EANN： event adversarial neural networks for multi-modal fake news detection ［C］// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York： ACM， 2018： 849-857. 10.1145/3219819.3219903
17	ZHANG H W， FANG Q， QIAN S S， et al. Multi-modal knowledge-aware event memory network for social media rumor detection ［C］// Proceedings of the 27th ACM International Conference on Multimedia. New York： ACM， 2019： 1942-1951. 10.1145/3343031.3350850
18	KHATTAR D， GOUD J S， GUPTA M， et al. MVAE： multimodal variational autoencoder for fake news detection ［C］// Proceedings of the 2019 World Wide Web Conference. New York： ACM， 2019： 2915-2921. 10.1145/3308558.3313552
19	ZHOU X Y， WU J D， ZAFARANI R. SAFE： similarity-aware multi-modal fake news detection ［C］// Proceedings of the 2020 Pacific-Asia Conference on Knowledge Discovery and Data Mining， LNCS12085. Cham： Springer， 2020： 354-367.
20	XUE J X， WANG Y B， TIAN Y C， et al. Detecting fake news by exploring the consistency of multimodal data［J］. Information Processing and Management， 2021， 58（5）： No.102610. 10.1016/j.ipm.2021.102610
21	MIKOLOV T， CHEN K， CORRADO G， et al. Efficient estimation of word representations in vector space［EB/OL］. （2013-09-07）［2021-03-10］. . 10.3126/jiee.v3i1.34327
22	ANTOL S， AGRAWAL A， LU J S， et al. VQA： visual question answering ［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 2425-2433. 10.1109/iccv.2015.279
23	VINYALS O， TOSHEV A， BENGIO S， et al. Show and tell： a neural image caption generator ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 3156-3164. 10.1109/cvpr.2015.7298935

数据集	虚假信息数	真实信息数	图片数
Weibo	4 749	4 779	9 528
CCF竞赛	4 324	5 521	9 845

数据集	虚假信息数	真实信息数	图片数
Weibo	4 749	4 779	9 528
CCF竞赛	4 324	5 521	9 845

数据集	基准模型	准确率	虚假信息			真实信息
数据集	基准模型	准确率	Precision	Recall	F1	Precision	Recall	F1
CCF竞赛	Text	0.613	0.664	0.532	0.538	0.601	0.66	0.659
	Visual	0.571	0.517	0.712	0.599	0.659	0.455	0.539
	VQA	0.715	0.814	0.622	0.706	0.636	0.832	0.719
	NeuralTalk	0.681	0.733	0.661	0.695	0.634	0.674	0.633
	att-RNN	0.739	0.788	0.651	0.712	0.682	0.805	0.741
	EANN	0.766	0.812	0.623	0.705	0.741	0.822	0.806
	MVAE	0.774	0.808	0.736	0.771	0.743	0.814	0.777
	MMDF	0.793	0.821	0.689	0.779	0.776	0.877	0.823
Weibo	Text	0.643	0.662	0.578	0.617	0.609	0.685	0.647
	Visual	0.608	0.610	0.605	0.607	0.607	0.611	0.609
	VQA	0.736	0.797	0.634	0.706	0.695	0.838	0.760
	NeuralTalk	0.726	0.794	0.713	0.692	0.684	0.840	0.754
	att-RNN	0.772	0.854	0.656	0.742	0.72	0.889	0.795
	EANN	0.793	0.796	0.806	0.801	0.790	0.780	0.785
	MVAE	0.814	0.765	0.874	0.833	0.863	0.734	0.775
	MMDF	0.838	0.815	0.886	0.849	0.866	0.786	0.824

数据集	基准模型	准确率	虚假信息			真实信息
数据集	基准模型	准确率	Precision	Recall	F1	Precision	Recall	F1
CCF竞赛	Text	0.613	0.664	0.532	0.538	0.601	0.66	0.659
	Visual	0.571	0.517	0.712	0.599	0.659	0.455	0.539
	VQA	0.715	0.814	0.622	0.706	0.636	0.832	0.719
	NeuralTalk	0.681	0.733	0.661	0.695	0.634	0.674	0.633
	att-RNN	0.739	0.788	0.651	0.712	0.682	0.805	0.741
	EANN	0.766	0.812	0.623	0.705	0.741	0.822	0.806
	MVAE	0.774	0.808	0.736	0.771	0.743	0.814	0.777
	MMDF	0.793	0.821	0.689	0.779	0.776	0.877	0.823
Weibo	Text	0.643	0.662	0.578	0.617	0.609	0.685	0.647
	Visual	0.608	0.610	0.605	0.607	0.607	0.611	0.609
	VQA	0.736	0.797	0.634	0.706	0.695	0.838	0.760
	NeuralTalk	0.726	0.794	0.713	0.692	0.684	0.840	0.754
	att-RNN	0.772	0.854	0.656	0.742	0.72	0.889	0.795
	EANN	0.793	0.796	0.806	0.801	0.790	0.780	0.785
	MVAE	0.814	0.765	0.874	0.833	0.863	0.734	0.775
	MMDF	0.838	0.815	0.886	0.849	0.866	0.786	0.824

对比模型	CCF竞赛	Weibo
MMDF和MVAE	t=9.016，p=1.721E-5	t=12.079，p=2.548E-6
MMDF和EANN	t=12.961，p=2.398E-6	t=24.382，p=4.381E-9
MMDF和att-RNN	t=18.365，p=1.964E-6	t=28.896，p=2.333E-8

Multi-modal deep fusion for false information detection

基于多模态深度融合的虚假信息检测

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 23

Related Articles 15

Recommended Articles

Metrics

[1]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[2]	Ying HUANG, Jiayu YANG, Jiahao JIN, Bangrui WAN. Siamese mixed information fusion algorithm for RGBT tracking [J]. Journal of Computer Applications, 2024, 44(9): 2878-2885.
[3]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[4]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[5]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.
[6]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[7]	Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594.
[8]	Shangbin MO, Wenjun WANG, Ling DONG, Shengxiang GAO, Zhengtao YU. Single-channel speech enhancement based on multi-channel information aggregation and collaborative decoding [J]. Journal of Computer Applications, 2024, 44(8): 2611-2617.
[9]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[10]	Song XU, Wenbo ZHANG, Yifan WANG. Lightweight video salient object detection network based on spatiotemporal information [J]. Journal of Computer Applications, 2024, 44(7): 2192-2199.
[11]	Dahai LI, Zhonghua WANG, Zhendong WANG. Dual-branch low-light image enhancement network combining spatial and frequency domain information [J]. Journal of Computer Applications, 2024, 44(7): 2175-2182.
[12]	Wenliang WEI, Yangping WANG, Biao YUE, Anzheng WANG, Zhe ZHANG. Deep learning model for infrared and visible image fusion based on illumination weight allocation and attention [J]. Journal of Computer Applications, 2024, 44(7): 2183-2191.
[13]	Wu XIONG, Congjun CAO, Xuefang SONG, Yunlong SHAO, Xusheng WANG. Handwriting identification method based on multi-scale mixed domain attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2225-2232.
[14]	Huanhuan LI, Tianqiang HUANG, Xuemei DING, Haifeng LUO, Liqing HUANG. Public traffic demand prediction based on multi-scale spatial-temporal graph convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2065-2072.
[15]	Dianhui MAO, Xuebo LI, Junling LIU, Denghui ZHANG, Wenjing YAN. Chinese entity and relation extraction model based on parallel heterogeneous graph and sequential attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2018-2025.