Deepfake detection method based on fusion of multi-modal physical prior features

doi:10.11772/j.issn.1001-9081.2025070826

Journal of Computer Applications ›› 2026, Vol. 46 ›› Issue (8): 2515-2523.DOI: 10.11772/j.issn.1001-9081.2025070826

• Cyber security • Previous Articles Next Articles

Deepfake detection method based on fusion of multi-modal physical prior features

Renkun LYU¹, Peng SUN¹^,²(), Yubo LANG¹, Hong GUO², Zhe SHEN³, Di TIAN⁴

^1.College of Public Security Information Technology and Intelligence，Criminal Investigation Police University of China，Shenyang Liaoning 110854，China
^2.Key Laboratory of Forensic Science，Ministry of Justice，Shanghai 200063，China
^3.College of Civil Aviation，Shenyang Aerospace University，Shenyang Liaoning 110136，China
^4.Network Security Corps，Liaoning Provincial Public Security Bureau，Shenyang Liaoning 110032，China

Received:2025-07-23 Revised:2025-09-25 Accepted:2025-09-26 Online:2025-11-05 Published:2026-08-10
Contact: Peng SUN
About author:LYU Renkun， born in 2000， M. S. candidate. His research interests include deepfake detection， image processing.
LANG Yubo， born in 1989， Ph. D. candidate， lecturer. His research interests include image processing， video/image forgery detection.
GUO Hong， born in 1978， M. S. Her research interests include digital forensics， information security.
SHEN Zhe， born in 1981， Ph. D.， lecturer. Her research interests include image processing.
TIAN Di， born in 1975. His research interests include network and information security.
Supported by:
National Natural Science Foundation of China(61307016);Open Project of Key Laboratory of Forensic Science, Ministry of Justice(KF202317);Postgraduate Education and Teaching Reform Research Project of Liaoning Province(LNYJG2023317);Science and Technology Program of Shenyang — Special Project for Social Governance(24-213-3-41);University-Level Key Research Project(ZDGGJH2507)

多模态物理先验特征融合的深度伪造检验方法

吕仁堃¹, 孙鹏¹^,²(), 郎宇博¹, 郭弘², 沈喆³, 田迪⁴

^1.中国刑事警察学院公安信息技术与情报学院，沈阳 110854
^2.司法部司法鉴定重点实验室，上海 200063
^3.沈阳航空航天大学民航学院，沈阳 110136
^4.辽宁省公安厅网络安全总队，沈阳 110032

通讯作者: 孙鹏
作者简介:吕仁堃（2000—），男，山东烟台人，硕士研究生，CCF会员，主要研究方向：深度伪造检验、图像处理
孙鹏（1978—），男，辽宁沈阳人，教授，博士生导师，博士，主要研究方向：视频侦查、图像处理、多媒体取证
郎宇博（1989—），男，辽宁沈阳人，讲师，博士研究生，主要研究方向：图像处理、视频/图像伪造检验
郭弘（1978—），女，上海人，硕士，主要研究方向：电子数据取证、信息安全
沈喆（1981—），女，辽宁沈阳人，讲师，博士，主要研究方向：图像处理
田迪（1975—），男，辽宁沈阳人，主要研究方向：网络信息安全。
基金资助:
国家自然科学基金资助项目(61307016);司法部司法鉴定重点实验室开放课题(KF202317);辽宁省研究生教育教学改革研究项目(LNYJG2023317);沈阳市科技计划项目社会治理科技专项(24-213-3-41);校级重点攻关计划项目(ZDGGJH2507)

Abstract

Abstract:

The existing deepfake detection methods mainly model on the basis of pixel-level clues of images， and seldom consider the impact of the synthesis process on the forged images. Although good detection results are achieved， it is difficult to explain the detection process. Therefore， a multi-modal physical prior feature fusion-based explainable detection method for deepfakes was proposed. First， optical flow features， illumination features， edge features and DCT（Discrete Cosine Transform） features were used to describe the inter-frame motion differences in temporal videos， the illumination inconsistency in single-frame videos， and edge artifact information， respectively， so as to obtain multi-modal physical prior features with explainability. Second， a multi-modal mixture of experts network was proposed to construct expert sub-networks for different modalities， and after cross-modal attention weighting， the sub-networks were fused through a gated unit and input into the discriminative network for classification. Third， the SIAM （Spatial Intersection Attention Module） was introduced into the discriminative network， and the fully connected structure was replaced by the KAN （Kolmogorov-Arnold Network） structure. Finally， the multi-modal physical prior features were used to train different expert sub-networks， respectively， and the Shapley value analysis of different input features was given， thereby constructing a pre-feature-post-explanation explainable analysis framework to provide pixel-level explanations for model inference and prediction. Experimental results show that compared with algorithms such as CORE（COnsistent REpresentation learning）， SRM（Rich Models for Steganalysis）， and UCF（Uncovering Common Features）， the proposed method achieves the best performance on AUC and accuracy， with an accuracy range of 97.35% to 98.75% and an average accuracy of 98.22% on FaceForensics++ dataset， and the model’s interpretability also is improved significantly.

Key words: deepfake detection, physical prior feature, mixture of experts network, interpretability analysis

摘要：

现有深度伪造检验方法多数依据图像像素级线索建模，较少考虑合成过程对伪造图像的影响，尽管取得较好的检验效果，也很难解释检验过程。因此，提出一种多模态物理先验特征融合的深度伪造可解释检验方法。首先，使用光流特征、光照特征、边缘特征和离散余弦变换（DCT）特征分别描述时序视频中的帧间运动差异性、单帧视频中的光照不一致性和边缘伪影信息，得到具有可解释性的多模态物理先验特征；其次，提出多模态混合专家网络，分别构建不同模态的专家子网络，并把子网络在跨模态注意力加权后经门控单元融合，并输入判别网络中实现分类；然后，在判别网络中引入SIAM（Spatial Intersection Attention Module），并将全连接结构替换为KAN（Kolmogorov-Arnold Network）结构；最后，利用多模态物理先验特征分别对不同的专家子网络进行训练，给出对不同输入特征的Shapley值分析，构建事前特征-事后解释的可解释分析框架，为模型推理和预测提供像素级解释。实验结果表明，与CORE（COnsistent REpresentation learning）、SRM（Rich Models for Steganalysis）和UCF（Uncovering Common Features）等算法相比，所提方法在FaceForensics++数据集上的ROC曲线下面积（AUC）和准确率最好，准确率范围为97.35%~98.75%，平均准确率达98.22%，模型可解释性也有较大提升。

关键词: 深度伪造检验, 物理先验特征, 混合专家网络, 可解释性分析

CLC Number:

TP391.41

Renkun LYU, Peng SUN, Yubo LANG, Hong GUO, Zhe SHEN, Di TIAN. Deepfake detection method based on fusion of multi-modal physical prior features[J]. Journal of Computer Applications, 2026, 46(8): 2515-2523.

吕仁堃, 孙鹏, 郎宇博, 郭弘, 沈喆, 田迪. 多模态物理先验特征融合的深度伪造检验方法[J]. 《计算机应用》唯一官方网站, 2026, 46(8): 2515-2523.

Figures/Tables 10

Fig. 1 Flow of proposed method

Fig. 2 Structure of cross-modal attention

Fig. 3 Structure of SC-SIAM module

Fig. 4 Training process of mixture of experts network

Tab. 1 Comparison of detection results using different methods

方法	DF		F2F		FS		NT
方法	ACC/%	AUC	ACC/%	AUC	ACC/%	AUC	ACC/%	AUC
Xception	96.58	0.979 9	96.70	0.978 5	97.23	0.983 3	92.72	0.938 5
Capsule	86.02	0.866 9	85.41	0.863 4	86.29	0.873 4	75.95	0.780 4
Face X-ray	96.51	0.979 4	97.73	0.987 2	97.68	0.987 1	91.84	0.929 0
CORE	96.49	0.978 7	97.22	0.980 3	97.35	0.982 3	92.30	0.933 9
UCF	98.15	0.988 3	97.53	0.984 0	98.07	0.989 6	93.37	0.944 1
F3Net	96.51	0.979 3	96.86	0.979 6	97.58	0.984 4	95.44	0.965 4
SRM	96.87	0.973 3	95.95	0.969 6	96.41	0.974 4	91.39	0.924 5
本文方法	98.75	0.998 7	98.11	0.992 1	98.66	0.996 9	97.35	0.983 5

Tab. 2 Comparison of parameters and floating point operations for different methods

方法	参数量/10⁶	GFLOPs	方法	参数量/10⁶	GFLOPs
Xception	21.86	16.71	UCF	46.95	24.60
Capsule	38.98	32.43	F3Net	22.57	12.84
Face X-ray	77.57	19.50	SRM	55.47	25.23
CORE	21.92	12.25	本文方法	23.35	8.86

Fig. 5 Visualized comparison of t-SNE dimensionality reduction

Tab. 3 Comparison of ablation experimental results with different modal fusions

模型	ACC/%	AUC
$w / o S p a t i a l$	92.65	0.980 1
$w / o T e m p o r a l$	97.42	0.993 5
$w / o F r e q u e n c y$	97.96	0.996 0
本文模型	98.75	0.998 7

Tab. 3 Comparison of ablation experimental results with different modal fusions

模型	ACC/%	AUC
$w / o S p a t i a l$	92.65	0.980 1
$w / o T e m p o r a l$	97.42	0.993 5
$w / o F r e q u e n c y$	97.96	0.996 0
本文模型	98.75	0.998 7

Tab. 4 Comparison of ablation experimental results with different improvements

Backbone	SIAM	KAN	ACC/%	AUC
√			98.08	0.995 8
√	√		98.12	0.997 4
√	√	√	98.75	0.998 7

Fig. 6 Results of Shapley interpretability analysis

References 37

[1]	Amerini I， Galteri L， Caldelli R， et al. Deepfake video detection through optical flow based CNN［C］// ICCVW 2019. Piscataway： IEEE， 2019： 1205-1207.
[2]	Sun D， Yang X， Liu M Y， et al. PWC-Net： CNNs for optical flow using pyramid， warping， and cost volume［C］// CVPR 2018. Piscataway： IEEE， 2018： 8934-8943.
[3]	吴文轩，周文柏，张卫明，等. 基于块间光照不一致性的深度伪造检测算法［J］. 网络与信息安全学报， 2023， 9（1）： 167-177.
	Wu Wenxuan， Zhou Wenbo， Zhang Weiming， et al. Deepfake detection method based on patch-wise lighting inconsistency［J］. Chinese Journal of Network and Information Security， 2023， 9（1）： 167-177.
[4]	Liu Z， Wang Y， Vaidya S， et al. KAN： Kolmogorov-Arnold networks［PP/OL］. V5. arXiv （2025-02-09）［2025-06-20］..
[5]	李纪成，刘琲贝，胡永健，等. 基于光照方向一致性的换脸视频检测［J］. 南京航空航天大学学报， 2020， 52（5）： 760-767.
	Li Jicheng， Liu Beibei， Hu Yongjian， et al. Deepfake video detection based on consistency of illumination direction［J］. Journal of Nanjing University of Aeronautics and Astronautics， 2020， 52（5）： 760-767.
[6]	杨珂，李永亮，何金栋，等. 基于掩码图像建模的深度伪造人脸检测［J］. 计算机应用， 2025， 45（S1）： 72-77.
	Yang Ke， Li Yongliang， He Jindong， et al. Deepfake face detection based on masked image modeling［J］. Journal of Computer Applications， 2025， 45（S1）： 72-77.
[7]	Fu X， Fu B， Chen S， et al. Faces blind your eyes： unveiling the content-irrelevant synthetic artifacts for deepfake detection［J］. IEEE Transactions on Image Processing， 2025， 34： 5686-5696.
[8]	Li L， Bao J， Zhang T， et al. Face X-ray for more general face forgery detection［C］// CVPR 2020. Piscataway： IEEE， 2020： 5000-5009.
[9]	Dong X， Bao J， Chen D， et al. Protecting celebrities from deepfake with identity consistency Transformer［C］// CVPR 2022. Piscataway： IEEE， 2022： 9458-9468.
[10]	Ganguly S， Ganguly A， Mohiuddin S， et al. ViXNet： vision Transformer with Xception network for deepfakes based video and image forgery detection［J］. Expert Systems with Applications， 2022， 210： No.118423.
[11]	Qian Y， Yin G， Sheng L， et al. Thinking in frequency： face forgery detection by mining frequency-aware clues［C］// ECCV 2020， LNCS 12357. Cham： Springer， 2020： 86-103.
[12]	Cheng Z， Wang Y， Wan Y， et al. DeepFake detection method based on multi-scale interactive dual-stream network［J］. Journal of Visual Communication and Image Representation， 2024， 104： No.104263.
[13]	Wang J， Wu Z， Ouyang W， et al. M2TR： multi-modal multi-scale transformers for deepfake detection［C］// ICMR 2022. New York： ACM， 2022： 615-623.
[14]	Cozzolino D， Rössler A， Thies J， et al. ID-Reveal： identity-aware deepfake video detection［C］// ICCV 2021. Piscataway： IEEE， 2021： 15088-15097.
[15]	Conotter V， Bodnari E， Boato G， et al. Physiologically-based detection of computer generated faces in video［C］// ICIP 2014. Piscataway： IEEE， 2014： 248-252.
[16]	Ciftci U A， Demir I， Yin L. FakeCatcher： detection of synthetic portrait videos using biological signals［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2020（Early Access）： 1.
[17]	Li Y， Chang M C， Lyu S. In ictu oculi： exposing AI created fake videos by detecting eye blinking［C］// WIFS 2018. Piscataway： IEEE， 2018： 1-7.
[18]	Sun Z， Han Y， Hua Z， et al. Improving the efficiency and robustness of deepfakes detection through precise geometric features［C］// CVPR 2021. Piscataway： IEEE， 2021： 3608-3617.
[19]	Cozzolino D， Pianese A， Nießner M， et al. Audio-visual person-of-interest deepfake detection［C］// CVPR 2023. Piscataway： IEEE， 2023： 943-952.
[20]	肖景博，殷琪林，卢伟，等. 基于视频流谱特征空间的深度伪造检测［J］. 中国科学：信息科学， 2024， 54（11）： 2572-2588.
	Xiao Jingbo， Yin Qilin， Lu Wei， et al. Deepfake detection based on video flow spectrum feature space［J］. SCIENTIA SINICA Informationis， 2024， 54（11）： 2572-2588.
[21]	Zhou Y， Lim S N. Joint audio-visual deepfake detection［C］// ICCV 2021. Piscataway： IEEE， 2021： 1480-1489.
[22]	Cai Z， Ghosh S， Dhall A， et al. Glitch in the matrix： a large scale benchmark for content driven audio-visual forgery detection and localization［J］. Computer Vision and Image Understanding， 2023， 236： No.103818.
[23]	Jacobs R A， Jordan M I， Nowlan S J， et al. Adaptive mixtures of local experts［J］. Neural Computation， 1991， 3（1）： 79-87.
[24]	Negroni V， Salvi D， Mezza A I， et al. Leveraging mixture of experts for improved speech deepfake detection［C］// ICASSP 2025. Piscataway： IEEE， 2025： 1-5.
[25]	Shazeer N， Mirhoseini A， Maziarz K， et al. Outrageously large neural networks： the sparsely-gated mixture-of-experts layer［EB/OL］. （2025-10-12）［2025-10-20］..
[26]	Rozemberczki B， Watson L， Bayer P， et al. The shapley value in machine learning［C］// IJCAI 2022. California： IJCAI， 2022： 5572-5579.
[27]	Han G， Huang S， Zhao F， et al. SIAM： a parameter-free， spatial intersection attention module［J］. Pattern Recognition， 2024， 153： No.110509.
[28]	Sun D， Roth S， Black M J. Secrets of optical flow estimation and their principles［C］// CVPR 2010. Piscataway： IEEE， 2010： 2432-2439.
[29]	Huang Z， Shi X， Zhang C， et al. FlowFormer： a Transformer architecture for optical flow［C］// ECCV 2022， LNCS 13677. Cham： Springer， 2022： 668-685.
[30]	Yao L， Lin Y， Muhammad S. An improved multi-scale image enhancement method based on retinex theory［J］. Journal of Medical Imaging and Health Informatics， 2018， 8（1）： 122-126.
[31]	Zhang H， Hu C， Min S， et al. TSFF-Net： a deep fake video detection model based on two-stream feature domain fusion［J］. PLoS ONE， 2024， 19（12）： No.e0311366.
[32]	Rössler A， Cozzolino D， Verdoliva L， et al. FaceForensics++： learning to detect manipulated facial images［C］// ICCV 2019. Piscataway： IEEE， 2019： 1-11.
[33]	Chollet F. Xception： deep learning with depthwise separable convolutions［C］// CVPR 2017.Piscataway： IEEE，2017：1800-1807.
[34]	Nguyen H H， Yamagishi J， Echizen I. Capsule-forensics： using capsule networks to detect forged images and videos［C］// ICASSP 2019. Piscataway： IEEE， 2019： 2307-2311.
[35]	Ni Y， Meng D， Yu C， et al. CORE： consistent representation learning for face forgery detection［C］// CVPR 2022. Piscataway： IEEE， 2022： 12-21.
[36]	Yan Z， Zhang Y， Fan Y， et al. UCF： uncovering common features for generalizable deepfake detection［C］// ICCV 2023. Piscataway： IEEE， 2023： 22355-22366.
[37]	Luo Y， Zhang Y， Yan J， et al. Generalizing face forgery detection with high-frequency features［C］// CVPR 2021. Piscataway： IEEE， 2021： 16312-16321.

Deepfake detection method based on fusion of multi-modal physical prior features

多模态物理先验特征融合的深度伪造检验方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 37

Related Articles 2

Recommended Articles

Metrics

[1]	Shengyou ZHENG, Yanxiang CHEN, Zuxing ZHAO, Haiyang LIU. Construction and benchmark detection of multimodal partial forgery dataset [J]. Journal of Computer Applications, 2024, 44(10): 3134-3140.
[2]	ZHANG Ya, JIN Xin, JIANG Qian, LEE Shin-jye, DONG Yunyun, YAO Shaowen. Deepfake image detection method based on autoencoder [J]. Journal of Computer Applications, 2021, 41(10): 2985-2990.