U-shaped feature pyramid network for image inpainting forensics

doi:10.11772/j.issn.1001-9081.2021122107

Abstract

Abstract:

Image inpainting is a common method of image tampering. Image inpainting methods based on deep learning can generate more complex structures and even new objects， making image inpainting forensics more challenging. Therefore， an end-to-end U-shaped Feature Pyramid Network （FPN） was proposed for image inpainting forensics. Firstly， multi-scale feature extraction was performed through the from-top-to-down VGG16 module， and then the from-bottom-to-up feature pyramid architecture was used to carry out up-sampling of the fused feature maps， and a U-shaped structure was formed by the overall process. Next， the global and local attention mechanisms were combined to highlight the inpainting traces. Finally， the fusion loss function was used to improve the prediction rate of the repaired area. Experimental results show that the proposed method achieves an average F1-score and Intersection over Union （IoU） value of 0.791 9 and 0.747 2 respectively on various deep inpainting datasets. Compared with the existing Localization of Diffusion-based Inpainting （LDI）， Patch-based Convolutional Neural Network （Patch-CNN） and High-Pass Fully Convolutional Network （HP-FCN） methods， the proposed method has better generalization ability， and also has stronger robustness to JPEG compression.

Key words: digital image forensics, deep image inpainting, tampering detection, Feature Pyramid Network (FPN), fusion loss

摘要：

图像修复是一种常见的图像篡改手段，而基于深度学习的图像修复方法能生成更复杂的结构乃至新的对象，使得图像修复取证工作更具有挑战性。因此，提出一种端到端的面向图像修复取证的U型特征金字塔网络（FPN）。首先，通过自上而下的VGG16模块进行多尺度特征提取，并利用自下而上的特征金字塔架构对融合后的特征图进行上采样，整体流程形成U型结构；然后，结合全局和局部注意力机制凸显修复痕迹；最后，使用融合损失函数以提高修复区域的预测率。实验结果表明，所提方法在多种深度修复数据集上的平均F1分数和IoU值分别为0.791 9和0.747 2，与现有的基于扩散的数字图像修复定位（LDI）、基于图像块的深度修复取证方法（Patch-CNN）和基于高通全卷积神经网络（HP-FCN）方法相比，所提方法具有更好的泛化能力，且对JPEG压缩也具有较强的鲁棒性。

关键词: 数字图像取证, 深度图像修复, 篡改检测, 特征金字塔网络, 融合损失

CLC Number:

TP309.2

Wanli SHEN, Yujin ZHANG, Wan HU. U-shaped feature pyramid network for image inpainting forensics[J]. Journal of Computer Applications, 2023, 43(2): 545-551.

沈万里, 张玉金, 胡万. 面向图像修复取证的U型特征金字塔网络[J]. 《计算机应用》唯一官方网站, 2023, 43(2): 545-551.

Figures/Tables 11

References 23

1	ZHANG Y L， DING F， KWONG S， et al. Feature pyramid network for diffusion-based image inpainting detection［J］. Information Sciences， 2021， 572： 29-42. 10.1016/j.ins.2021.04.042
2	冯浪，张玲，张晓龙. 基于扩张卷积的图像修复［J］. 计算机应用， 2020， 40（3）：825-831. 10.11772/j.issn.1001-9081.2019081471
	FENG L， ZHANG L， ZHANG X L. Image inpainting based on dilated convolution［J］. Journal of Computer Applications， 2020， 40（3）：825-831. 10.11772/j.issn.1001-9081.2019081471
3	ELHARROUSS O， ALMAADEED N， AL-MAADEED S， et al. Image inpainting： a review［J］. Neural Processing Letters， 2020， 51（2）： 2007-2028. 10.1007/s11063-019-10163-0
4	刘婷婷，张玉金，吴飞，等. 基于梯度域导向滤波增强的图像扩散修复取证［J］. 激光与光电子学进展， 2020， 57（8）：43-50. 10.3788/lop57.081003
	LIU T T， ZHANG Y J， WU F， et al. Diffusion-based image inpainting forensics via gradient domain guided filtering enhancement［J］. Laser and Optoelectronics Progress， 2020， 57（8）：43-50. 10.3788/lop57.081003
5	LI H D， LUO W Q， HUANG J W. Localization of diffusion-based inpainting in digital images［J］. IEEE Transactions on Information Forensics and Security， 2017， 12（12）： 3050-3064. 10.1109/tifs.2017.2730822
6	ZHU X S， QIAN Y J， ZHAO X F， et al. A deep learning approach to patch-based image inpainting forensics［J］. Signal Processing： Image Communication， 2018， 67： 90-99. 10.1016/j.image.2018.05.015
7	KUMAR N， MEENPAL T. Semantic segmentation-based image inpainting detection［M］// FAVORSKAYA M N， MEKHILEF S， PANDEY R K， et al. Innovations in Electrical and Electronic Engineering， LNEE 661. Singapore： Springer， 2021： 665-677. 10.1007/978-981-15-4692-1_51
8	ZHU X S， QIAN Y J， ZHAO X F， et al. A deep learning approach to patch-based image inpainting forensics［J］. Signal Processing： Image Communication， 2018， 67： 90-99. 10.1016/j.image.2018.05.015
9	WU Y， ABDALMAGEED W， NATARAJAN P. ManTra-Net： manipulation tracing network for detection and localization of image forgeries with anomalous features［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 9535-9544. 10.1109/cvpr.2019.00977
10	LI H D， HUANG J W. Localization of deep inpainting using high-pass fully convolutional network［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 8300-8309. 10.1109/iccv.2019.00839
11	GE S M， LI C Y， ZHAO S W， et al. Occluded face recognition in the wild by identity-diversity inpainting［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2020， 30（10）： 3387-3397. 10.1109/tcsvt.2020.2967754
12	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［EB/OL］. （2015-04-10）［2021-12-12］..
13	LIN T Y， DOLLÁR P， GIRSHICK R， et al. Feature pyramid networks for object detection［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 936-944. 10.1109/cvpr.2017.106
14	BOCHKOVSKIY A， WANG C Y， LIAO H Y M. YOLOv4： optimal speed and accuracy of object detection［EB/OL］. （2020-04-23）［2022-02-26］..
15	朱新山，钱永军，孙彪，等. 基于深度神经网络的图像修复取证算法［J］. 光学学报， 2018， 38（11）： No.1110005. 10.3788/aos201838.1110005
	ZHU X S， QIAN Y J， SUN B， et al. Image inpainting forensics algorithm based on deep neural network［J］. Acta Optica Sinica， 2018， 38（11）： No.1110005. 10.3788/aos201838.1110005
16	YU J， LIN Z， YANG J， et al. Generative image inpainting with contextual attention［C］// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 5505-5514. 10.1109/cvpr.2018.00577
17	WU H W， ZHOU J T. GIID-Net： generalizable image inpainting detection via neural architecture search and attention［EB/OL］. （2021-01-29）［2022-02-26］.. 10.1109/icip42928.2021.9506778
18	YAN Z Y， LI X M， LI M， et al. Shift-Net： image inpainting via deep feature rearrangement［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11218. Cham： Springer， 2018： 3-19.
19	DENG J， DONG W， SOCHER R， et al. ImageNet： a large-scale hierarchical image database［C］// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2009：248-255. 10.1109/cvpr.2009.5206848
20	IIZUKA S， SIMO-SERRA E， ISHIKAWA H. Globally and locally consistent image completion［J］. ACM Transactions on Graphics， 2017， 36（4）： No.107. 10.1145/3072959.3073659
21	WU H W， ZHOU J T， LI Y M. Deep generative model for image inpainting with local binary pattern learning and spatial attention［J］. IEEE Transactions on Multimedia， 2022， 24： 4016-4027. 10.1109/tmm.2021.3111491
22	YU T， GUO Z Y， JIN X， et al. Region normalization for image inpainting［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 12733-12740. 10.1609/aaai.v34i07.6967
23	NAZERI K， NG E， JOSEPH T， et al. EdgeConnect： generative image inpainting with adversarial edge learning［EB/OL］. （2019-01-11）［2022-02-26］.. 10.1109/iccvw.2019.00408

Input	Operator	Filter	Channel
M×N×3	Conv1-1	3×3	64
M×N×64	Conv1-2	3×3	64
M×N×64	Max-pooling	2×2	128
M×N×128	Conv2-1	3×3	128
M×N×128	Conv2-2	3×3	128
M×N×128	Max-pooling	2×2	256
M×N×256	Conv3-1	3×3	256
M×N×256	Conv3-2	3×3	256
M×N×256	Conv3-3	3×3	256
M×N×256	Max-pooling	2×2	512
M×N×512	Conv4-1	3×3	512
M×N×512	Conv4-2	3×3	512
M×N×512	Conv4-3	3×3	512
M×N×512	Max-pooling	2×2	512
M×N×512	Conv5-1	3×3	512
M×N×512	Conv5-2	3×3	512
M×N×512	Conv5-3	3×3	512

Input	Operator	Filter	Channel
M×N×3	Conv1-1	3×3	64
M×N×64	Conv1-2	3×3	64
M×N×64	Max-pooling	2×2	128
M×N×128	Conv2-1	3×3	128
M×N×128	Conv2-2	3×3	128
M×N×128	Max-pooling	2×2	256
M×N×256	Conv3-1	3×3	256
M×N×256	Conv3-2	3×3	256
M×N×256	Conv3-3	3×3	256
M×N×256	Max-pooling	2×2	512
M×N×512	Conv4-1	3×3	512
M×N×512	Conv4-2	3×3	512
M×N×512	Conv4-3	3×3	512
M×N×512	Max-pooling	2×2	512
M×N×512	Conv5-1	3×3	512
M×N×512	Conv5-2	3×3	512
M×N×512	Conv5-3	3×3	512

实验	Recall	Precision	IoU	F1
场景1	82.22	92.16	76.63	84.54
场景2	82.71	95.17	79.42	86.60
场景3	96.18	87.88	84.92	91.84
场景4	99.17	94.78	94.03	96.92

实验	Recall	Precision	IoU	F1
场景1	82.22	92.16	76.63	84.54
场景2	82.71	95.17	79.42	86.60
场景3	96.18	87.88	84.92	91.84
场景4	99.17	94.78	94.03	96.92

数据集	LDI		Patch-CNN		HP-FCN		本文方法
数据集	F1	IoU	F1	IoU	F1	IoU	F1	IoU
平均	41.65	33.60	57.17	41.95	66.94	59.22	79.19	74.72
GC	58.56	46.35	56.08	49.48	76.93	72.15	80.69	78.15
CA	39.43	23.12	58.37	41.22	52.38	44.58	80.05	78.29
SH	39.24	26.13	65.86	49.10	71.43	62.85	78.94	76.01
LB	48.46	37.27	49.56	32.94	68.57	60.10	78.17	70.95
RN	51.14	39.33	54.40	37.36	65.78	57.12	79.15	74.09
EC	43.91	29.42	58.76	41.61	66.58	58.48	77.67	70.83