Moving portrait debluring network based on multi-level jump residual group

doi:10.11772/j.issn.1001-9081.2022091457

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (10): 3244-3250.DOI: 10.11772/j.issn.1001-9081.2022091457

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Moving portrait debluring network based on multi-level jump residual group

Jiaqi JI, Zhenkun LU(), Fupeng XIONG, Tian ZHANG, Hao YANG

College of Electronic Information，Guangxi Minzu University，Nanning Guangxi 530006，China

Received:2022-10-08 Revised:2023-01-03 Accepted:2023-02-01 Online:2023-04-04 Published:2023-10-10
Contact: Zhenkun LU
About author:JI Jiaqi， born in 1997， M. S. candidate. His research interests include deep learning， image processing.
LU Zhenkun， born in 1979， Ph. D.， professor. His research interests include image processing， ultrasound testing and imaging，computer vision， deep learning.
XIONG Fupeng， born in 1998， M. S. candidate. His research interests include deep learning.
ZHANG Tian， born in 1999， M. S. candidate. Her researchinterests include deep learning， image processing.
YANG Hao， born in 1995， M. S. candidate. His research interests include deep learning， image processing.
Supported by:
National Natural Science Foundation of China(61561008);Natural Science Foundation of Guangxi(2018GXNSFAA294019)

基于多级跳跃残差组的运动人像去模糊网络

纪佳奇, 卢振坤(), 熊福棚, 张甜, 杨豪

广西民族大学电子信息学院，南宁 530006

通讯作者: 卢振坤
作者简介:纪佳奇（1997—），男，江苏徐州人，硕士研究生，主要研究方向：深度学习、图像处理
卢振坤（1979—），男，广西百色人，教授，博士，CCF会员，主要研究方向：图像处理、超声检测与成像、计算机视觉、深度学习.lzk06@sina.com
熊福棚（1998—），男，河南光山人，硕士研究生，主要研究方向：深度学习
张甜（1999—），女，陕西黄陵人，硕士研究生，主要研究方向：深度学习、图像处理
杨豪（1995—），男，广西钦州人，硕士研究生，主要研究方向：深度学习、图像处理。
基金资助:
国家自然科学基金资助项目(61561008);广西自然科学基金资助项目(2018GXNSFAA294019)

Abstract

Abstract:

To address the issues of blurred contours and lost details of portrait image with motion blur after restoration， a moving portrait deblurring method based on multi-level jump residual group Generation Adversarial Network （GAN） was proposed. Firstly， the residual block was improved to construct the multi-level jump residual group module， and the structure of PatchGAN was also improved to make GAN better combine with the image features of each layer. Secondly， the multi-loss fusion method was adopted to optimize the network to enhance the real texture of the reconstructed image. Finally， the end-to-end mode was used to perform blind deblurring on the motion blurred portrait image and output clear portrait image. Experimental results on CelebA dataset show that the Peak Signal-to-Noise Ratio （PSNR） and Structural SIMilarity （SSIM） of the proposed method are at least 0.46 dB and 0.05 higher than those of the Convolutional Neural Network （CNN）-based methods such as DeblurGAN （Deblur GAN）， Scale-Recurrent Network （SRN） and MSRAN （Multi-Scale Recurrent Attention Network）. At the same time， the proposed method has fewer model parameters， faster restoration， and more texture details in the restored portrait images.

Key words: image debluring, blind deblurring, Generative Adversarial Network (GAN), multi-level jump residual group, multi-loss fusion

摘要：

为解决复原后的运动模糊人像图像的轮廓模糊、细节丢失等问题，提出了基于多级跳跃残差组生成对抗网络（GAN）的运动人像去模糊方法。首先，改进残差块以构造多级跳跃残差组模块，并改进PatchGAN的结构以使GAN能够更好地结合各层的图像特征；其次，使用多损失融合的方法优化网络，从而增强重建后图像的真实纹理；最后，采用端到端的模式将运动模糊的人像图像进行盲去模糊操作，并输出清晰的人像图像。在CelebA数据集上的实验结果表明，相较于DeblurGAN（Deblur GAN）、尺度循环网络（SRN）和MSRAN（Multi-Scale Recurrent Attention Network）等基于卷积神经网络（CNN）的方法，所提方法的峰值信噪比（PSNR）和结构相似度（SSIM）分别至少提高了0.46 dB和0.05；同时，所提方法的模型参数更少，修复速度更快，且复原后的人像图像具有更多的纹理细节。

关键词: 图像去模糊, 盲去模糊, 生成对抗网络, 多级跳跃残差组, 多损失融合

CLC Number:

TP391.41

Jiaqi JI, Zhenkun LU, Fupeng XIONG, Tian ZHANG, Hao YANG. Moving portrait debluring network based on multi-level jump residual group[J]. Journal of Computer Applications, 2023, 43(10): 3244-3250.

纪佳奇, 卢振坤, 熊福棚, 张甜, 杨豪. 基于多级跳跃残差组的运动人像去模糊网络[J]. 《计算机应用》唯一官方网站, 2023, 43(10): 3244-3250.

Figures/Tables 16

Fig. 1 Flow of the proposed method

Fig. 2 Structure of generative network

Tab. 1 Model parameters of discriminative network

层	大小	卷积核大小	Stride	激活函数
Conv1	256×256×64	3×3	1	LeakyReLU
Conv2	128×128×64	3×3	2	LeakyReLU
Conv3	128×128×128	3×3	1	LeakyReLU
Conv4	64×64×128	3×3	2	LeakyReLU
Conv5	64×64×256	3×3	1	LeakyReLU
Conv6	32×32×256	3×3	2	LeakyReLU
Conv7	32×32×512	3×3	1	LeakyReLU
Conv8	16×16×512	3×3	2	LeakyReLU
Dense1	16×16×1 024			LeakyReLU
Dense2	16×16×1			Sigmoid

Fig. 3 Structure of multi-level jump residual group

Fig. 4 Comparison of feature maps of different VGG19 layers before and after activation

Tab. 2 Comparison of deblurring effects of different convolutional layers in VGG19

卷积层	PSNR/dB	SSIM	卷积层	PSNR/dB	SSIM
$Φ 3,1$	28.02	0.92	$Φ 3,3$	29.21	0.94
$Φ 3,2$	28.24	0.91	$Φ 4,3$	27.79	0.90

Tab. 2 Comparison of deblurring effects of different convolutional layers in VGG19

卷积层	PSNR/dB	SSIM	卷积层	PSNR/dB	SSIM
$Φ 3,1$	28.02	0.92	$Φ 3,3$	29.21	0.94
$Φ 3,2$	28.24	0.91	$Φ 4,3$	27.79	0.90

Tab. 3 Ablation experimental results of different loss functions

损失函数	PSNR/dB	SSIM
L_adv	28.16	0.91
L_P	28.03	0.91
L_adv+L_P	29.21	0.94

Fig. 5 Deblurring results of different loss functions

Tab. 4 Performance analysis of different numbers of residual blocks

残差块数	PSNR/dB	SSIM
1	26.85	0.75
2	27.23	0.82
3	29.21	0.94

Fig. 6 Deblurring results with different number of residual blocks

Tab. 5 Quantitative comparison results of the performance of different algorithms in removing Gaussian blur in portraits

方法	PSNR/dB	SSIM	生成网络参数量
Multiple-scale CNN	29.12	0.92	―
DeblurGAN	27.90	0.87	11 399 171
SRN	30.13	0.93	―
MSRAN	31.62	0.94	―
DMPHN	31.61	0.93	―
本文方法	32.19	0.96	2 926 659

Tab. 6 Quantitative comparison results of the performance of different algorithms in removing portrait motion blur

方法	PSNR/dB	SSIM	复原时间/s
Multiple-scale CNN	26.36	0.84	1.562
DeblurGAN	27.26	0.85	0.097
SRN	26.79	0.84	0.831
MSRAN	28.75	0.89	0.614
DMPHN	27.86	0.87	0.034
本文方法	29.21	0.94	0.089

Fig. 7 Comparison of deblurring results of different methods on Gaussian blurred potrait image in CelebA dataset

Fig. 8 Comparison of deblurring results of different methods on motion blurred portrait image in CelebA dataset

Tab. 7 Experimental results comparison of deblurring effects by different methos on GoPro dataset

方法	PSNR/dB	SSIM	方法	PSNR/dB	SSIM
Multi-scale CNN	28.28	0.89	MSRAN	29.45	0.92
DeblurGAN	28.25	0.87	DMPHN	30.51	0.92
SRN	29.33	0.90	本文方法	30.48	0.94

Fig. 9 Comparison of deblurring results of different methods on motion blurred images in GoPro dataset

References 35

1	WHITE R L. Image restoration using the damped Richardson-Lucy method［C］// Proceedings of the SPIE 2198， Instrumentation in Astronomy Ⅷ. Bellingham， WA： SPIE， 1994： 1342-1348. 10.1117/12.176819
2	FERGUS R， SINGH B， HERTZMANN A， et al. Removing camera shake from a single photograph［M］// ACM SIGGRAPH 2006 Papers. New York： ACM， 2006： 787-794. 10.1145/1179352.1141956
3	PAN J， SUN D， PFISTER H， et al. Blind image deblurring using dark channel prior［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 1628-1636. 10.1109/cvpr.2016.180
4	LI L， PAN J， LAI W S， et al. Learning a discriminative prior for blind image deblurring［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 6616-6625. 10.1109/cvpr.2018.00692
5	LEVIN A. Blind motion deblurring using image statistics［C］// Proceedings of the 19th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2006： 841-848. 10.7551/mitpress/7503.003.0110
6	YUAN X， ZHU J， LI X. Blur kernel estimation by structure sparse prior［J］. Applied Sciences， 2020， 10（2）： No.657. 10.3390/app10020657
7	LIU S， FENG Y， ZHANG S， et al. L₀ sparse regularization-based image blind deblurring approach for solid waste image restoration［J］. IEEE Transactions on Industrial Electronics， 2019， 66（12）： 9837-9845. 10.1109/tie.2019.2892681
8	XU L， REN J S J， LIU C， et al. Deep convolutional neural network for image deconvolution［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems — Volume 1. Cambridge： MIT Press， 2014： 1790-1798.
9	SUN J， CAO W， XU Z， et al. Learning a convolutional neural network for non-uniform motion blur removal［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 769-777. 10.1109/cvpr.2015.7298677
10	GONG D， YANG J， LIU L， et al. From motion blur to motion flow： a deep learning solution for removing heterogeneous motion blur［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 3806-3815. 10.1109/cvpr.2017.405
11	NAH S， KIM T H， LEE K M. Deep multi-scale convolutional neural network for dynamic scene deblurring［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 257-265. 10.1109/cvpr.2017.35
12	KUPYN O， BUDZAN V， MYKHAILYCH M， et al. DeblurGAN： blind motion deblurring using conditional adversarial networks［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8183-8192. 10.1109/cvpr.2018.00854
13	TAO X， GAO H， SHEN X， et al. Scale-recurrent network for deep image deblurring［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8174-8182. 10.1109/cvpr.2018.00853
14	WU Y， HONG C， ZHANG X， et al. Stack-based scale-recurrent network for face image deblurring［J］. Neural Processing Letters， 2021， 53（6）： 4419-4436. 10.1007/s11063-021-10604-9
15	陈贵强，何军，罗顺茺. 基于改进CycleGAN的视频监控人脸超分辨率恢复算法［J］. 计算机应用研究， 2021， 38（10）：3172-3176. 10.19734/j.issn.1001-3695.2020.11.0434
	CHEN G Q， HE J， LUO S C. Improved video surveillance face super-resolution recovery algorithm based on CycleGAN［J］. Application Research of Computers， 2021， 38（10）：3172-3176. 10.19734/j.issn.1001-3695.2020.11.0434
16	欧阳宁，邓超阳，林乐平. 基于自适应残差的运动图像去模糊［J］. 计算机工程与设计， 2021， 42（6）：1684-1690. 10.16208/j.issn1000-7024.2021.06.024
	OUYANG N， DENG C Y， LIN L P. Motion image deblurring based on adaptive residuals［J］. Computer Engineering and Design， 2021， 42（6）： 1684-1690. 10.16208/j.issn1000-7024.2021.06.024
17	魏海云，郑茜颖，俞金玲. 基于多尺度网络的运动模糊图像复原算法［J］. 计算机应用， 2022， 42（9）： 2838-2844. 10.11772/j.issn.1001-9081.2021081433
	WEI H Y， ZHENG Q Y， YU J L. Motion blurred image restoration algorithm based on multi-scale network［J］. Journal of Computer Applications， 2022， 42（9）： 2838-2844. 10.11772/j.issn.1001-9081.2021081433
18	刘万军，张正寰，曲海成. 融合DenseNet的多尺度图像去模糊模型［J］. 计算机工程与应用， 2021， 57（24）： 219-226.
	LIU W J， ZHANG Z H， QU H C. Multi-scale image deblurring model with DenseNet［J］. Computer Engineering and Applications， 2021， 57（24）：219-226.
19	王向军，欧阳文森. 多尺度循环注意力网络运动模糊图像复原方法［J］. 红外与激光工程， 2022， 51（6）： No.20210605.
	WANG X J， OUYANG W S. Multi-scale recurrent attention network for image motion deblurring［J］. Infrared and Laser Engineering， 2022， 51（6）： No.20210605.
20	ZHANG H， DAI Y， LI H， et al. Deep stacked hierarchical multi-patch network for image deblurring［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 5971-5979. 10.1109/cvpr.2019.00613
21	李福海，蒋慕蓉，杨磊，等. 基于生成对抗网络的梯度引导太阳斑点图像去模糊方法［J］. 计算机应用， 2021， 41（11）：3345-3352. 10.11772/j.issn.1001-9081.2020121898
	LI F H， JIANG M R， YANG L， et al. Solar speckle image deblurring method with gradient guidance based on generative adversarial network［J］. Journal of Computer Applications， 2021， 41（11）： 3345-3352. 10.11772/j.issn.1001-9081.2020121898
22	虞志军，王国栋，张镡月. 基于增强多尺度特征网络的图像去模糊［J］. 激光与光电子学进展， 2022， 59（22）： No.2215007. 10.3788/LOP202259.2215007
	YU Z J， WANG G D， ZHANG X Y. Image deblurring based on enhanced multi-scale feature network［J］. Laser and Optoelectronics Progress， 2022， 59（22）： No.2215007. 10.3788/LOP202259.2215007
23	王晨卿，荆涛，刘云鹏，等. 基于多尺度条件生成对抗网络的图像去模糊［J］. 计算机工程与设计， 2022， 43（4）：1074-1082.
	WANG C Q， JIN T， LIU Y P， et al. Image deblurring based on multi-scale conditional generative adversarial network［J］. Computer Engineering and Design， 2022， 43（4）：1074-1082.
24	魏丙财，张立晔，孟晓亮，等. 基于深度残差生成对抗网络的运动图像去模糊［J］. 液晶与显示， 2021， 36（12）： 1693-1701. 10.37188/cjlcd.2021-0120
	WEI B C， ZHANG L Y， MENG X L， et al. Motion image deblurring based on depth residual generative adversarial network［J］. Chinese Journal of Liquid Crystals and Displays， 2021， 36（12）： 1693-1701. 10.37188/cjlcd.2021-0120
25	LI C， WAND M. Precomputed real-time texture synthesis with Markovian generative adversarial networks［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9907. Cham： Springer， 2016： 702-716.
26	SCHULER C J， HIRSCH M， HARMELING S， et al. Learning to deblur［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2016， 38（7）： 1439-1451. 10.1109/tpami.2015.2481418
27	GOODFELLOW I J， POUGET-ABADIE J， MIRZA M， et al. Generative adversarial nets［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems — Volume 2. Cambridge： MIT Press， 2014： 2672-2680.
28	LEDIG C， THEIS L， HUSZÁR F， et al. Photo-realistic single image super-resolution using a generative adversarial network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 105-114. 10.1109/cvpr.2017.19
29	WANG X， YU K， WU S， et al. ESRGAN： enhanced super-resolution generative adversarial networks［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11133. Cham： Springer， 2019： 63-79.
30	HE K， ZHANG X， REN S， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
31	JOHNSON J， ALAHI A， LI F F. Perceptual losses for real-time style transfer and super-resolution［C］// Proceedings of the 2016 European Conference on Computer Vision. Berlin： Springer， 2016： 694-711. 10.1007/978-3-319-46475-6_43
32	ISOLA P， ZHU J Y， ZHOU T， et al. Image-to-image translation with conditional adversarial networks［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 5967-5976. 10.1109/cvpr.2017.632
33	DENG J， DONG W， SOCHER R， et al. ImageNet： a large-scale hierarchical image database［C］// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2009： 248-255. 10.1109/cvpr.2009.5206848
34	LIU Z， LUO P， WANG X， et al. Large-scale CelebFaces attributes （CelebA） dataset［DS/OL］. ［2022-06-14］..
35	KINGMA D P， BA J L. Adam： a method for stochastic optimization［EB/OL］. （2017-01-30）［2022-06-14］..

[1]	Anyang LIU, Huaici ZHAO, Wenlong CAI, Zechao XU, Ruideng XIE. Adaptive image deblurring generative adversarial network algorithm based on active discrimination mechanism [J]. Journal of Computer Applications, 2023, 43(7): 2288-2294.
[2]	Shaoquan CHEN, Jianping CAI, Lan SUN. Differential privacy generative adversarial network algorithm with dynamic gradient threshold clipping [J]. Journal of Computer Applications, 2023, 43(7): 2065-2072.
[3]	Xin JIN, Yangchuan LIU, Yechen ZHU, Zijian ZHANG, Xin GAO. Sinogram inpainting for sparse-view cone-beam computed tomography image reconstruction based on residual encoder-decoder generative adversarial network [J]. Journal of Computer Applications, 2023, 43(6): 1950-1957.
[4]	Jinwen GUO, Xinghua MA, Gongning LUO, Wei WANG, Yang CAO, Kuanquan WANG. Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer [J]. Journal of Computer Applications, 2023, 43(5): 1596-1605.
[5]	Jiagao WU, Shiwen ZHANG, Yudong JIANG, Linfeng LIU. Social-interaction GAN for pedestrian trajectory prediction based on state-refinement long short-term memory and attention mechanism [J]. Journal of Computer Applications, 2023, 43(5): 1565-1570.
[6]	Xiaoyu FAN, Suzhen LIN, Yanbo WANG, Feng LIU, Dawei LI. Reconstruction algorithm for highly undersampled magnetic resonance images based on residual graph convolutional neural network [J]. Journal of Computer Applications, 2023, 43(4): 1261-1268.
[7]	Hao WANG, Zicheng WANG, Chao ZHANG, Yunsheng MA. Generative adversarial network based data uncertainty quantification method [J]. Journal of Computer Applications, 2023, 43(4): 1094-1101.
[8]	Chunyong YIN, Liwen ZHOU. Unsupervised time series anomaly detection model based on re-encoding [J]. Journal of Computer Applications, 2023, 43(3): 804-811.
[9]	Lingling TAO, Bo LIU, Wenbo LI, Xiping HE. Controllable face editing algorithm with closed-form solution [J]. Journal of Computer Applications, 2023, 43(2): 601-607.
[10]	Gang CHEN, Yongwei LIAO, Zhenguo YANG, Wenying LIU. Image inpainting algorithm of multi-scale generative adversarial network based on multi-feature fusion [J]. Journal of Computer Applications, 2023, 43(2): 536-544.
[11]	Li’an ZHU, Hong ZHANG. Nonhomogeneous image dehazing based on dual-branch conditional generative adversarial network [J]. Journal of Computer Applications, 2023, 43(2): 567-574.
[12]	Ruoying WANG, Fan LYU, Liuqing ZHAO, Fuyuan HU. Floorplan generation algorithm integrating user requirements and boundary constraints [J]. Journal of Computer Applications, 2023, 43(2): 575-582.
[13]	HU Ziqi, XIE Kai, WEN Chang, LI Meiran, HE Jianbiao. Low dose CT image enhancement based on generative adversarial network [J]. Journal of Computer Applications, 2023, 43(1): 280-288.
[14]	Zanxia QIANG, Xianfu BAO. Residual attention deraining network based on convolutional long short-term memory [J]. Journal of Computer Applications, 2022, 42(9): 2858-2864.
[15]	Haiyun WEI, Qianying ZHENG, Jinling YU. Motion blurred image restoration algorithm based on multi-scale network [J]. Journal of Computer Applications, 2022, 42(9): 2838-2844.

Moving portrait debluring network based on multi-level jump residual group

基于多级跳跃残差组的运动人像去模糊网络

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 35

Related Articles 15

Recommended Articles

Metrics