Adaptive image deblurring generative adversarial network algorithm based on active discrimination mechanism

doi:10.11772/j.issn.1001-9081.2022060840

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (7): 2288-2294.DOI: 10.11772/j.issn.1001-9081.2022060840

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Adaptive image deblurring generative adversarial network algorithm based on active discrimination mechanism

Anyang LIU¹, Huaici ZHAO², Wenlong CAI¹(), Zechao XU², Ruideng XIE¹

^1.School of Electronics and Control Engineering，North China Institute of Aerospace Engineering，Langfang Hebei 065000，China
^2.Shenyang Institute of Automation，Chinese Academy of Sciences，Shenyang Liaoning 110169，China

Received:2022-06-10 Revised:2022-08-24 Accepted:2022-08-26 Online:2022-09-22 Published:2023-07-10
Contact: Wenlong CAI
About author:LIU Anyang， born in 1997， M. S. candidate. His research interests include artificial intelligence， image processing， pattern recognition.
ZHAO Huaici， born in 1974， Ph. D.， research fellow. His research interests include image processing， complex system modeling and simulation， communication and information processing.
CAI Wenlong， born in 1976， M. S.， associate professor. His research interests include intelligent control， weak signal processing.
XU Zechao， born in 1996， M. S. His research interests include image segmentation， deep learning.
XIE Ruideng， born in 1996， M. S. candidate. His research interests include control engineering， automation design in computer engineering.
Supported by:
Major Basic Research Project of National Equipment of China(51405-02A01);Key Research and Development Program of Hebei Province(20327216D)

基于主动判别机制的自适应生成对抗网络图像去模糊算法

刘安阳¹, 赵怀慈², 蔡文龙¹(), 许泽超², 解瑞灯¹

^1.北华航天工业学院电子与控制工程学院, 河北廊坊 065000
^2.中国科学院沈阳自动化研究所, 沈阳 110169

通讯作者: 蔡文龙
作者简介:刘安阳（1997—），男，山东济南人，硕士研究生，主要研究方向：人工智能、图像处理、模式识别；
赵怀慈（1974—），男，山东潍坊人，研究员，博士，主要研究方向：图像处理、复杂系统建模与仿真、通信与信息处理；
蔡文龙（1976—），男，河北衡水人，副教授，硕士，主要研究方向：智能控制、微弱信号处理；
许泽超（1996—），男，山东德州人，硕士，主要研究方向：图像分割、深度学习；
解瑞灯（1996—），男，山东菏泽人，硕士研究生，主要研究方向：控制工程、计算机工程自动化设计。
基金资助:
国家装备重大基础研究项目(51405-02A01);河北省重点研发计划项目(20327216D)

Abstract

Abstract:

Aiming at the problems that existing image deblurring algorithms suffer from diffusion and artifacts when dealing with edge loss and the use of full-frame deblurring in video processing does not meet real-time requirements， an Adaptive DeBlurring Generative Adversarial Network （ADBGAN）algorithm based on active discrimination mechanism was proposed. Firstly， an adaptive fuzzy discrimination mechanism was proposed， and an adaptive fuzzy processing network module was developed to make a priori judgment of fuzziness on the input image. When collecting the input， the blurring degree of the input image was judged in advance， and the input frame which was clear enough was eliminated to improve the running efficiency of the algorithm. Then， the incentive link of the attention mechanism was introduced in the process of fine feature extraction， so that weight normalization was carried out in the forward flow of feature extraction to improve the performance of the network to recover fine-grained features. Finally， the feature pyramid fine feature recovery structure was improved in the generator architecture， and a more lightweight feature fusion process was adopted to improve the running efficiency. In order to verify the effectiveness of the algorithm， detailed comparison experiments were conducted on the open source datasets GoPro and Kohler. Experimental results on GoPro dataset show that the visual fidelity of ADBGAN is 2.1 times that of Scale-Recurrent Network （SRN） algorithm， the Peak Signal-to-Noise Ratio （PSNR） of ADBGAN is improved by 0.762 dB compared with that of SRN algorithm， and ADBGAN has good image information recovery ability； in terms of video processing time，the actual processing time is reduced by 85.9% compared to SRN.The proposed algorithm can generate deblurred images with higher information quality efficiently.

Key words: image deblurring, image denoising, Generative Adversarial Network (GAN), adaptive discrimination, feature recovery

摘要：

针对现有图像去模糊算法在处理边缘丢失时出现弥散和伪影以及在视频处理中使用全帧去模糊方式导致不满足实时性需求的问题，提出一种基于主动判别机制的自适应生成对抗网络图像去模糊（ADBGAN）算法。首先，提出一种自适应模糊判别机制，开发了自适应模糊处理网络模块对输入图像进行模糊先验判断。在采集到输入时提前判断输入图像的模糊程度，从而剔除足够清晰的输入帧以提升算法运行效率。然后，在精细特征提取过程中引入注意力机制中的激励环节，从而在特征提取的流程中进行权重归一化来提升网络对精细特征的恢复能力。最后，在生成器架构中改进了特征金字塔精细特征恢复结构，并采用更轻量化的特征融合流程提高运行效率。为验证算法的有效性，在开源数据集GoPro和Kohler上进行了详细的对比实验。实验结果显示，在GoPro数据集中ADBGAN的视觉保真度是尺度循环网络（SRN）算法的2.1倍，并在峰值信噪比（PSNR）上较SRN算法提升了0.762 dB，具有良好的图像信息恢复能力；在视频数据处理时间上ADBGAN大幅超越了测试的所有算法，实测处理时间较SRN减少了85.9%。ADBGAN能够高效生成信息质量更高的去模糊图像。

关键词: 图像去模糊, 图像去噪, 生成对抗网络, 自适应判别, 特征恢复

CLC Number:

TP391.4

Anyang LIU, Huaici ZHAO, Wenlong CAI, Zechao XU, Ruideng XIE. Adaptive image deblurring generative adversarial network algorithm based on active discrimination mechanism[J]. Journal of Computer Applications, 2023, 43(7): 2288-2294.

刘安阳, 赵怀慈, 蔡文龙, 许泽超, 解瑞灯. 基于主动判别机制的自适应生成对抗网络图像去模糊算法[J]. 《计算机应用》唯一官方网站, 2023, 43(7): 2288-2294.

Figures/Tables 13

Fig. 1 GAN fuzzy image recovery idea

Fig. 2 Channel attention block

Fig. 3 Residual attention block

Fig. 4 Adaptive fuzzy judgment flow

Fig. 5 Overall framework of the proposed algorithm

Fig. 6 Generator network structure

Fig. 7 Discriminator network structure

Fig. 8 Subjective results on test images

Tab. 1 Experimental results of objective comparison

算法	GoPro性能测试				Kohler性能测试
算法	PSNR/dB	SSIM	VIF	AG	PSNR/dB	SSIM	VIF	AG
Nah算法^［18］	28.834	0.851	0.81	2.84	26.428	0.731	0.73	2.57
DeblurGAN算法^［5］	28.707	0.885	0.77	3.56	26.363	0.752	0.62	3.14
DeblurGANv2算法^［20］	29.061	0.889	0.85	4.03	27.272	0.788	0.73	3.63
SRN算法^［4］	29.115	0.883	0.42	4.51	27.554	0.794	0.34	4.32
MT-RNN算法^［19］	29.633	0.905	0.85	3.97	28.476	0.822	0.75	3.76
本文算法	29.877	0.921	0.89	5.21	28.492	0.824	0.80	4.95

Tab. 2 Test results of average running time

算法	单幅图像处理时间	视频数据处理时间
Nah算法^［18］	3.02	1 817
DeblurGAN算法^［5］	0.91	547
SRN算法^［4］	1.24	765
MT-RNN算法^［19］	0.32	192
本文算法	0.54	108

Tab. 3 Ablation experimental results

算法结构	PSNR/dB	SSIM	AG
Base	27.981	0.822	3.02
Base+FPN	28.522	0.852	3.11
Base+FPN+RAB	29.523	0.877	3.31
Base+FPN+RAB+Loss	29.401	0.904	4.63
Base+FPN+RAB+Double Discriminator	29.877	0.921	5.21

Tab. 4 Experimental results of object detection

Recall和AP指标变化/%	样本数		对照评分 $S o r i g i n$
Recall和AP指标变化/%	GoPro数据集	自制数据集	对照评分 $S o r i g i n$
降低（0，5）	44	21	0.864
提升（0，5）	852	224	0.722
提升［5，10）	755	235	0.632
提升10及以上	349	120	0.411

Tab. 4 Experimental results of object detection

Recall和AP指标变化/%	样本数		对照评分 $S o r i g i n$
Recall和AP指标变化/%	GoPro数据集	自制数据集	对照评分 $S o r i g i n$
降低（0，5）	44	21	0.864
提升（0，5）	852	224	0.722
提升［5，10）	755	235	0.632
提升10及以上	349	120	0.411

Fig. 9 Test images of object detection experiments

References 20

1	MITRA S K. 数字信号处理——基于计算机的方法［M］. 余翔宇，译. 4版. 北京：电子工业出版社， 2012：48-52.
	MITRA S K. Digital Signal Processing： A Computer-Based Approach［M］. YU X Y， translated. 4th ed. Beijing： Publishing House of Electronics Industry， 2012：48-52.
2	FERGUS R， SINGH B， HERTZMAN A. Removing camera shake from a single photograph［J］. ACM Transactions on Graphics， 2006， 25（3）： 787-794. 10.1145/1141911.1141956
3	LI X， ZHENG S C， JIA J Y. Unnatural L₀ sparse representation for natural image deblurring ［C］// Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2013： 1107-1114. 10.1109/cvpr.2013.147
4	TAO X， GAO H Y， WANG X Y， et al. Scale-recurrent network for deep image deblurring ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8174-8182. 10.1109/cvpr.2018.00853
5	KUPYN O， BUDZAN V， MYKHAILYCH M， et al. DeblurGAN： blind motion deblurring using conditional adversarial networks ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8183-8192. 10.1109/cvpr.2018.00854
6	LI J， JIA J J， XU D L. Unsupervised representation learning of image-based plant disease with deep convolutional generative adversarial networks ［C］// Proceedings of the 37th Chinese Control Conference. Piscataway： IEEE， 2018： 9159-9163. 10.23919/chicc.2018.8482813
7	吴迪，赵洪田，郑世宝.密集连接卷积网络图像去模糊［J］.中国图象图形学报， 2020， 25（5）： 890-899. 10.11834/jig.190400
	WU D， ZHAO H T， ZHENG S B. Motion deblurring method based on DenseNets［J］. Journal of Image and Graphics， 2020， 25（5）： 890-899. 10.11834/jig.190400
8	金燕，黄梦佳，姜智伟.基于聚集残差生成对抗网络的图像去模糊［J］.计算机辅助设计与图形学学报， 2022， 34（1）： 84-93. 10.3724/sp.j.1089.2022.18839
	JIN Y， HUANG M J， JIANG Z W. Image deblurring based on aggregate residual adversary networks［J］. Journal of Computer-Aided Design and Graphics， 2022， 34（1）： 84-93. 10.3724/sp.j.1089.2022.18839
9	HU J， SHEN L， ALBANIE S， et al. Squeeze-and-excitation networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017， 42（8）： 2011-2023.
10	HE K M， ZHENG X Y， REN S Q， et al. Deep residual learning for image recognition ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
11	LIN T Y， DOLLÁR P， GIESHICK R， et al. Feature pyramid networks for object detection ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 936-944. 10.1109/cvpr.2017.106
12	LI C， WAND M. Precomputed real-time texture synthesis with Markovian generative adversarial networks ［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9907. Cham： Springer， 2016： 702-716.
13	MAO X， LI Q， XIE H， et al. Least squares generative adversarial networks ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2813-2821. 10.1109/iccv.2017.304
14	JOHNSON J， ALAHI A， LI F F. Perceptual losses for real-time style transfer and super-resolution ［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9906. Cham： Springer， 2016： 694-711.
15	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［EB/OL］. （2015-04-10）［2022-06-01］. .
16	SUN J， CAO W F， XU Z B， et al. Learning a convolutional neural network for non-uniform motion blur removal ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 769-777. 10.1109/cvpr.2015.7298677
17	ZHANG K H， LUO W H， ZHONG Y R， et al. Deblurring by realistic blurring ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 2734-2743. 10.1109/cvpr42600.2020.00281
18	NAH S， KIM T H， LEE K M. Deep multi-scale convolutional neural network for dynamic scene deblurring ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 257-265. 10.1109/cvpr.2017.35
19	PARK D， KANG D U， KIM J， et al. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training ［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12351. Cham： Springer， 2020： 327-343.
20	KUPYN O， MARTYNIUK T， WU J R， et al. DeblurGAN-v2： deblurring （orders-of-magnitude） faster and better ［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 8877-8886. 10.1109/iccv.2019.00897

[1]	Yuwei DING, Hongbo SHI, Jie LI, Min LIANG. Image denoising network based on local and global feature decoupling [J]. Journal of Computer Applications, 2024, 44(8): 2571-2579.
[2]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[3]	Haoran WANG, Dan YU, Yuli YANG, Yao MA, Yongle CHEN. Domain transfer intrusion detection method for unknown attacks on industrial control systems [J]. Journal of Computer Applications, 2024, 44(4): 1158-1165.
[4]	Yi ZHENG, Cunyi LIAO, Tianqian ZHANG, Ji WANG, Shouyin LIU. Image denoising-based cell-level RSRP estimation method for urban areas [J]. Journal of Computer Applications, 2024, 44(3): 855-862.
[5]	Ruifeng HOU, Pengcheng ZHANG, Liyuan ZHANG, Zhiguo GUI, Yi LIU, Haowen ZHANG, Shubin WANG. Iterative denoising network based on total variation regular term expansion [J]. Journal of Computer Applications, 2024, 44(3): 916-921.
[6]	Sunjie YU, Hui ZENG, Shiyu XIONG, Hongzhou SHI. Incentive mechanism for federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(2): 344-352.
[7]	Pengbo WANG, Wuyang SHAN, Jun LI, Mao TIAN, Deng ZOU, Zhanfeng FAN. Robust splicing forensic algorithm against high-intensity salt-and-pepper noise [J]. Journal of Computer Applications, 2024, 44(10): 3177-3184.
[8]	Hui ZHOU, Yuling CHEN, Xuewei WANG, Yangwen ZHANG, Jianjiang HE. Deep shadow defense scheme of federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(1): 223-232.
[9]	Shaoquan CHEN, Jianping CAI, Lan SUN. Differential privacy generative adversarial network algorithm with dynamic gradient threshold clipping [J]. Journal of Computer Applications, 2023, 43(7): 2065-2072.
[10]	Xin JIN, Yangchuan LIU, Yechen ZHU, Zijian ZHANG, Xin GAO. Sinogram inpainting for sparse-view cone-beam computed tomography image reconstruction based on residual encoder-decoder generative adversarial network [J]. Journal of Computer Applications, 2023, 43(6): 1950-1957.
[11]	Jiagao WU, Shiwen ZHANG, Yudong JIANG, Linfeng LIU. Social-interaction GAN for pedestrian trajectory prediction based on state-refinement long short-term memory and attention mechanism [J]. Journal of Computer Applications, 2023, 43(5): 1565-1570.
[12]	Jinwen GUO, Xinghua MA, Gongning LUO, Wei WANG, Yang CAO, Kuanquan WANG. Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer [J]. Journal of Computer Applications, 2023, 43(5): 1596-1605.
[13]	Hao WANG, Zicheng WANG, Chao ZHANG, Yunsheng MA. Generative adversarial network based data uncertainty quantification method [J]. Journal of Computer Applications, 2023, 43(4): 1094-1101.
[14]	Xiaoyu FAN, Suzhen LIN, Yanbo WANG, Feng LIU, Dawei LI. Reconstruction algorithm for highly undersampled magnetic resonance images based on residual graph convolutional neural network [J]. Journal of Computer Applications, 2023, 43(4): 1261-1268.
[15]	Chunyong YIN, Liwen ZHOU. Unsupervised time series anomaly detection model based on re-encoding [J]. Journal of Computer Applications, 2023, 43(3): 804-811.

Adaptive image deblurring generative adversarial network algorithm based on active discrimination mechanism

基于主动判别机制的自适应生成对抗网络图像去模糊算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 20

Related Articles 15

Recommended Articles

Metrics