Solar speckle image deblurring method with gradient guidance based on generative adversarial network

doi:10.11772/j.issn.1001-9081.2020121898

Journal of Computer Applications ›› 2021, Vol. 41 ›› Issue (11): 3345-3352.DOI: 10.11772/j.issn.1001-9081.2020121898

• Multimedia computing and computer simulation • Previous Articles Next Articles

Solar speckle image deblurring method with gradient guidance based on generative adversarial network

Fuhai LI¹, Murong JIANG¹(), Lei YANG², Junyi CHEN²

^1.School of Information Science and Engineering，Yunnan University，Kunming Yunnan 650500，China
^2.Yunnan Observatories，Chinese Academy of Sciences，Kunming Yunnan 650216，China

Received:2020-12-04 Revised:2021-05-13 Accepted:2021-08-03 Online:2021-05-13 Published:2021-11-10
Contact: Murong JIANG
About author:LI Fuhai，born in 1994，M. S. candidate. His research interests include deep learning，image reconstruction
JIANG Murong，born in 1963，Ph. D.，professor. Her research interests include image restoration，image segmentation
YANG Lei，born in 1980，M. S.，senior engineer. His research interests include astronomical image reconstruction
CHEN Junyi，born in 1979，M. S.，senior engineer. His research interests include high performance computing.
Supported by:
the National Natural Science Foundation of China(11773073);the University Science and Technology Innovation Team Support Project of Yunnan Province （IRTSTYN）

基于生成对抗网络的梯度引导太阳斑点图像去模糊方法

李福海¹, 蒋慕蓉¹(), 杨磊², 谌俊毅²

^1.云南大学信息学院，昆明 650500
^2.中国科学院云南天文台，昆明 650216

通讯作者: 蒋慕蓉
作者简介:李福海（1994—），男，重庆人，硕士研究生，主要研究方向：深度学习、图像重建
蒋慕蓉（1963—），女，湖南邵阳人，教授，博士，主要研究方向：图像复原、图像分割
杨磊（1980—），男，云南大理人，高级工程师，硕士，主要研究方向：天文图像重建
谌俊毅（1979—），男，湖南益阳人，高级工程师，硕士，主要研究方向：高性能计算。
基金资助:
国家自然科学基金资助项目(11773073);云南省高校科技创新团队支持项目（IRTSTYN）

Abstract

Abstract:

With the existing deep learning algorithms， it is difficult to restore the highly blurred solar speckle images taken by Yunnan Observatories， and it is difficult to reconstruct the high-frequency information of images. In order to solve the problems， a deblurring method for restoring the solar speckle images and recovering the high-frequency information of images based on Generative Adversarial Network （GAN） and gradient information was proposed. The proposed method was consisted of one generator and two discriminators. Firstly， the image multi-scale features were obtained by the generator with the Feature Pyramid Network （FPN） framework， and these features were input into the gradient branch hierarchically to capture the smaller details in the form of gradient map， and the solar speckle image with high-frequency information was reconstructed by combining the gradient branch results and the FPN results. Then， based on the conventional adversarial discriminator， another discriminator was added to ensure the gradient map generated by the gradient branch more realistic. Finally， a joint training loss including pixel content loss， perceptual loss and adversarial loss was introduced to guide the model to perform high-resolution reconstruction of solar speckle images. Experimental results show that， compared with the existing deep learning deblurring method， the proposed method with image preprocessing has stronger ability to recover the high-frequency information， and significantly improves the Peak Signal-to-Noise Ratio （PSNR） and Structural SIMilarity （SSIM） indicators， reaching 27.801 0 dB and 0.851 0 respectively. The proposed method can meet the needs for high-resolution reconstruction of solar observation images.

Key words: deblurring, Generative Adversarial Network (GAN), gradient guidance, local detail, solar speckle

摘要：

针对云南天文台拍摄的高度模糊的太阳斑点图像采用现有深度学习算法恢复难度大、高频信息难以重建等问题，提出了一种基于生成对抗网络（GAN）与梯度信息联合的去模糊方法来重建太阳斑点图，并很好地恢复出图像的高频信息。该方法由一个生成器与两个鉴别器构成：首先，生成器采用特征金字塔网络（FPN）框架来获取图像多尺度特征，再将这些特征分层次输入梯度分支以梯度图的形式捕获更小的局部特征；然后，联合梯度分支结果与FPN结果共同重建出具有高频信息的太阳斑点图像；其次，在常规对抗鉴别器的基础上，增加了一个鉴别器用于保证由梯度分支产生的梯度图更加真实；最后，引入一个包括像素内容损失、感知损失和对抗损失的联合训练损失来引导模型进行太阳斑点图像高分辨率重建。实验结果表明，进行图像预处理后的所提方法与现有的深度学习去模糊方法相比，高频信息恢复能力更强，峰值信噪比（PSNR）和结构相似性（SSIM）指标均有显著提高，分别达到27.801 0 dB与0.851 0，能够满足太阳观测图像高分辨率重建的需要。

关键词: 去模糊, 生成对抗网络, 梯度引导, 局部细节, 太阳斑点

CLC Number:

TP391.41

Fuhai LI, Murong JIANG, Lei YANG, Junyi CHEN. Solar speckle image deblurring method with gradient guidance based on generative adversarial network[J]. Journal of Computer Applications, 2021, 41(11): 3345-3352.

李福海, 蒋慕蓉, 杨磊, 谌俊毅. 基于生成对抗网络的梯度引导太阳斑点图像去模糊方法[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3345-3352.

Figures/Tables 16

Fig. 1 Network structure of proposed method

Fig. 2 Schematic diagram of generator structure

Fig. 3 Schematic diagram of discriminator structure

Fig. 4 Part of dataset

Tab. 1 Evaluation results of different methods on test set

方法	PSNR/dB	SSIM
SPSR	22.965 5	0.638 3
DRN-L	24.274 2	0.652 4
DeblurGANv2-Mobile	24.476 4	0.652 6
DeblurGANv2-Inception	24.064 0	0.674 1
Cycle-GAN¹	25.123 7	0.744 2
Cycle-GAN²	23.620 1	0.694 7
本文方法	27.603 9	0.833 4
本文方法+预处理	27.801 0	0.851 0

Fig. 5 Comparison of reconstruction results of different methods

Fig. 6 Detail comparison of reconstruction results of different methods

Fig. 7 Comparison of gradient branch information acquisition of different methods

Tab. 2 Performance comparison of different preprocessing methods

方法	SSIM	模型尺寸/MB	训练时间/h
无预处理	0.833 4	271.3	86.8
Mobile	0.851 0	291.5	103.8
Inception	0.853 2	510.7	143.2

Fig. 8 SSIM comparison of different preprocessing methods in different training time

Tab. 3 Impact of different gradient branch structures

方法	PSNR/dB	SSIM
FPN编码器分支	25.261 6	0.775 1
去掉分支	24.840 6	0.727 5

Fig. 9 Results comparison of different gradient branch construction methods

Tab. 4 Evaluation results of different methods on DVD dataset

方法	PSNR/dB	SSIM
DeblurGANv2-Mobile	28.54	0.929 4
DeblurGANv2-Inception	28.85	0.932 7
本文方法	28.69	0.931 2
本文方法+预处理	28.76	0.921 8

Tab. 5 Evaluation results of different methods on GOPRO dataset

方法	PSNR/dB	SSIM
DL	24.64	0.841 9
DeepDeblur	29.08	0.913 5
SRN	30.26	0.934 2
DeblurGANv2-Mobile	28.17	0.925 4
DeblurGANv2-Inception	29.55	0.934 4
本文方法	28.85	0.921 2
本文方法+预处理	28.92	0.923 2

Fig. 10 Reconstruction result comparison of different methods on public datasets （Image 1）

Fig. 11 Reconstruction result comparison of different methods on public datasets （Image 2）

References 26

1	张兰强.太阳高分辨率力成像多层共轭自适应光学技术研究［D］.北京：中国科学院大学，2014：20-28.
	ZHANG L Q. Research on multi-conjugate adaptive optics technology for solar high-resolution force imaging ［D］. Beijing： University of Chinese Academy of Sciences， 2014： 20-28.
2	刘忠.天文图象高分辨重建及空域性质研究［D］.北京：中国科学院大学，2003：25-39.
	LIU Z. Research on high-resolution image reconstruction and spatial characteristics of astronomical images ［D］. Beijing： University of Chinese Academy of Sciences， 2003： 25-39.
3	邱耀辉，刘忠，卢汝为，等.天文图像空域重建新方法：迭代位移叠加法［J］.光学学报，2001，21（2）：186-191. 10.3321/j.issn:0253-2239.2001.02.015
	QIU Y H， LIU Z， LU R W， et al. A new method for astronomical image reconstruction in spatial region： iterative shift-and-add technique ［J］. Acta Optica Sinica， 2001， 21（2）： 186-191. 10.3321/j.issn:0253-2239.2001.02.015
4	向永源.太阳高分辨高速重建算法的研究［D］.北京：中国科学院大学，2016：19-22.
	XIANG Y Y. Research on high speed high resolution solar reconstruction algorithm ［D］. Beijing： University of Chinese Academy of Sciences， 2016： 19-22.
5	LIN T Y， DOLLÁR P， GIRSHICK R， et al. Feature pyramid networks for object detection ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 936-944. 10.1109/cvpr.2017.106
6	MA C， RAO Y M， CHENG Y， et al. Structure-preserving super resolution with gradient guidance ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 7766-7775. 10.1109/cvpr42600.2020.00779
7	JOHNSON J， ALAHI A， LI F F. Perceptual losses for real-time style transfer and super-resolution ［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS9906. Cham： Springer， 2016： 1604-1613.
8	GU J J， LU H N， ZUO W M， et al. Blind super-resolution with iterative kernel correction ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 1604-1613. 10.1109/cvpr.2019.00170
9	GUO Y， CHEN J， WANG J D， et al. Closed-loop matters： dual regression networks for single image super-resolution ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 5406-5415. 10.1109/cvpr42600.2020.00545
10	KAUFMAN A， FATTAL R. Deblurring using analysis synthesis networks pair ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 5810-5819. 10.1109/cvpr42600.2020.00585
11	YANG F Z， YANG H， FU J L， et al. Learning texture transformer network for image super-resolution ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 5790-5799. 10.1109/cvpr42600.2020.00583
12	GOODFELLOW I J， POUGET-ABADIE J， MIRZA M， et al. Generative adversarial networks ［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2014： 2672-2680.
13	RADFORD A， METZ L， CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks ［EB/OL］. （2016-01-07）［2020-08-27］. .
14	ZHU J Y， PARK T， ISOLA P， et al. Unpaired image-to-image translation using cycle-consistent adversarial networks ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2242-2251. 10.1109/iccv.2017.244
15	ENGIN D， GENC A， EKENEL H K. Cycle-dehaze： enhanced CycleGAN for single image dehazing ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2018： 938-946. 10.1109/cvprw.2018.00127
16	KUPYN O， MARTYNIUK T， WU J R， et al. DeblurGAN-v2： deblurring （orders-of-magnitude） faster and better ［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 8877-8886. 10.1109/iccv.2019.00897
17	JOLICOEUR-MARTINEAU A. The relativistic discriminator： a key element missing from standard GAN ［EB/OL］. （2018-09-10）［2020-08-27］. .
18	GULRAJANI I， AHMED F， ARJOVSKY M， et al. Improved training of Wasserstein GANs ［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2017： 5769-5779.
19	REN Y H， JIANG M R， FU Z X， et al. Reconstruction of single-frame solar speckle image with cycle consistency loss and perceptual loss ［C］// Proceedings of the 2019 6th International Conference on Information Science and Control Engineering. Piscataway： IEEE， 2019： 439-443. 10.1109/icisce48695.2019.00094
20	JIA P， HUANG Y， CAI B J， et al. Solar image restoration with the CycleGAN based on multi-fractal properties of texture features ［J］. The Astrophysical Journal Letters， 2019， 881（2）： Article No.L30. 10.3847/2041-8213/ab365f
21	WANG X T， YU K， WU S X， et al. ESRGAN： enhanced super resolution generative adversarial networks ［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS11133. Cham： Springer， 2018： 63-79.
22	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition ［EB/OL］. （2015-04-10）［2020-08-27］. . 10.5244/c.28.6
23	LEDIG C， THEIS L， HUSZÁR F， et al. Photo-realistic single image super-resolution using a generative adversarial network ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 105-114. 10.1109/cvpr.2017.19
24	SUN J， CAO W F， XU Z B， et al. Learning a convolutional neural network for non-uniform motion blur removal ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 769-777. 10.1109/cvpr.2015.7298677
25	NAH S， KIM T H， LEE K M. Deep multi-scale convolutional neural network for dynamic scene deblurring ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 257-265. 10.1109/cvpr.2017.35
26	TAO X， GAO H Y， SHEN X Y， et al. Scale-recurrent network for deep image deblurring ［C］// Proceedings of the 2018 IEEE/CVG Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8174-8182. 10.1109/cvpr.2018.00853

[1]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[2]	Haoran WANG, Dan YU, Yuli YANG, Yao MA, Yongle CHEN. Domain transfer intrusion detection method for unknown attacks on industrial control systems [J]. Journal of Computer Applications, 2024, 44(4): 1158-1165.
[3]	Sunjie YU, Hui ZENG, Shiyu XIONG, Hongzhou SHI. Incentive mechanism for federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(2): 344-352.
[4]	Hui ZHOU, Yuling CHEN, Xuewei WANG, Yangwen ZHANG, Jianjiang HE. Deep shadow defense scheme of federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(1): 223-232.
[5]	Anyang LIU, Huaici ZHAO, Wenlong CAI, Zechao XU, Ruideng XIE. Adaptive image deblurring generative adversarial network algorithm based on active discrimination mechanism [J]. Journal of Computer Applications, 2023, 43(7): 2288-2294.
[6]	Shaoquan CHEN, Jianping CAI, Lan SUN. Differential privacy generative adversarial network algorithm with dynamic gradient threshold clipping [J]. Journal of Computer Applications, 2023, 43(7): 2065-2072.
[7]	Xin JIN, Yangchuan LIU, Yechen ZHU, Zijian ZHANG, Xin GAO. Sinogram inpainting for sparse-view cone-beam computed tomography image reconstruction based on residual encoder-decoder generative adversarial network [J]. Journal of Computer Applications, 2023, 43(6): 1950-1957.
[8]	Jiagao WU, Shiwen ZHANG, Yudong JIANG, Linfeng LIU. Social-interaction GAN for pedestrian trajectory prediction based on state-refinement long short-term memory and attention mechanism [J]. Journal of Computer Applications, 2023, 43(5): 1565-1570.
[9]	Jinwen GUO, Xinghua MA, Gongning LUO, Wei WANG, Yang CAO, Kuanquan WANG. Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer [J]. Journal of Computer Applications, 2023, 43(5): 1596-1605.
[10]	Hao WANG, Zicheng WANG, Chao ZHANG, Yunsheng MA. Generative adversarial network based data uncertainty quantification method [J]. Journal of Computer Applications, 2023, 43(4): 1094-1101.
[11]	Xiaoyu FAN, Suzhen LIN, Yanbo WANG, Feng LIU, Dawei LI. Reconstruction algorithm for highly undersampled magnetic resonance images based on residual graph convolutional neural network [J]. Journal of Computer Applications, 2023, 43(4): 1261-1268.
[12]	Chunyong YIN, Liwen ZHOU. Unsupervised time series anomaly detection model based on re-encoding [J]. Journal of Computer Applications, 2023, 43(3): 804-811.
[13]	Li’an ZHU, Hong ZHANG. Nonhomogeneous image dehazing based on dual-branch conditional generative adversarial network [J]. Journal of Computer Applications, 2023, 43(2): 567-574.
[14]	Ruoying WANG, Fan LYU, Liuqing ZHAO, Fuyuan HU. Floorplan generation algorithm integrating user requirements and boundary constraints [J]. Journal of Computer Applications, 2023, 43(2): 575-582.
[15]	Lingling TAO, Bo LIU, Wenbo LI, Xiping HE. Controllable face editing algorithm with closed-form solution [J]. Journal of Computer Applications, 2023, 43(2): 601-607.

Solar speckle image deblurring method with gradient guidance based on generative adversarial network

基于生成对抗网络的梯度引导太阳斑点图像去模糊方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 26

Related Articles 15

Recommended Articles

Metrics