Image super-resolution reconstruction algorithm based on Laplacian pyramid generative adversarial network

doi:10.11772/j.issn.1001-9081.2020081299

Journal of Computer Applications ›› 2021, Vol. 41 ›› Issue (4): 1020-1026.DOI: 10.11772/j.issn.1001-9081.2020081299

Special Issue: CCF第35届中国计算机应用大会（CCF NCCA 2020）

• The 35 CCF National Conference of Computer Applications (CCF NCCA 2020) • Previous Articles Next Articles

Image super-resolution reconstruction algorithm based on Laplacian pyramid generative adversarial network

DUAN Youxiang¹, ZHANG Hanxiao¹, SUN Qifeng¹, SUN Youkai²

1. College of Computer Science and Technology, China University of Petroleum, Qingdao Shandong 266580, China;
2. Geophysical Research Institute of Sinopec Shengli Oilfield Company Limited, Dongying Shandong 257000, China

Received:2020-08-25 Revised:2020-09-15 Online:2020-11-05 Published:2021-04-10
Supported by:
This work is partially supported by the Fundamental Research Funds for the Central Universities (20CX05017A), the Major Scientific and Technological Project of CNPC (ZD2019-183-006).

基于拉普拉斯金字塔生成对抗网络的图像超分辨率重建算法

段友祥¹, 张含笑¹, 孙歧峰¹, 孙友凯²

1. 中国石油大学(华东)计算机科学与技术学院, 山东青岛 266580;
2. 中国石化胜利油田分公司物探研究院, 山东东营 257000

通讯作者: 张含笑
作者简介:段友祥（1964—），男，山东东营人，教授，博士，CCF会员，主要研究方向：网络与服务计算、计算机技术在油气领域的应用；张含笑（1995—），女，山东潍坊人，硕士研究生，主要研究方向：人工智能；孙歧峰（1976—），男，山东东营人，讲师，博士，主要研究方向：计算机技术在油气领域的应用；孙友凯（1974—），男，山东陵城人，高级工程师，硕士，主要研究方向：云计算。
基金资助:
中央高校基本科研业务费专项资金资助项目（20CX05017A）；中石油重大科技项目（ZD2019-183-006）。

Abstract

Abstract: Concerning the problems of poor reconstructing performance with large-scale factors and requirement of separate training in image reconstruction with different scales in current image super-resolution reconstruction algorithms, an image super-resolution reconstruction algorithm based on Laplacian pyramid Generative Adversarial Network(GAN) was proposed. The pyramid structure generator of the proposed algorithm was used to realize the multi-scale image reconstruction, so as to reduce the difficulty in learning large-scale factors by progressive up-sampling, and dense connection was used between layers to enhance feature propagation, which effectively avoided the vanishing gradient problem. In the algorithm, Markovian discriminator was used to map the input data into the result matrix, and the generator was guided to pay attention to the local features of the image in the process of training, which enriched the details of the reconstructed images. Experimental results show that, when performing 2-times, 4-times and 8-times image reconstruction on Set5 and other benchmark datasets, the average Peak Signal-to-Noise Ratio(PSNR) of the proposed algorithm reaches 33.97 dB, 29.15 dB, 25.43 dB respectively, and the average Structural SIMilarity(SSIM) of the algorithm reaches 0.924, 0.840, 0.667 respectively, outperforming to those of other algorithms such as Super Resolution Convolutional Neural Network(SRCNN), fast and accurate image Super-Resolution with deep Laplacian pyramid Network(LapSRN) and Super-Resolution GAN(SRGAN), and the images reconstructed by the proposed algorithm retain more vivid textures and fine-grained details in subjective vision.

Key words: super-resolution reconstruction, large-scale factor, dense connection, Laplacian pyramid, Generative Adversarial Network (GAN)

摘要： 针对目前的图像超分辨率重建算法中存在的大尺度因子的重建效果较差、不同尺度的图像重建均需要单独训练等问题，提出一种基于拉普拉斯金字塔生成对抗网络（GAN）的图像超分辨率重建算法。算法中的生成器使用金字塔结构实现多尺度的图像重建，从而以渐进上采样的方式降低了大尺度因子的学习难度，并在层与层之间使用密集连接加强特征传播，从而有效避免了梯度弥散问题。算法中使用马尔可夫判别器将输入数据映射为结果矩阵，并在训练的过程中引导生成器关注图像的局部特征，从而丰富了重建图像的细节。实验结果表明：在Set5等基准数据集上分别进行放大2倍、4倍、8倍的图像重建时，所提算法的平均峰值信噪比（PSNR）分别达到了33.97 dB、29.15 dB、25.43 dB，平均结构相似性（SSIM）分别达到了0.924、0.840、0.667，相比用于超分辨率重建的卷积神经网络（SRCNN）、深度拉普拉斯金字塔超分辨率重建网络（LapSRN）、用于超分辨率重建的生成对抗式网络（SRGAN）等其他算法有较大提升，且其重建的图像在主观视觉上保留了更多生动的纹理和小颗粒细节。

关键词: 超分辨率重建, 大尺度因子, 密集连接, 拉普拉斯金字塔, 生成对抗网络

CLC Number:

TP391

DUAN Youxiang, ZHANG Hanxiao, SUN Qifeng, SUN Youkai. Image super-resolution reconstruction algorithm based on Laplacian pyramid generative adversarial network[J]. Journal of Computer Applications, 2021, 41(4): 1020-1026.

段友祥, 张含笑, 孙歧峰, 孙友凯. 基于拉普拉斯金字塔生成对抗网络的图像超分辨率重建算法[J]. 计算机应用, 2021, 41(4): 1020-1026.

References

[1] ZHANG S,LIANG G,PAN S,et al. A fast medical image super resolution method based on deep learning network[J]. IEEE Access,2019,7:12319-12327.
[2] YANG X,WU W,LIU K,et al. Long-distance object recognition with image super resolution:a comparative study[J]. IEEE Access,2018,6:13429-13438.
[3] KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics, Speech, and Signal Processing,1981,29(6):1153-1160.
[4] 符祥, 郭宝龙. 图像插值技术综述[J]. 计算机工程与设计, 2009,30(1):141-144,193.(FU X,GUO B L. Overview of image interpolation technology[J]. Computer Engineering and Design,2009,30(1):141-144,193.)
[5] KARIMI E,KANGARLOO K,JAVADI S. A survey on superresolution methods for image reconstruction[J]. International Journal of Computer Applications,2014,90(3):32-39.
[6] LU Z,WU C,CHEN D,et al. Overview on image super resolution reconstruction[C]//Proceedings of the 26th Chinese Control and Decision Conference. Piscataway:IEEE,2014:2009-2014.
[7] 黄陶冶, 赵建伟, 周正华. 双层可变形卷积网络的超分辨率图像重建[J]. 计算机应用,2019,39(S2):68-74.(HUANG T Y, ZHAO J W,ZHOU Z H. Super-resolution image reconstruction using two-layer deformable convolution networks[J]. Journal of Computer Applications,2019,39(S2):68-74.)
[8] WANG Z,LIU D,YANG J,et al. Deep networks for image superresolution with sparse prior[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2015:370-378.
[9] 孙旭, 李晓光, 李嘉锋, 等. 基于深度学习的图像超分辨率复原研究进展[J]. 自动化学报,2017,43(5):697-709.(SUN X,LI X G,LI J F,et al. Review on deep learning based image superresolution restoration algorithms[J]. Acta Automatica Sinica, 2017,43(5):697-709.)
[10] DONG C,LOY C C,HE K,et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-307.
[11] 南方哲, 钱育蓉, 行艳妮, 等. 基于深度学习的单图像超分辨率重建研究综述[J]. 计算机应用研究,2020,37(2):321-326. (NAN F Z,QIAN Y R,XING Y N,et al. Survey of single image super resolution based on deep learning[J]. Application Research of Computers,2020,37(2):321-326.)
[12] LAI W S,HUANG J B,AHUJA N,et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:5835-5843.
[13] HA V K,REN J,XU X,et al. Deep learning based single image super-resolution:a survey[C]//Proceedings of the 9th International Conference on Brain Inspired Cognitive Systems, LNCS 10989. Cham:Springer,2018:106-119.
[14] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2014:2672-2680.
[15] LEDIG C,THEIS L,HUSZÁR F,et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:105-114.
[16] WANG X,YU K,WU S,et al. ESRGAN:enhanced superresolution generative adversarial networks[C]//Proceedings of the 15th European Conference on Computer Vision,LNCS 11133. Cham:Springer,2018:63-79.
[17] WANG Y, PERAZZI F, MCWILLIAMS B, et al. A fully progressive approach to single-image super-resolution[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE, 2018:977-986.
[18] LIM B,SON S,KIM H,et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE,2017:1132-1140.
[19] ISOLA P,ZHU J Y,ZHOU T,et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:5967-5976.
[20] LAN Z,LIN M,LI X,et al. Beyond Gaussian pyramid:multiskip feature stacking for action recognition[C]//Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2015:204-212.
[21] BURT P,ADELSON E. The Laplacian pyramid as a compact image code[J]. IEEE Transactions on Communications,1983,31(4):532-540.
[22] PARIS S,HASINOFF S W,KAUTZ J. Local Laplacian filters:edge-aware image processing with a Laplacian pyramid[J]. Communications of the ACM,2015,58(3):81-91.
[23] DENTON E,CHINTALA S,SZLAM A,et al. Deep generative image models using a Laplacian pyramid of adversarial networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2015:1486-1494.
[24] TONG T,LI G,LIU X,et al. Image super-resolution using dense skip connections[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:4809-4817.
[25] SHI W,CABALLERO J,HUSZÁR F,et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1874-1883.
[26] BIAN Y A,LI X,LIU Y,et al. Parallel coordinate descent newton method for efficient L1-regularized loss minimization[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019,30(11):3233-3245.
[27] BÜHLMANN P,YU B. Boosting with the L2 loss:regression and classification[J]. Journal of the American Statistical Association, 2003,98(462):324-339.
[28] NICOLSON A,PALIWAL K K. Deep learning for minimum meansquare error approaches to speech enhancement[J]. Speech Communication,2019,111:44-55.

Image super-resolution reconstruction algorithm based on Laplacian pyramid generative adversarial network

基于拉普拉斯金字塔生成对抗网络的图像超分辨率重建算法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[2]	Haoran WANG, Dan YU, Yuli YANG, Yao MA, Yongle CHEN. Domain transfer intrusion detection method for unknown attacks on industrial control systems [J]. Journal of Computer Applications, 2024, 44(4): 1158-1165.
[3]	Sunjie YU, Hui ZENG, Shiyu XIONG, Hongzhou SHI. Incentive mechanism for federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(2): 344-352.
[4]	Hui ZHOU, Yuling CHEN, Xuewei WANG, Yangwen ZHANG, Jianjiang HE. Deep shadow defense scheme of federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(1): 223-232.
[5]	Anyang LIU, Huaici ZHAO, Wenlong CAI, Zechao XU, Ruideng XIE. Adaptive image deblurring generative adversarial network algorithm based on active discrimination mechanism [J]. Journal of Computer Applications, 2023, 43(7): 2288-2294.
[6]	Shaoquan CHEN, Jianping CAI, Lan SUN. Differential privacy generative adversarial network algorithm with dynamic gradient threshold clipping [J]. Journal of Computer Applications, 2023, 43(7): 2065-2072.
[7]	Min LIANG, Jiayi LIU, Jie LI. Image super-resolution reconstruction method based on iterative feedback and attention mechanism [J]. Journal of Computer Applications, 2023, 43(7): 2280-2287.
[8]	Xin JIN, Yangchuan LIU, Yechen ZHU, Zijian ZHANG, Xin GAO. Sinogram inpainting for sparse-view cone-beam computed tomography image reconstruction based on residual encoder-decoder generative adversarial network [J]. Journal of Computer Applications, 2023, 43(6): 1950-1957.
[9]	Jinwen GUO, Xinghua MA, Gongning LUO, Wei WANG, Yang CAO, Kuanquan WANG. Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer [J]. Journal of Computer Applications, 2023, 43(5): 1596-1605.
[10]	Jiagao WU, Shiwen ZHANG, Yudong JIANG, Linfeng LIU. Social-interaction GAN for pedestrian trajectory prediction based on state-refinement long short-term memory and attention mechanism [J]. Journal of Computer Applications, 2023, 43(5): 1565-1570.
[11]	Xiaoyu FAN, Suzhen LIN, Yanbo WANG, Feng LIU, Dawei LI. Reconstruction algorithm for highly undersampled magnetic resonance images based on residual graph convolutional neural network [J]. Journal of Computer Applications, 2023, 43(4): 1261-1268.
[12]	Hao WANG, Zicheng WANG, Chao ZHANG, Yunsheng MA. Generative adversarial network based data uncertainty quantification method [J]. Journal of Computer Applications, 2023, 43(4): 1094-1101.
[13]	Chunyong YIN, Liwen ZHOU. Unsupervised time series anomaly detection model based on re-encoding [J]. Journal of Computer Applications, 2023, 43(3): 804-811.
[14]	Lingling TAO, Bo LIU, Wenbo LI, Xiping HE. Controllable face editing algorithm with closed-form solution [J]. Journal of Computer Applications, 2023, 43(2): 601-607.
[15]	Li’an ZHU, Hong ZHANG. Nonhomogeneous image dehazing based on dual-branch conditional generative adversarial network [J]. Journal of Computer Applications, 2023, 43(2): 567-574.