Image super-resolution reconstruction based on deep progressive back-projection attention network

doi:10.11772/j.issn.1001-9081.2019122155

Abstract

Abstract: Focused on the problems of Single Image Super-Resolution (SISR) reconstruction methods, such as the loss of high frequency information during the process of image reconstruction, the introduction of noise during the process of upsampling and the difficulty of determining the interdependence relationships between the channels of the feature map, a deep progressive back-projection attention network was proposed. Firstly, a progressive upsampling method was used to gradually scale the Low Resolution (LR) image to a given magnification in order to alleviate problems such as high-frequency information loss caused by upsampling. Then, at each stage of progressive upsampling, iterative back-projection idea was merged to learn mapping relationship between High Resolution (HR) and LR feature maps and reduce the introduced noise in the upsampling process. Finally, the attention mechanism was used to dynamically allocate attention resources to the feature maps generated at different stages of the progressive back-projection network, so that the interdependence relationships between the feature maps were learned by the network model. Experimental results show that the proposed method can increase the Peak Signal-to-Noise Ratio (PSNR) by up to 3.16 dB and the structural similarity by up to 0.218 4.

Key words: super-resolution, progressive upsampling, back-projection network, attention mechanism, deep learning

摘要： 针对多数单帧图像超分辨率（SISR）方法在重建预测图像时存在高频信息丢失和上采样过程中会引入噪声以及特征图各通道之间的相互依赖关系难以确定等问题，提出了深度渐进式反投影注意力网络。首先使用渐进式上采样方法将低分辨率（LR）图像逐步缩放至给定的倍率，缓解上采样过程中造成的高频信息丢失等问题；然后在渐进式上采样的每个阶段融合迭代反投影思想，学习高分辨率（HR）和LR特征图之间的映射关系并减少上采样过程中引入的噪声；最后使用注意力机制为渐进式反投影网络不同阶段产生的特征图动态分配注意力资源，使网络模型学习到各特征图之间的相互依赖关系。实验结果表明，所提出的方法相比主流的超分辨率方法，峰值信噪比（PSNR）最高可增加3.16 dB，结构相似性最高可提升0.218 4。

关键词: 超分辨率, 渐进式上采样, 反投影网络, 注意力机制, 深度学习

CLC Number:

TP391.41

HU Gaopeng, CHEN Ziliu, WANG Xiaoming, ZHANG Kaifang. Image super-resolution reconstruction based on deep progressive back-projection attention network[J]. Journal of Computer Applications, 2020, 40(7): 2077-2083.

胡高鹏, 陈子鎏, 王晓明, 张开放. 基于深度渐进式反投影注意力网络的图像超分辨率重建[J]. 计算机应用, 2020, 40(7): 2077-2083.

References

[1] DONG C,LOY C C,HE K,et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-307.
[2] KIM J,LEE J K,LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1646-1654.
[3] KIM J,KWON LEE J,MU LEE K. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the 2016 IEEE International Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1637-1645.
[4] SHI W,CABALLERO J,HUSZÁR F,et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1874-1883.
[5] LAI W S,HUANG J B,AHUJA N,et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:5835-5843.
[6] LIM B,SON S,KIM H,et al. Enhanced deep residual networks for single image super-resolution[C]//Proceeding of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE,2017:1132-1140.
[7] LEDIG C,THEIS L,HUSZÁR F,et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:105-114.
[8] HARIS M,SHAKHNAROVICH G,UKITA N. Deep back-projection networks for super-resolution[C]//Proceeding of the 2018 IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:1664-1673.
[9] ZHANG Y,LI K,LI K,et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the 15th European Conference on Computer Vision,LNCS 11211. Cham:Springer,2018:294-310.
[10] 邵文泽, 韦志辉. 基于各向异性MRF建模的多帧图像变分超分辨率重建[J]. 电子学报,2009,37(6):1256-1263.(SHAO W Z,WEI Z H. Multi-frame super-resolution reconstruction based on anisotropic Markov random field modeling[J]. Acta Electronica Sinica,2009,37(6):1256-1263.)
[11] 李展, 张庆丰, 孟小华, 等. 多分辨率图像序列的超分辨率重建[J]. 自动化学报,2012,38(11):1804-1814.(LI Z,ZHANG Q F,MENG X H,et al. Super-resolution reconstruction of multi-resolution image sequences[J]. Acta Automatica Sinica,2012,38(11):1804-1814.)
[12] 任福全, 邱天爽, 韩军, 等. 基于二阶广义全变差的多帧图像超分辨率重建[J]. 电子学报,2015,43(7):1275-1280.(REN F Q,QIU T S,HAN J,et al. Multiframe image super resolution based on second order total generalized variation[J]. Acta Electronica Sinica,2015,43(7):1275-1280.)
[13] 严宏海, 卜方玲, 徐新. 基于结构张量的视频超分辨率算法[J]. 计算机应用,2016,36(7):1944-1948,1987.(YAN H H, PU F L,XU X. Video super-resolution algorithm based on structure tensor[J]. Journal of Computer Applications,2016,36(7):1944-1948,1987.)
[14] 吉晓红, 熊淑华, 何小海, 等. 多阶导数自适应视频超分辨率重建[J]. 计算机应用,2016,36(4):1092-1095,1150.(JI X H, XIONG S H,HE X H,et al. Adaptive video super-resolution reconstruction algorithm based on multi-order derivative[J]. Journal of Computer Applications,2016,36(4):1092-1095,1150.)
[15] 孙琰玥, 何小海, 宋海英, 等. 一种用于视频超分辨率重建的块匹配图像配准方法[J]. 自动化学报,2011,37(1):37-43. (SUN Y Y,HE X H,SONG H Y,et al. A block-matching image registration algorithm for video super-resolution reconstruction[J]. Acta Automatica Sinica,2011,37(1):37-43.)
[16] TIMOFTE R,AGUSTSSON E,VAN GOOL L,et al. NTIRE 2017 challenge on single image super-resolution:methods and results[C]//Proceeding of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE, 2017:1110-1121.
[17] HE K,ZHANG X,REN S,et al. Delving deep into rectifiers:surpassing human-level performance on ImageNet classification[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE,2015:1026-1034.
[18] KINGMA D P,BA J. Adam:a method for stochastic optimization[EB/OL].[2018-12-22]. https://arxiv.org/pdf/1412.6980.pdf.
[19] YE Y,SHAN J,BRUZZONE L,et a1. Robust registration of multimodal remote sensing images based on structural similarity[J]. IEEE Transactions on Geoscience and Remote Sensing,2017,55(5):2941-2958.
[20] TAI Y,YANG J,LIU X,et al. MemNet:a persistent memory network for image restoration[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:4539-4547.
[21] DONG C,LOY C C,TANG X. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 14th European Conference on Computer Vision,LNCS 9906. Cham:Springer,2016:391-407.
[22] BEVILACQUA M,ROUMY A,GUILLEMOT C,et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceeding of the 2012 British Machine Vision Conference. Durham:BMVA Press,2012:No. 135.
[23] ZEYDE R,ELAD M,PROTTER M. On single image scale-up using sparse-representations[C]//Proceeding of the 7th International Conference on Curves and Surfaces,LNCS 6920. Berlin:Springer,2010:711-730.
[24] MARTIN D,FOWLKES C,TAL D,et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceeding of the 8th IEEE International Conference on Computer Vision. Piscataway:IEEE,2001:416-423.
[25] HUANG J B,SINGH A,AHUJA N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of the 2015 Computer Vision and Pattern Recognition. Piscataway:IEEE, 2015:5197-5206.
[26] WANG Z,LIU D,YANG J,et al. Deep networks for image superresolution with sparse prior[C]//Proceeding of the 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2015:370-378.
[27] ZHANG K,ZUO W,ZHANG L. Learning a single convolutional super-resolution network for multiple degradations[C]//Proceeding of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:3262-3271.
[28] ZHANG Y,TIAN Y,KONG Y,et al. Residual dense network for image super-resolution[C]//Proceeding of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:2472-2481.