Improved algorithm of image super resolution based on residual neural network

doi:10.11772/j.issn.1001-9081.2017061461

Abstract

Abstract: To efficiently improve the effects of image Super Resolution (SR), a multi-stage cascade residual convolution neural network model was proposed. Firstly, two-stage SR image reconstruction method was used to reconstruct the 2-times SR image and then reconstruct the 4-times SR image; secondly, residual layer and jump layer were used to predict the texture information of the high resolution space in the first and second stages, and deconvolution layer was used to reconstruct 2-times and 4-times SR images. Finally, two multi-task loss functions were constructed respectively by the results of two stages. And the loss of the first stage guided that of the second one, which accelerated the network training and enhanced the supervision and guidance of the network learning. The experimental results show that compared with bilinear algorithm, bicubic algorithm, Super Resolution using Convolutional Neural Network (SRCNN) algorithm and Fast Super Resolution Convolutional Neural Network (FSRCNN) algorithm, the proposed model can better construct the details and texture of images, which avoids the image over smoothing after iterating, and achieves higher Peak Signal-to-Noise Ratio (PSNR) and Mean Structural SIMilarity (MSSIM).

Key words: Super Resolution (SR), Deep Learning (DL), residual block, jump layer, deconvolution, multi-task loss

摘要： 为更有效地提升图像的超分辨率（SR）效果，提出了一种多阶段级联残差卷积神经网络模型。首先，该模型采用了两阶段超分辨率图像重建方法先重建2倍超分辨率图像，再重建4倍超分辨率图像；其次，第一阶段与第二阶段皆使用残差层和跳层结构预测出高分辨率空间的纹理信息，由反卷积层分别重建出2倍与4倍大小的超分辨率图像；最后，以两阶段的结果分别构建多任务损失函数，利用第一阶段的损失指导第二阶段的损失，从而提高网络的训练速度，加强网络学习中的监督指导。实验结果表明，与bilinear算法、bicubic算法、基于卷积神经网络的图像超分辨率（SRCNN）算法和加速的超分辨率卷积神经网络（FSRCNN）算法相比，所提模型能更好地重建出图像的细节和纹理，避免了经过迭代之后造成的图像过度平滑，获得更高的峰值信噪比（PSNR）和平均结构相似度（MSSIM）。

关键词: 超分辨率, 深度学习, 残差块, 跳层, 反卷积, 多任务损失

CLC Number:

TP391.413

WANG Yining, QIN Pinle, LI Chuanpeng, CUI Yuhao. Improved algorithm of image super resolution based on residual neural network[J]. Journal of Computer Applications, 2018, 38(1): 246-254.

王一宁, 秦品乐, 李传朋, 崔雨豪. 基于残差神经网络的图像超分辨率改进算法[J]. 计算机应用, 2018, 38(1): 246-254.

References

[1] PELED S, YESHURUN Y. Superresolution in MRI:application to human white matter fiber tract visualization by diffusion tensor imaging[J]. Magnetic Resonance in Medicine Official Journal of the Society of Magnetic Resonance in Medicine, 2001, 45(1):29.
[2] SHI W, CABALLERO J, LEDIG C, et al. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch[C]//MICCAI 2013:Proceedings of the 2013 International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin:Springer, 2013:9-16.
[3] THORNTON M W, ATKINSON P M, HOLLAND D A. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J]. International Journal of Remote Sensing, 2006, 27(3):473-491.
[4] GUNTURK B K, BATUR A U, ALTUNBASAK Y, et al. Eigenface-domain super-resolution for face recognition[J]. IEEE Transactions on Image Processing, 2003, 12(5):597-606.
[5] ZHANG L, ZHANG H, SHEN H, et al. A super-resolution reconstruction algorithm for surveillance images[J]. Signal Processing, 2010, 90(3):848-859.
[6] 苏衡,周杰,张志浩.超分辨率图像重建方法综述[J].自动化学报,2013,39(8):1202-1213.(SU H, ZHOU J, ZHANG Z H. A summary of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8):1202-1213.)
[7] KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics Speech & Signal Processing, 1981, 29(6):1153-1160.
[8] 袁琪,荆树旭.改进的序列图像超分辨率重建方法[J].计算机应用,2009,29(12):3310-3313.(YUAN Q, JIN S X. Improved sequence image super-resolution reconstruction method[J]. Journal of Computer Applications, 2009, 29(12):3310-3313.)
[9] TIMOFTE R, DE V, GOOL L V. Anchored neighborhood regression for fast example-based super-resolution[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2013:1920-1927.
[10] TIMOFTE R, SMET V D, GOOL L V. A+:adjusted anchored neighborhood regression for fast super-resolution[C]//Proceedings of the 2014 Asian Conference on Computer Vision. Berlin:Springer, 2014:111-126.
[11] YANG C Y, YANG M H. Fast direct super-resolution by simple functions[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision. Washington, DC:IEEE Computer Society, 2013:561-568.
[12] YANG J, WRIGHT J, HUANG T, et al. Image super-resolution as sparse representation of raw image patches[C]//CVPR 2008:Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2008:1-8.
[13] YANG J, WRIGHT J, HUANG T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11):2861-2873.
[14] WANG J, ZHU S, GONG Y. Resolution enhancement based on learning the sparse association of image patches[J]. Pattern Recognition Letters, 2010, 31(1):1-10.
[15] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 2012 International Conference on Neural Information Processing Systems. West Chester, OH:Curran Associates Inc., 2012:1097-1105.
[16] DONG C, CHEN C L, HE K, et al. Learning a deep convolutional network for image super-resolution[C]//ECCV 2014:Proceedings of the 2014 European Conference on Computer Vision. Berlin:Springer, 2014:184-199.
[17] DONG C, CHEN C L, TANG X. Accelerating the super-resolution convolutional neural network[C]//ECCV 2016:Proceedings of the 2016 European Conference on Computer Vision. Berlin:Springer, 2016:391-407.
[18] SUN J, SHUM H Y. Image super-resolution using gradient profile prior:US9064476[P]. 2015-06-23.
[19] NAIR V, HINTON G E. Rectified linear units improve restricted boltzmann machines[C]//Proceedings of the 2010 International Conference on International Conference on Machine Learning. Madison, WI:Omnipress, 2010:807-814.
[20] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//CVPR 2016:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2016:770-778.
[21] JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the 2016 European Conference on Computer Vision. Berlin:Springer, 2016:694-711.
[22] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2017-03-21]. https://arxiv.org/abs/1409.1556.
[23] REN S, HE K, GIRSHICK R, et al. Object detection networks on convolutional feature maps[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(7):1476-1481.
[24] SRIVASTAVA R K, GREFF K, SCHMIDHUBER J. Training very deep networks[EB/OL].[2017-03-20]. https://arxiv.org/abs/1507.06228.
[25] SAXE A M, MCCLELLAND J L, GANGULI S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks[EB/OL].[2017-03-12]. http://arxiv.org/abs/1312.6120.
[26] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2017-03-10]. https://arxiv.org/abs/1409.1556.
[27] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the 2012 British Machine Vision Conference.[S.l.]:BMVC Press, 2012:135-1-135-10.
[28] ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations[C]//Proceedings of the 2010 International Conference on Curves and Surfaces. Berlin:Springer, 2010:711-730.
[29] JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the 2016 European Conference on Computer Vision. Berlin:Springer, 2016:694-711.