Enhanced algorithm of image super-resolution based on dual-channel convolutional neural networks

doi:10.11772/j.issn.1001-9081.2018040820

Abstract

Abstract: The single-channel image super-resolution method can not achieve both fast convergence and high quality texture detail restoration. In order to solve the problem, a new Enhanced algorithm of image Super-Resolution based on Dual-channel Convolutional neural network (EDCSR) was proposed. Firstly, the network was divided into deep channel and shallow channel. Deep channel was used to extract detailed texture information of images, and shallow channel was mainly used to restore the overall contour of images. Then, the advantages of residual learning were used by the deep channel to deepen network and reduce parameters of model, eliminate the network degradation problem caused by too deep network. The long and short-term memory blocks were constructed to eliminate the artifacts and noise caused by the deconvolution layer. The texture information of image at different scales were extracted by a multi-scale method, while the shallow channel only needed to be responsible for restoring the main contour of image. Finally, the dual-channel losses were integrated to optimize the network continuously, which guided the network to generate high-resolution images. The experimental results show that, compared with the End-to-End image super-resolution algorithm via Deep and Shallow convolutional networks (EEDS), the proposed algorithm converges more quickly, image edge and texture reconstruction effects are significantly improved, the Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilarity (SSIM) are improved averagely by 0.15 dB and 0.0031 on data set Set5, while these are improved averagely by 0.18 dB and 0.0035 on data set Set14.

Key words: super-resolution, dual-channel, residual learning, deconvolution, convolution kernel parameter, long and short term memory block

摘要： 针对单通道图像超分辨率方法难以同时实现快速的收敛性能以及高质量的纹理细节恢复的问题，提出一种基于双通道卷积神经网络的图像超分辨率增强算法。首先，网络分为深层通道和浅层通道，深层通道用于提取图像的详细纹理信息，浅层通道用于恢复图像的总体轮廓。然后，深层通道利用残差学习的优势，加深网络并降低模型参数规模，消除因网络过深导致的网络退化问题，构造长短期记忆块消除由反卷积层造成的伪影现象和噪声，采用多尺度方式，提取图像不同尺度的纹理信息，而浅层通道只需负责恢复图像主要轮廓。最后，融合两通道损失对网络不断优化，指导网络生成高分辨率图像。实验结果表明，相比基于深层和浅层卷积神经网络的端到端图像超分辨率算法（EEDS），所提算法收敛更迅速，图像边缘和纹理重建效果明显提升，其峰值信噪比（PSNR）和结构相似性（SSIM）在Set5数据集上平均提高了0.15 dB、0.0031，在和Set14数据集上平均提高了0.18 dB、0.0035。

关键词: 超分辨率, 双通道, 残差学习, 反卷积, 卷积核参数, 长短期记忆块

CLC Number:

TP391.41

JIA Kai, DUAN Xintao, LI Baoxia, GUO Daidou. Enhanced algorithm of image super-resolution based on dual-channel convolutional neural networks[J]. Journal of Computer Applications, 2018, 38(12): 3563-3569.

贾凯, 段新涛, 李宝霞, 郭玳豆. 基于双通道卷积神经网络的图像超分辨率增强算法[J]. 计算机应用, 2018, 38(12): 3563-3569.

References

[1] ZHANG L P, ZHANG H Y, SHEN H F, et al. A super-resolution reconstruction algorithm for surveillance images[J]. Signal Processing, 2010, 90(3):848-859.
[2] THORNTON M W, ATKINSON P M, HOLLAND D A. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J]. International Journal of Remote Sensing, 2006, 27(3):473-491.
[3] SHI W Z, CABALLERO J, LEDIG C, et al. Cardiac image super-resolution with global correspondence using multi-atlas PpatchMatch[J]. Medical Image Computing and Computer-Assisted Intervention, 2013, 16(Pt 3):9-16.
[4] GUNTURK B K, BATUR A U, ALTUNBASAK Y, et al. Eigenface-domain super-resolution for face recognition[J]. IEEE Transactions on Image Processing, 2003, 12(5):597-606.
[5] KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics Speech and Signal Processing, 2003, 29(6):1153-1160.
[6] 孙旭,李晓光,李嘉锋,等.基于深度学习的图像超分辨率复原研究进展[J].自动化学报,2017,43(5):697-709.(SUN X, LI X G, LI J F, et al. Review on deep learning based image super-resolution restoration algorithms[J]. Acta Automatica Sinica, 2017, 43(5):697-709.)
[7] IRANI M, PELEG S. Motion analysis for image enhancement:resolution, occlusion, and transparency[J]. Journal of Visual Communication and Image Representation, 1993, 4(4):324-335.
[8] BAUSCHKE H H, BORWEIN J M. On projection algorithms for solving convex feasibility problems[J]. SIAM Review, 1996, 38(3):367-426.
[9] SCHULTZ R R, STEVENSON R L. Extraction of high-resolution frames from video sequences[J]. IEEE Transactions on Image Processing, 1996, 5(6):996-1011.
[10] TIMOFTE R, de SMET V, van GOOL L. Anchored neighborhood regression for fast example-based super-resolution[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2013:1920-1927.
[11] TIMOFTE R, de SMET V, van GOOL L. A+:adjusted anchored neighborhood regression for fast super-resolution[C]//Proceedings of the 12th Asian Conference on Computer Vision Computer Vision. Berlin:Springer, 2014:111-126.
[12] YANG J C, WRIGHT J, HUANG T S, et al. Image super-resolution as sparse representation of raw image patches[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2008:1-8.
[13] YANG J C, WRIGHT J, HUANG T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11):2861-2873.
[14] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. North Miami Beach, FL:Curran Associates Inc. 2012:1097-1105.
[15] DONG C, CHEN C L, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307.
[16] DONG C, CHEN C L, TANG X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 2016 European Conference on Computer Vision. Berlin:Springer, 2016:391-407.
[17] WANG Y F, WANG L J, WANG H Y, et al. End-to-end image super-resolution via deep and shallow convolutional networks[EB/OL].[2018-04-01]. http://cn.arxiv.org/pdf/1607.07680.
[18] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE., 2016:1646-1654.
[19] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2018-04-01]. https://arxiv.org/pdf/1409.1556.pdf.
[20] TAI Y, YANG J, LIU X M, et al. MemNet:a persistent memory network for image restoration[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2017:4549-4557.
[21] HE K, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2016:770-778.
[22] NAIR V, HINTON G E. Rectified linear units improve restricted boltzmann machines[C]//Proceedings of the 27th International Conference on Machine Learning. Madison, Wisconsin:Omnipress, 2010:807-814.
[23] HE K M, ZHANG X Y, REN S Q, et al. Delving deep into rectifiers:surpassing human-level performance on ImageNet classification[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2015:1026-1034.
[24] SUN J, XU Z B, SHUM H-Y, et al. Image super-resolution using gradient profile prior[C]//CVPR 2008:Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2008:1-8.
[25] NAH S, KIM T H, LEE K M. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2017:257-265.
[26] ZEILER M D, FERGUS R. Visualizing and understanding convolutional networks[C]//Proceedings of the 2014 European Conference on Computer Vision, LNCS 8689. Berlin:Springer, 2014:818-833.
[27] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11):2278-2324.
[28] HORÉ A, ZIOU D. Image quality metrics:PSNR vs. SSIM[C]//Proceedings of the 20th International Conference on Pattern Recognition. Piscataway, NJ:IEEE, 2010:2366-2369.
[29] YE Y X, SHAN J, BRUZZONE L, et al. Robust registration of multimodal remote sensing images based on structural similarity[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5):2941-2958.
[30] TIMOFTE R, AGUSTSSON E, van GOOL L, et al. NTIRE 2017 challenge on single image super-resolution:methods and results[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2017:1110-1121.
[31] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the 2012 British Machine Vision Conference. Durham:BMVA Press, 2012:135.1-135.10.
[32] ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference on Curves and Surfaces. Berlin:Springer, 2010:711-730
[33] HUANG J B, SINGH A, AHUJA N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2015:5197-5206.