Image super-resolution reconstruction based on hybrid deep convolutional network

doi:10.11772/j.issn.1001-9081.2019122149

Abstract

Abstract: Aiming at the problems of blurred image, large noise, and poor visual perception in the traditional image super-resolution reconstruction methods, a method of image super-resolution reconstruction based on hybrid deep convolutional network was proposed. Firstly, the low-resolution image was scaled down to the specified size in the up-sampling phase. Secondly, the initial features of the low-resolution image were extracted in the feature extraction phase. Thirdly, the extracted initial features were sent to the convolutional coding and decoding structure for image feature denoising. Finally, high-dimensional feature extraction and computation were performed on the reconstruction layer using the dilated convolution in order to reconstruct the high-resolution image, and the residual learning was used to quickly optimize the network in order to reduce the noise and make the reconstructed image have better definition and visual effect. Based on the Set14 dataset and scale of 4x, the proposed method was compared with Bicubic interpolation (Bicubic), Anchored neighborhood regression (A+), Super-Resolution Convolutional Neural Network (SRCNN), Very Deep Super-Resolution network (VDSR), Restoration Encoder-Decoder Network (REDNet). In the super-resolution experiments, compared with the above methods, the proposed method has the Peak Signal-to-Noise Ratio (PSNR) increased by 2.73 dB,1.41 dB,1.24 dB,0.72 dB and 1.15 dB respectively, and the Structural SIMilarity (SSIM) improved by 0.067 3,0.020 9,0.019 7,0.002 6 and 0.004 6 respectively. The experimental results show that the hybrid deep convolutional network can effectively perform super-resolution reconstruction of the image.

Key words: image super-resolution reconstruction, image feature denoising, hybrid deep convolutional network, deconvolution, dilated convolution

摘要： 针对传统图像超分辨率重建方法存在的重建图像模糊、噪声量大、视觉感差等问题，提出了一种基于混合深度卷积网络的图像超分辨率重建方法。首先，在上采样阶段将低分辨率图像放缩至指定大小；然后，在特征提取阶段提取低分辨率图像的初始特征；接着，将提取到的初始特征送入卷积编解码结构进行图像特征去噪；最后，在重建层用空洞卷积进行高维特征提取与运算，重建出高分辨率图像，并且使用残差学习快速优化网络，在降低噪声的同时，使重建图像的清晰度及视觉效果更优。在Set14数据集放大尺度×4的基准下，将所提方法与双三次插值（Bicubic）、锚定邻域回归（A+）、超分辨卷积神经网络（SRCNN）、极深度超分辨网络（VDSR）、编解码网络（REDNet）等超分辨率重建方法进行对比。在超分辨实验中，所提方法与对比方法比较，峰值信噪比（PSNR）分别提升了2.73 dB、1.41 dB、1.24 dB、0.72 dB和1.15 dB，结构相似性（SSIM）分别提高了0.067 3，0.020 9，0.019 7，0.002 6和0.004 6。实验结果表明，混合深度卷积网络能够有效地对图像进行超分辨率重建。

关键词: 图像超分辨率重建, 图像特征去噪, 混合深度卷积网络, 反卷积, 空洞卷积

CLC Number:

TP391.4

HU Xueying, GUO Hairu, ZHU Rong. Image super-resolution reconstruction based on hybrid deep convolutional network[J]. Journal of Computer Applications, 2020, 40(7): 2069-2076.

胡雪影, 郭海儒, 朱蓉. 基于混合深度卷积网络的图像超分辨率重建[J]. 计算机应用, 2020, 40(7): 2069-2076.

References

[1] GLASNER D,BAGON S,IRANI M. Super-resolution from a single image[C]//Proceedings of the IEEE 12th International Conference on Computer Vision. Piscataway:IEEE,2009:349-356.
[2] IRANI M,PELEG S. Improving resolution by image registration[J]. CVGIP:Graphical Models and Image Processing,1991,53(3):231-239.
[3] 孔玲莉, 黄华, 齐春, 等. 图像超分辨率研究的最新进展[J]. 光学技术,2004,30(3):374-377.(KONG L L,HUANG H,QI C, et al. Recent development in super-resolution[J]. Optica:Technique,2004,30(3):374-377.)
[4] 江静, 张雪松. 图像超分辨率重建算法综述[J]. 红外技术, 2012,34(1):24-30.(JIANG J,ZHANG X S,A review of superresolution reconstruction algorithms[J]. Infrared Technology, 2012,34(1):24-30.)
[5] HA V K,REN J,XU X,et al. Deep learning based single image super-resolution:a survey[J]. International Journal of Automation and Computing,2019,16(4):413-426.
[6] ZEILER M D,FERGUS R. Visualizing and understanding convolutional networks[C]//Proceedings of the 13th European Conference on Computer Vision, LNCS 8689. Cham:Springer, 2014:818-833.
[7] DONG C,LOY C C,HE K,et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-307.
[8] KIM J,LEE J K,LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1637-1645.
[9] KIM J,LEE J K,LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1646-1654.
[10] WANG X,YU K,WU S,et al. ESRGAN:enhanced super-resolution generative adversarial networks[C]//Proceedings of the 15th European Conference on Computer Vision,LNCS 11133. Cham:Springer,2018:63-79.
[11] MAO X,SHEN C,YANG Y. Image restoration using convolutional auto-encoders with symmetric skip connections[C]//NIPS 2016:Advances in Neural Information Processing Systems 29. North Miami Beach,FL:Curran Associates Inc.,2016:2802-2810.
[12] ZHANG Y,LI K,LI K,et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the 15th European Conference on Computer Vision,LNCS 11211. Cham:Springer,2018:294-310.
[13] HUI Z,WANG X,GAO X. Fast and accurate single image superresolution via information distillation network[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:723-731.
[14] 邵保泰, 汤心溢, 金璐, 等. 基于生成对抗网络的单帧红外图像超分辨算法[J]. 红外与毫米波学报,2018,37(4):427-432. (SHAO B T,TANG X Y,JIN L,et al. Single frame infrared image super-resolution algorithm based on generative adversarial nets[J]. Journal of Infrared and Millimeter Waves,2018,37(4):427-432.)
[15] 徐军, 刘慧, 郭强, 等. 结合反卷积的CT图像超分辨重建网络[J]. 计算机辅助设计与图形学学报,2018,30(11):2084-2092(XU J,LIU H,GUO Q,et al. Super-resolution reconstruction of CT images using neural network combined with deconvolution[J]. Journal of Computer-Aided Design and Computer Graphics,2018, 30(11):2084-2092.)
[16] 朱福珍, 刘越, 黄鑫, 等. 改进的稀疏表示遥感图像超分辨重建[J]. 光学精密工程,2019,27(3):718-725.(ZHU F Z,LIU Y, HUANG X,et al. Remote sensing image super-resolution based on improved sparse representation[J]. Optical Precision Engineering,2019,27(3):718-725.)
[17] 邓承志. 图像稀疏表示理论及其应用研究[D]. 武汉:华中科技大学,2008:18.(DENG C Z. Image sparse representation theory and its application[D]. Wuhan:Huazhong University of Science and Technology,2008:18.)
[18] ZEILER M D,KRISHNAN D,TAYLOR G W,et al. Deconvolutional networks[C]//Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2010:2528-2535.
[19] 陈扬钛, 钟平. 基于L1正则化反卷积网络的遥感图像表述与复原方法[J]. 数字技术与应用,2011(5):158-160,184.(CHEN Y T,ZHONG P. Remote sensing image representation and restoration method based on L1 regularized deconvolution network[J]. Digital Technology and Application, 2011(5):158-160,184.)
[20] SHELHAMER E, LONG J,DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, Piscataway:IEEE, 2017,39(4):640-651.
[21] XU L,REN J S J,LIU C,et al. Deep convolutional neural network for image deconvolution[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2014:1790-1798.
[22] RADFORD A,METZ L,CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL].[2019-09-12]. https://arxiv.org/pdf/1511.06434.pdf.
[23] LECUN Y L,BOTTOU L,BENGIO Y,et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE,1998,86(11):2278-2324.
[24] KRIZHEVSKY A,SUTSKEVER I,HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2012:1097-1105.
[25] SIMONYAN K,ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2019-09-12]. https://arxiv.org/pdf/1409.1556.pdf.
[26] HE K,ZHANG X,REN S,et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:770-778.
[27] YU F,KOLTUN V. Multi-scale context aggregation by dilated convolutions[EB/OL].[2019-09-12]. https://arxiv.org/pdf/1511.07122.pdf.
[28] KALCHBRENNER N,ESPEHOLT L,SIMONYAN K,et al. Neural machine translation in linear time[EB/OL].[2019-09-12]. https://arxiv.org/pdf/1610.10099.pdf.
[29] VAN DEN OORD A,DIELEMAN S,ZEN H,et al. WaveNet:a generative model for raw audio[EB/OL].[2019-09-08]. https://arxiv.org/pdf/1609.03499.pdf.
[30] AHARON M,ELAD M,BRUCKSTEIN A. K-SVD:an algorithm for designing overcomplete dictionaries for sparse representation[J]. IEEE Transactions on Signal Processing,2006,54(11):4311-4322.
[31] HERVÉ A,WILLIAMS L J. Principal component analysis[J]. WIREs Computational Statistics,2010,2(4):433-459.
[32] RHEE S,KANG M G. Discrete cosine transform based regularized high-resolution image reconstruction algorithm[J]. Optical Engineering,1999,38(8):1348-1356.
[33] NAIR V,HINTON G E. Rectified linear units improve restricted Boltzmann machines[C]//Proceedings of the 27th International Conference on Machine Learning. Madison, WI:Omnipress, 2010:807-814.
[34] MAO X,SHEN C,YANG Y,et al. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. Red Hook,NY:Curran Associates Inc.,2016:2810-2818.
[35] 沈恒范. 概率论与数理统计教程[M]. 北京:高等教育出版社, 2006:56-58.(SHEN H F. Probability Theory and Mathematical Statistics Course[M]. Beijing:Higher Education Press,2006:56-58.)
[36] TURAGA D S,CHEN Y,CAVIEDES J,et al. No reference PSNR estimation for compressed pictures[J]. Signal Processing:Image Communication,2004,19(2):173-184.
[37] KINGMA D P,BA J. Adam:a method for stochastic optimization[EB/OL].[2019-09-12]. https://arxiv.org/pdf/1412.6980.pdf.
[38] MARTIN D,FOWLKES C,TAL D,et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision. Piscataway:IEEE,2002:416-423.
[39] YANG J,WRIGHT J,HUANG T S,et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing,2010,19(11):2861-2873.
[40] TIMOFTE R,DE SMET V,VAN GOOL L. A+:adjusted anchored neighborhood regression for fast super-resolution[C]//Proceedings of the 12th Asian Conference on Computer Vision,LNCS 9006. Cham:Springer,2014:111-126.
[41] BEVILACQUA M,ROUMY A,GUILLEMOT C,et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the 2012 British Machine Vision Conference. Durham:BMVA Press,2012:No. 135.
[42] ZEYDE R,ELAD M,PROTTER M. On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference on Curves and Surfaces, LNCS 6920. Berlin:Springer,2010:711-730.
[43] HUANG J B,SINGH A,AHUJA N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2015:5197-5206.
[44] KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics Speech,and Signal Processing,1981,29(6):1153-1160.