基于双通道卷积神经网络的图像超分辨率增强算法

doi:10.11772/j.issn.1001-9081.2018040820

计算机应用 ›› 2018, Vol. 38 ›› Issue (12): 3563-3569.DOI: 10.11772/j.issn.1001-9081.2018040820

• 虚拟现实与多媒体计算 • 上一篇下一篇

基于双通道卷积神经网络的图像超分辨率增强算法

贾凯, 段新涛, 李宝霞, 郭玳豆

河南师范大学计算机与信息工程学院, 河南新乡 453007

收稿日期:2018-04-20 修回日期:2018-07-04 出版日期:2018-12-10 发布日期:2018-12-15
通讯作者: 段新涛
作者简介:贾凯(1991-),男,河南周口人,硕士研究生,主要研究方向:深度学习、数字图像处理;段新涛(1972-),男,河南新乡人,副教授,博士,CCF会员,主要研究方向:深度学习、盲信号处理、数字图像处理;李宝霞(1993-),女,河南周口人,硕士研究生,主要研究方向:深度学习、图像处理;郭玳豆(1993-),男,河南安阳人,硕士研究生,主要研究方向:深度学习、图像处理。
基金资助:
国家自然科学基金资助项目（U1204606）；河南省科技发展重点项目（172102210335）；河南省高校重点科研项目（16A520058）。

Enhanced algorithm of image super-resolution based on dual-channel convolutional neural networks

JIA Kai, DUAN Xintao, LI Baoxia, GUO Daidou

College of Computer and Information Engineering, Henan Normal University, Xinxiang Henan 453007, China

Received:2018-04-20 Revised:2018-07-04 Online:2018-12-10 Published:2018-12-15
Contact: 段新涛
Supported by:
This work is partially supported by the National Natural Science Foundation of China (U1204606), the Key Program for Science and Technology Development of Henan Province (172102210335), the Key Scientific Research Project in Henan Universities (16A520058).

摘要/Abstract

摘要： 针对单通道图像超分辨率方法难以同时实现快速的收敛性能以及高质量的纹理细节恢复的问题，提出一种基于双通道卷积神经网络的图像超分辨率增强算法。首先，网络分为深层通道和浅层通道，深层通道用于提取图像的详细纹理信息，浅层通道用于恢复图像的总体轮廓。然后，深层通道利用残差学习的优势，加深网络并降低模型参数规模，消除因网络过深导致的网络退化问题，构造长短期记忆块消除由反卷积层造成的伪影现象和噪声，采用多尺度方式，提取图像不同尺度的纹理信息，而浅层通道只需负责恢复图像主要轮廓。最后，融合两通道损失对网络不断优化，指导网络生成高分辨率图像。实验结果表明，相比基于深层和浅层卷积神经网络的端到端图像超分辨率算法（EEDS），所提算法收敛更迅速，图像边缘和纹理重建效果明显提升，其峰值信噪比（PSNR）和结构相似性（SSIM）在Set5数据集上平均提高了0.15 dB、0.0031，在和Set14数据集上平均提高了0.18 dB、0.0035。

关键词: 超分辨率, 双通道, 残差学习, 反卷积, 卷积核参数, 长短期记忆块

Abstract: The single-channel image super-resolution method can not achieve both fast convergence and high quality texture detail restoration. In order to solve the problem, a new Enhanced algorithm of image Super-Resolution based on Dual-channel Convolutional neural network (EDCSR) was proposed. Firstly, the network was divided into deep channel and shallow channel. Deep channel was used to extract detailed texture information of images, and shallow channel was mainly used to restore the overall contour of images. Then, the advantages of residual learning were used by the deep channel to deepen network and reduce parameters of model, eliminate the network degradation problem caused by too deep network. The long and short-term memory blocks were constructed to eliminate the artifacts and noise caused by the deconvolution layer. The texture information of image at different scales were extracted by a multi-scale method, while the shallow channel only needed to be responsible for restoring the main contour of image. Finally, the dual-channel losses were integrated to optimize the network continuously, which guided the network to generate high-resolution images. The experimental results show that, compared with the End-to-End image super-resolution algorithm via Deep and Shallow convolutional networks (EEDS), the proposed algorithm converges more quickly, image edge and texture reconstruction effects are significantly improved, the Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilarity (SSIM) are improved averagely by 0.15 dB and 0.0031 on data set Set5, while these are improved averagely by 0.18 dB and 0.0035 on data set Set14.

Key words: super-resolution, dual-channel, residual learning, deconvolution, convolution kernel parameter, long and short term memory block

中图分类号:

TP391.41

贾凯, 段新涛, 李宝霞, 郭玳豆. 基于双通道卷积神经网络的图像超分辨率增强算法[J]. 计算机应用, 2018, 38(12): 3563-3569.

JIA Kai, DUAN Xintao, LI Baoxia, GUO Daidou. Enhanced algorithm of image super-resolution based on dual-channel convolutional neural networks[J]. Journal of Computer Applications, 2018, 38(12): 3563-3569.

参考文献

[1] ZHANG L P, ZHANG H Y, SHEN H F, et al. A super-resolution reconstruction algorithm for surveillance images[J]. Signal Processing, 2010, 90(3):848-859.
[2] THORNTON M W, ATKINSON P M, HOLLAND D A. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J]. International Journal of Remote Sensing, 2006, 27(3):473-491.
[3] SHI W Z, CABALLERO J, LEDIG C, et al. Cardiac image super-resolution with global correspondence using multi-atlas PpatchMatch[J]. Medical Image Computing and Computer-Assisted Intervention, 2013, 16(Pt 3):9-16.
[4] GUNTURK B K, BATUR A U, ALTUNBASAK Y, et al. Eigenface-domain super-resolution for face recognition[J]. IEEE Transactions on Image Processing, 2003, 12(5):597-606.
[5] KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics Speech and Signal Processing, 2003, 29(6):1153-1160.
[6] 孙旭,李晓光,李嘉锋,等.基于深度学习的图像超分辨率复原研究进展[J].自动化学报,2017,43(5):697-709.(SUN X, LI X G, LI J F, et al. Review on deep learning based image super-resolution restoration algorithms[J]. Acta Automatica Sinica, 2017, 43(5):697-709.)
[7] IRANI M, PELEG S. Motion analysis for image enhancement:resolution, occlusion, and transparency[J]. Journal of Visual Communication and Image Representation, 1993, 4(4):324-335.
[8] BAUSCHKE H H, BORWEIN J M. On projection algorithms for solving convex feasibility problems[J]. SIAM Review, 1996, 38(3):367-426.
[9] SCHULTZ R R, STEVENSON R L. Extraction of high-resolution frames from video sequences[J]. IEEE Transactions on Image Processing, 1996, 5(6):996-1011.
[10] TIMOFTE R, de SMET V, van GOOL L. Anchored neighborhood regression for fast example-based super-resolution[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2013:1920-1927.
[11] TIMOFTE R, de SMET V, van GOOL L. A+:adjusted anchored neighborhood regression for fast super-resolution[C]//Proceedings of the 12th Asian Conference on Computer Vision Computer Vision. Berlin:Springer, 2014:111-126.
[12] YANG J C, WRIGHT J, HUANG T S, et al. Image super-resolution as sparse representation of raw image patches[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2008:1-8.
[13] YANG J C, WRIGHT J, HUANG T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11):2861-2873.
[14] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. North Miami Beach, FL:Curran Associates Inc. 2012:1097-1105.
[15] DONG C, CHEN C L, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307.
[16] DONG C, CHEN C L, TANG X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 2016 European Conference on Computer Vision. Berlin:Springer, 2016:391-407.
[17] WANG Y F, WANG L J, WANG H Y, et al. End-to-end image super-resolution via deep and shallow convolutional networks[EB/OL].[2018-04-01]. http://cn.arxiv.org/pdf/1607.07680.
[18] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE., 2016:1646-1654.
[19] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2018-04-01]. https://arxiv.org/pdf/1409.1556.pdf.
[20] TAI Y, YANG J, LIU X M, et al. MemNet:a persistent memory network for image restoration[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2017:4549-4557.
[21] HE K, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2016:770-778.
[22] NAIR V, HINTON G E. Rectified linear units improve restricted boltzmann machines[C]//Proceedings of the 27th International Conference on Machine Learning. Madison, Wisconsin:Omnipress, 2010:807-814.
[23] HE K M, ZHANG X Y, REN S Q, et al. Delving deep into rectifiers:surpassing human-level performance on ImageNet classification[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2015:1026-1034.
[24] SUN J, XU Z B, SHUM H-Y, et al. Image super-resolution using gradient profile prior[C]//CVPR 2008:Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2008:1-8.
[25] NAH S, KIM T H, LEE K M. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2017:257-265.
[26] ZEILER M D, FERGUS R. Visualizing and understanding convolutional networks[C]//Proceedings of the 2014 European Conference on Computer Vision, LNCS 8689. Berlin:Springer, 2014:818-833.
[27] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11):2278-2324.
[28] HORÉ A, ZIOU D. Image quality metrics:PSNR vs. SSIM[C]//Proceedings of the 20th International Conference on Pattern Recognition. Piscataway, NJ:IEEE, 2010:2366-2369.
[29] YE Y X, SHAN J, BRUZZONE L, et al. Robust registration of multimodal remote sensing images based on structural similarity[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5):2941-2958.
[30] TIMOFTE R, AGUSTSSON E, van GOOL L, et al. NTIRE 2017 challenge on single image super-resolution:methods and results[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2017:1110-1121.
[31] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the 2012 British Machine Vision Conference. Durham:BMVA Press, 2012:135.1-135.10.
[32] ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference on Curves and Surfaces. Berlin:Springer, 2010:711-730
[33] HUANG J B, SINGH A, AHUJA N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2015:5197-5206.

基于双通道卷积神经网络的图像超分辨率增强算法

Enhanced algorithm of image super-resolution based on dual-channel convolutional neural networks

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	牛瑞华, 杨俊, 邢斓馨, 吴仁彪. 基于卷积注意力模块和双通道网络的微表情识别算法[J]. 计算机应用, 2021, 41(9): 2552-2559.
[2]	卞凌志, 王直杰. 基于增强多维多粒度级联森林的信用评分模型[J]. 计算机应用, 2021, 41(9): 2539-2544.
[3]	闫钧华, 侯平, 张寅, 吕向阳, 马越, 王高飞. 基于双通道卷积神经网络的图像单失真类型判定方法[J]. 计算机应用, 2021, 41(6): 1761-1766.
[4]	黄梨, 卢龙. 基于长距离依赖编码与深度残差U-Net的缺血性卒中病灶分割[J]. 计算机应用, 2021, 41(6): 1820-1827.
[5]	牛康力, 谌雨章, 沈君凤, 曾张帆, 潘永才, 王绎冲. 基于深度学习的双通道夜视图像复原方法[J]. 计算机应用, 2021, 41(6): 1775-1784.
[6]	梁敏, 王昊榕, 张瑶, 李杰. 基于加速残差网络的图像超分辨率重建方法[J]. 计算机应用, 2021, 41(5): 1438-1444.
[7]	卞鹏程, 郑忠龙, 李明禄, 何依然, 王天翔, 张大伟, 陈丽媛. 基于注意力融合网络的视频超分辨率重建[J]. 计算机应用, 2021, 41(4): 1012-1019.
[8]	段友祥, 张含笑, 孙歧峰, 孙友凯. 基于拉普拉斯金字塔生成对抗网络的图像超分辨率重建算法[J]. 计算机应用, 2021, 41(4): 1020-1026.
[9]	王永金, 左羽, 吴恋, 崔忠伟, 赵晨洁. 基于注意力机制的图像超分辨率重建[J]. 计算机应用, 2021, 41(3): 845-850.
[10]	李萌, 秦品乐, 曾建潮, 李俊伯. 基于磁共振影像层间插值的超分辨率及多视角融合[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3362-3367.
[11]	林静, 黄玉清, 李磊民. 基于球形矩匹配与特征判别的图像超分辨率重建[J]. 计算机应用, 2020, 40(8): 2345-2350.
[12]	胡高鹏, 陈子鎏, 王晓明, 张开放. 基于深度渐进式反投影注意力网络的图像超分辨率重建[J]. 计算机应用, 2020, 40(7): 2077-2083.
[13]	胡雪影, 郭海儒, 朱蓉. 基于混合深度卷积网络的图像超分辨率重建[J]. 计算机应用, 2020, 40(7): 2069-2076.
[14]	代强, 程曦, 王永梅, 牛子未, 刘飞. 基于轻量自动残差缩放网络的图像超分辨率重建[J]. 计算机应用, 2020, 40(5): 1446-1452.
[15]	陈赛健, 朱远平. 基于生成对抗网络的文本图像联合超分辨率与去模糊方法[J]. 计算机应用, 2020, 40(3): 859-864.