Single image super-resolution reconstruction method based on improved convolutional neural network

doi:10.11772/j.issn.1001-9081.2018091887

Journal of Computer Applications ›› 2019, Vol. 39 ›› Issue (5): 1440-1447.DOI: 10.11772/j.issn.1001-9081.2018091887

• Virtual reality and multimedia computing • Previous Articles Next Articles

Single image super-resolution reconstruction method based on improved convolutional neural network

LIU Yuefeng, YANG Hanxi, CAI Shuang, ZHANG Chenrong

School of Information Engineering, Inner Mongolia University of Science & Technology, Baotou Inner Mongolia 014000, China

Received:2018-09-10 Revised:2018-11-19 Online:2019-05-14 Published:2019-05-10
Supported by:
This work is partially supported by the Inner Mongolia Natural Science Foundation (2018MS06019).

基于改进卷积神经网络的单幅图像超分辨率重建方法

刘月峰, 杨涵晰, 蔡爽, 张晨荣

内蒙古科技大学信息工程学院, 内蒙古包头 014000

通讯作者: 刘月峰
作者简介:刘月峰(1977-),男,内蒙古包头人,副教授,博士,主要研究方向:深度学习、图像处理;杨涵晰(1994-),女,内蒙古包头人,硕士研究生,主要研究方向:深度学习、图像处理;蔡爽(1993-),女,山东菏泽人,硕士研究生,主要研究方向:深度学习、入侵检测;张晨荣(1993-),男,内蒙古巴彦淖尔人,硕士研究生,主要研究方向:深度学习、知识图谱。
基金资助:
内蒙古自然科学基金资助项目（2018MS06019）。

Abstract

Abstract: Aiming at the problem of edge distortion and fuzzy texture detail information in reconstructed images, an image super-resolution reconstruction method based on improved Convolutional Neural Network (CNN) was proposed. Firstly, various preprocessing operations were performed on the underlying feature extraction layer by three interpolation methods and five sharpening methods, and the images which were only subjected to one interpolation operation and the images which were sharpened after interpolation operation were arranged into a 3D matrix. Then, the 3D feature map formed by the preprocessing was used as the multi-channel input of a deep residual network in the nonlinear mapping layer to obtain deeper texture detail information. Finally, for reducing image reconstruction time, sub-pixel convolution was introduced into the reconstruction layer to complete image reconstruction operation. Experimental results on several common datasets show that the proposed method achieves better restored texture detail information and high-frequency information in the reconstructed image compared with the classical methods. Furthermore, the Peak Signal-to-Noise Ratio (PSNR) was increased by 0.23 dB on average, and the structural similarity was increased by 0.0066 on average. The proposed method can better maintain the texture details of the reconstructed image and reduce the image edge distortion under the premise of ensuring the image reconstruction time, improving the performance of image reconstruction.

Key words: single image super-resolution reconstruction, deep learning, Convolutional Neural Network (CNN), multi-channel convolution, sub-pixel convolution

摘要： 对于重建图像存在的边缘失真和纹理细节信息模糊的问题，提出一种基于改进卷积神经网络（CNN）的图像超分辨率重建方法。首先在底层特征提取层以三种插值方法和五种锐化方法进行多种预处理操作，并将只进行一次插值操作的图像和先进行一次插值后进行一次锐化的图像合并排列成三维矩阵；然后在非线性映射层将预处理后构成的三维特征映射作为深层残差网络的多通道输入，以获取更深层次的纹理细节信息；最后在重建层为减少图像重建时间在网络结构中引入亚像素卷积来完成图像重建操作。在多个常用数据集上的实验结果表明，与经典方法相比，所提方法重建图像的纹理细节信息和高频信息能得到更好的恢复，峰值信噪比（PSNR）平均增加0.23 dB，结构相似性（SSIM）平均增加0.0066。在保证图像重建时间的前提下，所提方法更好地保持重建图像的纹理细节并减少图像边缘失真，提升重建图像的性能。

关键词: 单幅图像超分辨率重建, 深度学习, 卷积神经网络, 多通道卷积, 亚像素卷积

CLC Number:

LIU Yuefeng, YANG Hanxi, CAI Shuang, ZHANG Chenrong. Single image super-resolution reconstruction method based on improved convolutional neural network[J]. Journal of Computer Applications, 2019, 39(5): 1440-1447.

刘月峰, 杨涵晰, 蔡爽, 张晨荣. 基于改进卷积神经网络的单幅图像超分辨率重建方法[J]. 计算机应用, 2019, 39(5): 1440-1447.

References

[1] 孙旭, 李晓光, 李嘉锋, 等.基于深度学习的图像超分辨率复原研究进展[J]. 自动化学报, 2017, 43(5): 697-709. (SUN X, LI X G, LI J F, et al. Review on deep learning based image super-resolution restoration algorithms[J]. Acta Automatica Sinica, 2017, 43(5): 697-709.)
[2] 苏衡, 周杰, 张志浩.超分辨率图像重建方法综述[J]. 自动化学报, 2013, 39(8): 1202-1213. (SU H, ZHOU J, ZHANG Z H. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8): 1202-1213.)
[3] 李浪宇, 苏卓, 石晓红, 等.图像超分辨率重建中的细节互补卷积模型[J]. 中国图象图形学报, 2018, 23(4): 572-582. (LI L Y, SU Z, SHI X H, et al. Mutual-detail convolution model for image super-resolution reconstruction[J]. Journal of Image and Graphics, 2018, 23(4): 572-582.)
[4] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]// Proceedings of the 2016 IEEE Conference on Computer on Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 1646-1654.
[5] TIAN J, MA K K. A survey on super-resolution imaging[J]. Signal, Image and Video Processing, 2011, 5(3): 329-342.
[6] YANG C Y, MA C, YANG M H. Single-image super-resolution: a benchmark[C]// Proceedings of the 2014 European Conference on Computer Vision. Berlin: Springer, 2014: 372-386.
[7] YANG J C, WRIGHT J, HUANG T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873.
[8] 李云飞, 符冉迪, 金炜, 等.多通道卷积的图像超分辨率方法[J]. 中国图象图形学报, 2017, 22(12): 1690-1700. (LI Y F, FU R D, JIN W, et al. Image super-resolution using multi-channel convolution[J]. Journal of Image and Graphics, 2017, 22(12): 1690-1700.)
[9] ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations[C]// Proceedings of the 2010 International Conference on Curves and Surfaces, LNCS 6920. Berlin: Springer, 2010: 711-730.
[10] TIMOFTE R, DE V, GOOL L V. Anchored neighborhood regression for fast example-based super-resolution[C]// Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway, NJ: IEEE, 2013: 1920-1927.
[11] TIMOFTE R, ROTHE R, GOOL L V. A⁺: adjusted anchored neighborhood regression for fast super-resolution[C]// Proceedings of the 12th Asian Conference on Computer Vision. Berlin: Springer, 2015: 111-126.
[12] CHANG H, XIONG Y, YEUNG D Y. Super-resolution through neighbor embedding[C]// Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC: IEEE Computer Society, 2004: 275-282.
[13] DONG C, CHEN C L, HE K, et al. Learning a deep convolutional network for image super-resolution[C]// ECCV 2014: Proceedings of the 2014 European Conference on Computer Vision. Berlin: Springer, 2014: 184-199.
[14] 徐冉, 张俊格, 黄凯奇.利用双通道卷积神经网络的图像超分辨率算法[J]. 中国图象图形学报, 2016, 21(5): 556-564. (XU R, ZHANG J G, HUANG K Q. Image super-resolution using two-channel convolutional neural networks[J]. Journal of Image and Graphics, 2016, 21(5): 556-564.)
[15] 刘娜, 李翠华.基于多层卷积神经网络学习的单帧图像超分辨率重建方法[J]. 中国科技论文, 2015, 10(2): 201-206. (LIU N, LI C H. Single image super-resolution reconstruction via deep convolutional neural network[J]. China Sciencepaper, 2015, 10(2): 201-206.)
[16] YOUM G Y, BAE S H, KIM M. Image super-resolution based on convolution neural networks using multi-channel input[C]// Proceedings of the 2006 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop. Piscataway, NJ: IEEE, 2016:1-5.
[17] SHI W, CABALLERO J, HUSZAR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 1874-1883.
[18] LEDIG C, THEIS L, HUSZAR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 105-114.
[19] DONG C, CHEN C L, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
[20] JIA Y Q, SHELHAMER E, DONAHUE J, et al. Caffe: convolutional architecture for fast feature embedding[C]// Proceedings of the 22nd ACM International Conference on Multimedia. New York: ACM, 2014: 675-678.
[21] MEGHA G, YASHPAL L, VIVEK L. Analytical relation & comparison of PSNR and SSIM on babbon image and human eye perception using Matlab[J]. International Journal of Advanced Research in Engineering and Applied Sciences, 2015, 4(5): 108-119.

Single image super-resolution reconstruction method based on improved convolutional neural network

基于改进卷积神经网络的单幅图像超分辨率重建方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	CHEN Chengrui, SUN Ning, HE Shibiao, LIAO Yong. Deep learning-based joint channel estimation and equalization algorithm for C-V2X communications [J]. Journal of Computer Applications, 2021, 41(9): 2687-2693.
[2]	SONG Zhongshan, LIANG Jiarui, ZHENG Lu, LIU Zhenyu, TIE Jun. Remote sensing scene classification based on bidirectional gated scale feature fusion [J]. Journal of Computer Applications, 2021, 41(9): 2726-2735.
[3]	LI Kangkang, ZHANG Jing. Multi-layer encoding and decoding model for image captioning based on attention mechanism [J]. Journal of Computer Applications, 2021, 41(9): 2504-2509.
[4]	ZHANG Yongbin, CHANG Wenxin, SUN Lianshan, ZHANG Hang. Detection method of domains generated by dictionary-based domain generation algorithm [J]. Journal of Computer Applications, 2021, 41(9): 2609-2614.
[5]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[6]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[7]	WANG Hebing, ZHANG Chunmei. Facial landmark detection based on ResNeXt with asymmetric convolution and squeeze excitation [J]. Journal of Computer Applications, 2021, 41(9): 2741-2747.
[8]	ZHENG Zhiqiang, HU Xin, WENG Zhi, WANG Yuhe, CHENG Xi. Cattle eye image feature extraction method based on improved DenseNet [J]. Journal of Computer Applications, 2021, 41(9): 2780-2784.
[9]	XIE Defeng, JI Jianmin. Syntax-enhanced semantic parsing with syntax-aware representation [J]. Journal of Computer Applications, 2021, 41(9): 2489-2495.
[10]	DAI Yurou, YANG Qing, ZHANG Fengli, ZHOU Fan. Trajectory prediction model of social network users based on self-supervised learning [J]. Journal of Computer Applications, 2021, 41(9): 2545-2551.
[11]	HE Zhenghai, XIAN Yantuan, WANG Meng, YU Zhengtao. Case reading comprehension method combining syntactic guidance and character attention mechanism [J]. Journal of Computer Applications, 2021, 41(8): 2427-2431.
[12]	ZENG Xiangyin, ZHENG Bochuan, LIU Dan. Detection of left and right railway tracks based on deep convolutional neural network and clustering [J]. Journal of Computer Applications, 2021, 41(8): 2324-2329.
[13]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[14]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[15]	HUANG Chengcheng, DONG Xiaoxiao, LI Zhao. Deep pipeline 5×5 convolution method based on two-dimensional Winograd algorithm [J]. Journal of Computer Applications, 2021, 41(8): 2258-2264.