基于自注意力网络的图像超分辨率重建

doi:10.11772/j.issn.1001-9081.2019010158

计算机应用 ›› 2019, Vol. 39 ›› Issue (8): 2391-2395.DOI: 10.11772/j.issn.1001-9081.2019010158

• 虚拟现实与多媒体计算 • 上一篇下一篇

基于自注意力网络的图像超分辨率重建

欧阳宁^1,2, 梁婷², 林乐平^1,2

1. 认知无线电与信息处理省部共建教育部重点实验室(桂林电子科技大学), 广西桂林 541004;
2. 桂林电子科技大学信息与通信学院, 广西桂林 541004

收稿日期:2019-01-23 修回日期:2019-03-15 发布日期:2019-04-15 出版日期:2019-08-10
通讯作者: 林乐平
作者简介:欧阳宁(1972-),男,湖南宁远人,教授,硕士,主要研究方向:数字图像处理、智能信息处理;梁婷(1993-),女,广西梧州人,硕士研究生,主要研究方向:模式识别、深度学习;林乐平(1980-),女,广西桂平人,副教授,博士,主要研究方向:机器学习、智能信息处理、图像信号处理。
基金资助:
国家自然科学基金资助项目（61661017）；中国博士后科学基金资助项目（2016M602923XB）；认知无线电与信息处理重点实验室基金资助项目（CRKL160104，CRKL150103，2011KF11）；广西自然科学基金资助项目（2017GXNSFBA198212，2014GXNSFDA118035，2016GXNSFAA38014）；广西科技创新能力与条件建设计划项目（桂科能1598025-21）；桂林电子科技大学研究生教育创新计划项目（2016YJCXB02）；桂林科技开发项目（20150103-6）。

Self-attention network based image super-resolution

OUYANG Ning^1,2, LIANG Ting², LIN Leping^1,2

1. Key Laboratory of Cognitive Radio and Information Processing of Ministry of Education(Guilin University of Electronic Technology), Guilin Guangxi 541004, China;
2. School of Information and Communication, Guilin University of Electronic Technology, Guilin Guangxi 541004, China

Received:2019-01-23 Revised:2019-03-15 Online:2019-04-15 Published:2019-08-10
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61661017), the China Postdoctoral Science Foundation (2016M602923XB), the Fundation of the Key Laboratory of Cognitive Radio and Information Processing of Ministry of Education (CRKL160104, CRKL150103, 2011KF11), the Natural Science Foundation of Guangxi (2017GXNSFBA198212, 2014GXNSFDA118035, 2016GXNSFAA38014), the Scientific and Technological Innovation Ability and Condition Construction Plan of Guangxi (1598025-21), the Innovation Project of GUET Graduate Education (2016YJCXB02), the Scientific and Technological Bureau of Guilin (20150103-6).

摘要/Abstract

摘要： 针对图像超分辨率重建中纹理细节等高频信息恢复的问题，提出一种基于自注意力网络的图像超分辨率重建方法。该网络框架利用两个重建阶段逐步地将图像的精确度从粗到细进行恢复。在第一阶段中，首先将低分辨率（LR）图像作为输入通过一个卷积神经网络（CNN），并输出一个粗精度的高分辨率（HR）图像；然后将粗精度图像作为输入并产生更加精细的高分辨率图像。在第二阶段中，使用自注意力模块计算特征之间所有位置的关联性，通过捕捉特征的全局依赖关系来提高纹理细节的恢复能力。在基准数据集上的实验结果表明，与现有基于深度神经网路的超分辨率重建算法相比，所提算法不仅图像视觉效果最好，而且在数据集Set5和BDSD100上的峰值信噪比（PSNR）平均提高了0.1 dB、0.15 dB，表明该网络可以通过增强特征的全局表达能力来重建出高质量图像。

关键词: 深度卷积神经网络, 从粗到细, 自注意力, 全局依赖关系, 超分辨率

Abstract: Concerning the recovery problem of high-frequency information like texture details in image super-resolution reconstruction, an image super-resolution reconstruction method based on self-attention network was proposed. Two reconstruction stages were used to gradually restore the image accuracy from-coarse-to-fine. In the first stage, firstly, a Low-Resolution (LR) image was taken as the input through a Convolutional Neural Network (CNN), and a High-Resolution (HR) image was output with coarse precision; then, the coarse HR image was used as the input and a finer HR image was produced. In the second stage, the correlation of all positions between features was calculate by the self-attention module, and the global dependencies of features were captured to enhance texture details. Experimental results on the benchmark datasets show that, compared with the state-of-the-art deep neural networks based super-resolution algorithms, the proposed algorithm not only has the best visual effect, but also has the Peak Signal-to-Noise Ratio (PSNR) improved averagely by 0.1dB and 0.15dB on Set5 and BDSD100. It indicates that the network can enhance the global representation ability of features to reconstruct high quality images.

Key words: deep convolutional neural network, from-coarse-to-fine, self-attention, global dependency, super-resolution

中图分类号:

TP391.41

欧阳宁, 梁婷, 林乐平. 基于自注意力网络的图像超分辨率重建[J]. 计算机应用, 2019, 39(8): 2391-2395.

OUYANG Ning, LIANG Ting, LIN Leping. Self-attention network based image super-resolution[J]. Journal of Computer Applications, 2019, 39(8): 2391-2395.

参考文献

[1] DENTON E L, CHINTALA S, SZLAM A, et al. Deep generative image models using a Laplacian pyramid of adversarial networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge, MA:MIT Press, 2015:1486-1494.
[2] ILG E, MAYER N, SAIKIA T, et al. FlowNet 2.0:Evolution of optical flow estimation with deep networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2017:1647-1655.
[3] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2016:770-778.
[4] DONG C, LOY C, HE K, at al. Image super-resolution using deep convolutional networks[J]//IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307.
[5] DONG C, LOY C C, TANG X. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 2016 European Conference on Computer Vision, LNCS 9906. Cham:Springer, 2016:391-407.
[6] KIM J, LEE J K, and LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2016:1646-1654.
[7] REN H, EL-KHAMY M, LEE J. Image super resolution based on fusing multiple convolution neural networks[C]//Proceedings of the 2017 IEEE Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2017:1050-1057.
[8] LAI W, HUANG J, AHUJA N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2017:5835-5843.
[9] ZHOU L, WANG Z, WANG S, et al. Coarse-to-fine image super-resolution using convolutional neural networks[C]//Proceedings of the 2018 International Conference on Multimedia Modeling, LNCS 10705. Cham:Springer, 2018:73-81.
[10] GLASNER D, BAGON S, IRANI M. Super-resolution from a single image[C]//Proceedings of the IEEE 12th Intermational Conference on Computer Vision. Piscataway, NJ:IEEE, 2009:349-356.
[11] SIMONYAN K, ZISSEMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv E-print, 2015:arXiv:1409.1556.
[12] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st Conference on Neural Information Processing Systems. Cambridge, MA:MIT Press, 2017:6000-6010.
[13] ZHANG H, GOODFELLOW I., METAXAS D, et al. Self-attention generative adversarial networks[J]. arXiv E-print, 2019:arXiv:1805.08318.
[14] HE K, ZHANG X, REN S, et al. Identity mappings in deep residual networks[C]//Proceedings of the 2016 European Conference on Computer Vision, LNCS 9908. Cham:Springer, 2016:630-645.
[15] NAH S, KIM T H, LEE K M. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2017:257-265.
[16] AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution:dataset and study[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2017:1122-1131.
[17] KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1981, 29(6):1153-1160.
[18] 孙旭,李晓光,李嘉锋,等.基于深度学习的图像超分辨率复原研究进展[J].自动化学报,2017,43(5):697-709. (SUN X, LI X G, LI J F, et al. Review on deep learning based image super-resolution restoration algorithms[J]. Acta Automatica Sinica, 2017, 43(5):697-709.)

基于自注意力网络的图像超分辨率重建

Self-attention network based image super-resolution

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	党伟超, 李涛, 白尚旺, 高改梅, 刘春霞. 基于自注意力长短期记忆网络的Web软件系统实时剩余寿命预测方法[J]. 计算机应用, 2021, 41(8): 2346-2351.
[2]	牛康力, 谌雨章, 沈君凤, 曾张帆, 潘永才, 王绎冲. 基于深度学习的双通道夜视图像复原方法[J]. 计算机应用, 2021, 41(6): 1775-1784.
[3]	梁敏, 王昊榕, 张瑶, 李杰. 基于加速残差网络的图像超分辨率重建方法[J]. 计算机应用, 2021, 41(5): 1438-1444.
[4]	卞鹏程, 郑忠龙, 李明禄, 何依然, 王天翔, 张大伟, 陈丽媛. 基于注意力融合网络的视频超分辨率重建[J]. 计算机应用, 2021, 41(4): 1012-1019.
[5]	段友祥, 张含笑, 孙歧峰, 孙友凯. 基于拉普拉斯金字塔生成对抗网络的图像超分辨率重建算法[J]. 计算机应用, 2021, 41(4): 1020-1026.
[6]	王永金, 左羽, 吴恋, 崔忠伟, 赵晨洁. 基于注意力机制的图像超分辨率重建[J]. 计算机应用, 2021, 41(3): 845-850.
[7]	吴锐, 刘宇, 冯凯. 基于双域自注意力机制的行人属性识别[J]. 计算机应用, 2021, 41(2): 372-378.
[8]	李萌, 秦品乐, 曾建潮, 李俊伯. 基于磁共振影像层间插值的超分辨率及多视角融合[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3362-3367.
[9]	林静, 黄玉清, 李磊民. 基于球形矩匹配与特征判别的图像超分辨率重建[J]. 计算机应用, 2020, 40(8): 2345-2350.
[10]	陈佳伟, 韩芳, 王直杰. 基于自注意力门控图卷积网络的特定目标情感分析[J]. 计算机应用, 2020, 40(8): 2202-2206.
[11]	李生武, 张选德. 基于自注意力机制的多域卷积神经网络的视觉追踪[J]. 计算机应用, 2020, 40(8): 2219-2224.
[12]	胡高鹏, 陈子鎏, 王晓明, 张开放. 基于深度渐进式反投影注意力网络的图像超分辨率重建[J]. 计算机应用, 2020, 40(7): 2077-2083.
[13]	胡雪影, 郭海儒, 朱蓉. 基于混合深度卷积网络的图像超分辨率重建[J]. 计算机应用, 2020, 40(7): 2069-2076.
[14]	代强, 程曦, 王永梅, 牛子未, 刘飞. 基于轻量自动残差缩放网络的图像超分辨率重建[J]. 计算机应用, 2020, 40(5): 1446-1452.
[15]	陈赛健, 朱远平. 基于生成对抗网络的文本图像联合超分辨率与去模糊方法[J]. 计算机应用, 2020, 40(3): 859-864.