计算机应用 ›› 2020, Vol. 40 ›› Issue (10): 3041-3047.DOI: 10.11772/j.issn.1001-9081.2020020185

• 虚拟现实与多媒体计算 • 上一篇    下一篇

结合感知边缘约束与多尺度融合网络的图像超分辨率重建方法

欧阳宁1,2, 韦羽2, 林乐平1,2   

  1. 1. 认知无线电与信息处理省部共建教育部重点实验室(桂林电子科技大学), 广西 桂林 541004;
    2. 桂林电子科技大学 信息与通信学院, 广西 桂林 541004
  • 收稿日期:2020-02-24 修回日期:2020-04-02 出版日期:2020-10-10 发布日期:2020-04-24
  • 通讯作者: 林乐平
  • 作者简介:欧阳宁(1972-),男,湖南宁远人,教授,硕士,主要研究方向:数字图像处理、智能信息处理;韦羽(1995-),男,广西玉林人,硕士研究生,主要研究方向:模式识别、深度学习;林乐平(1980-),女,广西桂平人,副教授,博士,主要研究方向:机器学习、智能信息处理、图像信号处理。
  • 基金资助:
    国家自然科学基金资助项目(61661017,61967005,U1501252);中国博士后科学基金面上项目(2016M602923XB);广西自然科学基金资助项目(2017GXNSFBA198212);广西科技基地和人才专项(AD19110060);认知无线电与信息处理教育部重点实验室资助项目(CRKL190107,CRKL160104);桂林电子科技大学研究生教育创新计划项目(2019YCXS022)。

Image super-resolution reconstruction method combining perceptual edge constraint and multi-scale fusion network

OUYANG Ning1,2, WEI Yu2, LIN Leping1,2   

  1. 1. Key Laboratory of Cognitive Radio and Information Processing, Ministry of Education;(Guilin University of Electronic Technology), Guilin Guangxi 541004, China;
    2. School of Information and Communication, Guilin University of Electronic Technology, Guilin Guangxi 541004, China
  • Received:2020-02-24 Revised:2020-04-02 Online:2020-10-10 Published:2020-04-24
  • Supported by:
    This work is partially supported by the National Natural Science Foundation of China (61661017, 61967005, U1501252), the Surface Program of China Postdoctoral Science Foundation (2016M602923XB), the Natural Science Foundation of Guangxi (2017GXNSFBA198212), the Science and Technology Base and Talent Project of Guangxi (AD19110060), the Project of the Key Laboratory of Cognitive Radio and Information Processing of Ministry of Education (CRKL190107, CRKL160104), the Graduate Education Innovation Program of Guilin University of Electronic Technology (2019YCXS022).

摘要: 针对图像超分辨率重建模型需要大量参数去捕获低分辨率(LR)图像和高分辨率(HR)图像之间的统计关系,以及使用L1L2损失优化的网络模型不能有效恢复图像高频细节等问题,提出一种结合感知边缘约束与多尺度融合网络的图像超分辨率重建方法。该方法基于由粗到细的思想,设计了一种两阶段的网络模型。第一阶段通过卷积神经网络(CNN)提取图像特征,并将图像特征上采样至HR大小,得到粗糙特征;第二阶段使用多尺度估计将低维统计模型逐步逼近高维统计模型,将第一阶段输出的粗糙特征作为输入来提取图像多尺度特征,并通过注意力融合模块逐步融合不同尺度特征,以精细化第一阶段提取的特征。同时,该方法引入一种更丰富的卷积特征用于边缘检测,并将其作为感知边缘约束来优化网络,以更好地恢复图像高频细节。在Set5、Set14和BSDS100等基准数据集上进行实验,结果表明与现有的基于CNN的超分辨率重建方法相比,该方法不但能够重建出更为清晰的边缘和纹理,而且在×3和×4放大因子下的峰值信噪比(PSNR)和结构相似度(SSIM)都取得了一定的提升。

关键词: 卷积神经网络, 多尺度, 注意力融合, 感知边缘约束, 超分辨率重建

Abstract: Aiming at the problems that the image super-resolution reconstruction model requires a large number of parameters to capture the statistical relationship between Low-Resolution (LR) images and High-Resolution (HR) images, and the use of network models optimized by L1 or L2 loss cannot effectively recover the high-frequency details of the images, an image super-resolution reconstruction method combining perceptual edge constraint and multi-scale fusion network was proposed. Based on the idea from coarse to fine, a two-stage network model was designed in this method. At the first stage, Convolutional Neural Network (CNN) was used to extract image features and upsample the image features to the HR size in order to obtain rough features. At second stage, multi-scale estimation was used to gradually approximate the low-dimensional statistical model to the high-dimensional statistical model. The rough features output at the first stage were used as the input to extract the multi-scale features of the image, and the features of different scales were gradually fused together through the attention fusion module in order to refine the features extracted at the first stage. At the same time, a class of richer convolutional features was introduced for edge detection and used as the perceptual edge constraint to optimize the network, so as to better recover the high-frequency details of the images. Experimental results on benchmark datasets such as Set5, Set14 and BSDS100 show that compared with the existing CNN-based super-resolution reconstruction methods, the proposed method not only reconstructs sharper edges and textures, but also achieves certain improvements in Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilarity index (SSIM) when magnification factor is 3 and 4.

Key words: Convolutional Neural Network (CNN), multi-scale, attention fusion, perceptual edge constraint, super-resolution reconstruction

中图分类号: