真实复杂场景下基于残差收缩网络的单幅图像超分辨率方法

doi:10.11772/j.issn.1001-9081.2022111697

《计算机应用》唯一官方网站 ›› 2023, Vol. 43 ›› Issue (12): 3903-3910.DOI: 10.11772/j.issn.1001-9081.2022111697

• 多媒体计算与计算机仿真 • 上一篇下一篇

真实复杂场景下基于残差收缩网络的单幅图像超分辨率方法

李颖¹^,², 黄超¹^,², 孙成栋¹^,², 徐勇¹^,²()

^1.哈尔滨工业大学(深圳) 计算机科学与技术学院, 广东深圳 518055
^2.深圳市视觉目标检测与判识重点实验室(哈尔滨工业大学(深圳)), 广东深圳 518055

收稿日期:2022-11-24 修回日期:2023-07-06 接受日期:2023-07-07 发布日期:2023-07-24 出版日期:2023-12-10
通讯作者: 徐勇
作者简介:李颖（1998—），男，四川成都人，硕士研究生，主要研究方向：计算机视觉、超分辨率
黄超（1991—），男，河南信阳人，博士研究生，主要研究方向：模式识别、深度学习
孙成栋（2002—），男，湖北黄冈人，主要研究方向：计算机视觉、超分辨率；
基金资助:
国家自然科学基金资助项目(61876051);深圳市科创委资助项目(JSGG20220831104402004)

Single image super-resolution method based on residual shrinkage network in real complex scenes

Ying LI¹^,², Chao HUANG¹^,², Chengdong SUN¹^,², Yong XU¹^,²()

^1.College of Computer Science and Technology，Harbin Institute of Technology，Shenzhen，Shenzhen Guangdong 518055，China
^2.Shenzhen Key Laboratory of Visual Object Detection and Recognition （Harbin Institute of Technology，Shenzhen），Shenzhen Guangdong 518055，China

Received:2022-11-24 Revised:2023-07-06 Accepted:2023-07-07 Online:2023-07-24 Published:2023-12-10
Contact: Yong XU
About author:LI Ying， born in 1998， M. S. candidate. His research interests include computer vision， super-resolution.
HUANG Chao， born in 1991， Ph. D. candidate. His research interests include pattern recognition， deep learning.
SUN Chengdong， born in 2002. His research interests include computer vision， super-resolution.
Supported by:
National Natural Science Foundation of China(61876051);Project of Shenzhen Science and Technology Innovation Committee(JSGG20220831104402004)

摘要/Abstract

摘要：

真实世界中极少存在成对的高低分辨率图像对，传统的基于图像对训练模型的单幅图像超分辨率（SR）方法采用合成数据集的方式得到训练集时仅考虑了双线性下采样退化，且传统图像超分辨率方法在面向真实的未知退化图像时重建效果较差。针对上述问题，提出一种面向真实复杂场景的图像超分辨率方法。首先，采用不同焦距对景物进行拍摄并配准得到相机采集的真实高低分辨率图像对，构建一个场景多样的数据集CSR（Camera Super-Resolution dataset）；其次，为了尽可能地模拟真实世界中的图像退化过程，根据退化因素参数随机化和非线性组合退化改进图像退化模型，并且结合高低分辨率图像对数据集和图像退化模型以合成训练集；最后，由于数据集中考虑了退化因素，引入残差收缩网络和U-Net改进基准模型，尽可能地减少退化因素在特征空间中的冗余信息。实验结果表明，所提方法在复杂退化条件下相较于次优BSRGAN（Blind Super-Resolution Generative Adversarial Network）方法，在RealSR和CSR测试集中PSNR指标分别提高了0.7 dB和0.14 dB，而SSIM分别提高了0.001和0.031。所提方法在复杂退化数据集上的客观指标和视觉效果均优于现有方法。

关键词: 超分辨率, 复杂场景, 图像退化模型, 残差收缩网络

Abstract:

There are very few paired high and low resolution images in the real world. The traditional single image Super-Resolution （SR） methods typically use pairs of high-resolution and low-resolution images to train models， but these methods use the way of synthetizing dataset to obtain training set， which only consider bilinear downsampling as image degradation process. However， the image degradation process in the real word is complex and diverse， and traditional image super-resolution methods have poor reconstruction performance when facing real unknown degraded images. Aiming at those problems， a single image super-resolution method was proposed for real complex scenes. Firstly， high- and low-resolution images were captured by the camera with different focal lengths， and these images were registered as image pairs to form a dataset CSR（Camera Super-Resolution dataset） of various scenes. Secondly， to simulate the image degradation process in the real world as much as possible， the image degradation model was improved by the parameter randomization of degradation factors and the nonlinear combination degradation. Besides， the dataset of high- and low-resolution image pairs and the image degradation model were combined to synthetize training set. Finally， as the degradation factors were considered in the dataset， residual shrinkage network and U-Net were embedded into the benchmark model to reduce the redundant information caused by degradation factors in the feature space as much as possible. Experimental results indicate that compared with the BSRGAN （Blind Super-Resolution Generative Adversarial Network） method， under complex degradation conditions， the proposed method improves the PSNR by 0.7 dB and 0.14 dB， and improves SSIM by 0.001 and 0.031 respectively on the RealSR and CSR test sets. The proposed method has better objective indicators and visual effect than the existing methods on complex degradation datasets.

Key words: Super-Resolution (SR), complex scene, image degradation model, residual shrinkage network

中图分类号:

TP391.4

李颖, 黄超, 孙成栋, 徐勇. 真实复杂场景下基于残差收缩网络的单幅图像超分辨率方法[J]. 计算机应用, 2023, 43(12): 3903-3910.

Ying LI, Chao HUANG, Chengdong SUN, Yong XU. Single image super-resolution method based on residual shrinkage network in real complex scenes[J]. Journal of Computer Applications, 2023, 43(12): 3903-3910.

图/表 12

图1 残差收缩网络的结构

Fig.1 Structure of residual shrinkage network

图2 相机以不同焦距拍摄的图像对示例

Fig.2 Example of image pair captured by camera with different focal lengths

图3 SIFT算法关键点匹配结果

Fig.3 Result of keypoint matching by SIFT algorithm

图4 高低分辨率图像对示例

Fig.4 Example of high- and low-resolution image pair

图5 图像复杂退化模型

Fig.5 Complex image degradation model

图6 CSRGAN模型的结构

Fig.6 Structure of CSRGAN model

图7 RSDN的结构

Fig.7 Structure of RSDN

图8 RSDB的结构

Fig.8 Structure of RSDB

表1 各模型在RealSR和CSR测试集上的重建指标对比

Tab.1 Comparison of reconstruction indicators of different models on RealSR and CSR test sets

模型	RealSR						CSR
	D1		D2		CD		D1		D2		CD
	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
RRDB	20.25	0.312	22.35	0.510	23.60	0.589	20.25	0.315	23.62	0.538	25.07	0.634
ESRGAN	15.59	0.115	20.20	0.385	22.08	0.457	15.45	0.130	21.74	0.451	23.01	0.474
EDSR	20.54	0.326	22.49	0.517	23.67	0.593	20.80	0.339	23.81	0.547	25.17	0.639
RWSR-DF2K	18.16	0.201	21.40	0.428	22.31	0.461	18.16	0.199	22.66	0.480	23.26	0.471
SRMD	20.80	0.341	22.60	0.524	23.68	0.595	21.12	0.359	23.92	0.554	25.22	0.641
DASR	19.44	0.269	21.91	0.481	22.93	0.533	19.24	0.253	22.99	0.504	23.82	0.549
BSRGAN	22.69	0.659	23.23	0.661	24.13	0.683	23.44	0.666	24.45	0.673	25.46	0.698
CSRGAN	24.11	0.660	23.93	0.665	24.20	0.681	25.87	0.700	25.80	0.706	26.05	0.724

图9 各模型在不同退化条件下的视觉效果对比

Fig. 9 Comparison of visual effects of different models under different degradation conditions

表2 在测试集RealSR和CSR上的消融实验结果对比

Tab.2 Comparison of ablation experimental results on RealSR and CSR test sets

模型	RealSR						CSR
	D1		D2		CD		D1		D2		CD
	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
ESRGAN	15.59	0.115	20.20	0.385	22.08	0.457	15.45	0.130	21.74	0.451	23.01	0.474
M1	24.09	0.586	23.65	0.606	23.98	0.611	25.53	0.623	25.50	0.662	25.79	0.666
M2	24.09	0.649	23.87	0.653	24.18	0.666	25.65	0.682	25.58	0.693	26.06	0.708
CSRGAN	24.11	0.660	23.93	0.665	24.20	0.681	25.87	0.700	25.80	0.706	26.05	0.724

图10 不同退化条件下的消融实验视觉效果对比

Fig. 10 Visual effect comparison of ablation experiments under different degradation conditions

参考文献 26

1	王一宁，赵青杉，秦品乐，等. 基于轻量密集神经网络的医学图像超分辨率重建算法［J］. 计算机应用， 2022， 42（8）： 2586-2592. 10.11772/j.issn.1001-9081.2021061093
	WANG Y N， ZHAO Q S， QIN P L， et al. Super-resolution reconstruction algorithm of medical image based on lightweight dense neural network［J］. Journal of Computer Applications， 2022， 42（8）： 2586-2592. 10.11772/j.issn.1001-9081.2021061093
2	KEYS R. Bicubic interpolation［J］. IEEE Transactions on Acoustics， Speech， and Signal Processing， 1981， 29（1）： 1153-1160. 10.1109/tassp.1981.1163711
3	王汇丰，徐岩，魏一铭，等. 基于并联卷积与残差网络的图像超分辨率重建［J］. 计算机应用， 2022， 42（5）： 1570-1576.
	WANG H F， XU Y， WEI Y M， et al. Image super-resolution reconstruction based on parallel convolution and residual network［J］. Journal of Computer Applications， 2022， 42（5）： 1570-1576.
4	DONG C， LOY C C， HE K， et al. Learning a deep convolutional network for image super-resolution ［C］// Proceedings of the 2014 European Conference on Computer Vision， LNCS 8692. Cham： Springer， 2014： 184-199.
5	DONG C， LOY C C， TANG X， et al. Accelerating the super-resolution convolutional neural network［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9906. Cham： Springer， 2016： 391- 407.
6	KIM J， LEE J K， LEE K M. Deeply-recursive convolutional network for image super-resolution ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 1637-1645. 10.1109/cvpr.2016.181
7	KIM J， LEE J K， LEE K M. Accurate image super-resolution using very deep convolutional networks［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 1646-1654. 10.1109/cvpr.2016.182
8	LIM B， SON S， KIM H， et al. Enhanced deep residual networks for single image super-resolution［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2017： 1132-1140. 10.1109/cvprw.2017.151
9	TARG S， ALMEIDA D， LYMAN K. ResNet in ResNet： generalizing residual architectures ［EB/OL］. （2016-03-25）［2021-12-12］. .
10	LEDIG C， THEIS L， HUSZÁR F， et al. Photo-realistic single image super-resolution using a generative adversarial network ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 105-114. 10.1109/cvpr.2017.19
11	WANG X， YU K， WU S， et al. ESRGAN： enhanced super-resolution generative adversarial networks ［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11133. Cham： Springer， 2019： 63-79.
12	LIU A， LIU Y， GU J， et al. Blind image super-resolution： a survey and beyond ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2023， 45（5）： 5461-5480. 10.1109/tpami.2023.3312313
13	ZHANG K， ZUO W， ZHANG L. Learning a single convolutional super-resolution network for multiple degradations［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 3262-3271. 10.1109/cvpr.2018.00344
14	WEI Y， GU S， LI Y， et al. Unsupervised real-world image super resolution via domain-distance aware training ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 13380-13389. 10.1109/cvpr46437.2021.01318
15	JI X， CAO Y， TAI Y， et al. Real-world super-resolution via kernel estimation and noise injection ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2020： 1914-1923. 10.1109/cvprw50498.2020.00241
16	ZHANG K， LIANG J， VAN GOOL L， et al. Designing a practical degradation model for deep blind image super-resolution［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2021： 4771-4780. 10.1109/iccv48922.2021.00475
17	ZHAO M， ZHONG S， FU X， et al. Deep residual shrinkage networks for fault diagnosis ［J］. IEEE Transactions on Industrial Informatics， 2020， 16（7）： 4681-4690. 10.1109/tii.2019.2943898
18	CHEN C， XIONG Z， TIAN X， et al. Camera lens super-resolution［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 1652-1660. 10.1109/cvpr.2019.00175
19	CAI J， ZENG H， YONG H， et al. Toward real-world single image super-resolution： a new benchmark and a new model［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 3086-3095. 10.1109/iccv.2019.00318
20	刘丛，王亚新. 基于双并行残差网络的遥感图像超分辨率重建［J］. 模式识别与人工智能， 2021， 34（8）： 760-767.
	LIU C， WANG Y X. Remote sensing image super-resolution reconstruction based on dual-parallel residual network ［J］. Pattern Recognition and Artificial Intelligence， 2021， 34（8）： 760-767.
21	WANG X， XIE L， DONG C， et al. Real-ESRGAN： training real-world blind super-resolution with pure synthetic data［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway： IEEE， 2021：1905-1914. 10.1109/iccvw54120.2021.00217
22	SCHÖNFELD E， SCHIELE B， KHOREVA A. A U-Net based discriminator for generative adversarial networks ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 8204-8213. 10.1109/cvpr42600.2020.00823
23	RONNEBERGER O， FISCHER P， BROX T. U-Net： convolutional networks for biomedical image segmentation ［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9351. Cham： Springer， 2015： 234-241.
24	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［EB/OL］. （2015-04-10）［2021-12-12］..
25	JOHNSON J， ALAHI A， LI F F. Perceptual losses for real-time style transfer and super-resolution ［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9906. Cham： Springer， 2016： 694-711.
26	AGUSTSSON E， TIMOFTE R. NTIRE 2017 challenge on single image super-resolution： dataset and study ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2017： 1122-1131. 10.1109/cvprw.2017.150

[1]	申利华, 李波. 基于特征金字塔网络和密集网络的肺部CT图像超分辨率重建[J]. 《计算机应用》唯一官方网站, 2023, 43(5): 1612-1619.
[2]	王一宁, 赵青杉, 秦品乐, 胡玉兰, 宗春梅. 基于轻量密集神经网络的医学图像超分辨率重建算法[J]. 《计算机应用》唯一官方网站, 2022, 42(8): 2586-2592.
[3]	徐成霞, 阎庆, 李腾, 苗开超. 基于联合注意力机制的单幅图像去雨算法[J]. 《计算机应用》唯一官方网站, 2022, 42(8): 2578-2585.
[4]	李佳, 郑元林, 廖开阳, 楼豪杰, 李世宇, 陈泽豪. 基于显著性深层特征的无参考图像质量评价算法[J]. 《计算机应用》唯一官方网站, 2022, 42(6): 1957-1964.
[5]	张晔, 刘蓉, 刘明, 陈明. 基于多通道注意力机制的图像超分辨率重建网络[J]. 《计算机应用》唯一官方网站, 2022, 42(5): 1563-1569.
[6]	王汇丰, 徐岩, 魏一铭, 王会真. 基于并联卷积与残差网络的图像超分辨率重建[J]. 《计算机应用》唯一官方网站, 2022, 42(5): 1570-1576.
[7]	刘朋伟, 高媛, 秦品乐, 殷喆, 王丽芳. 基于多感受野的生成对抗网络医学MRI影像超分辨率重建[J]. 《计算机应用》唯一官方网站, 2022, 42(3): 938-945.
[8]	李俊伯, 秦品乐, 曾建潮, 李萌. 基于超分辨率网络的CT三维重建算法[J]. 《计算机应用》唯一官方网站, 2022, 42(2): 584-591.
[9]	王素玉, 杨静, 李越. 基于双注意力机制信息蒸馏网络的图像超分辨率复原算法[J]. 《计算机应用》唯一官方网站, 2022, 42(1): 239-244.
[10]	牛康力, 谌雨章, 沈君凤, 曾张帆, 潘永才, 王绎冲. 基于深度学习的双通道夜视图像复原方法[J]. 计算机应用, 2021, 41(6): 1775-1784.
[11]	梁敏, 王昊榕, 张瑶, 李杰. 基于加速残差网络的图像超分辨率重建方法[J]. 计算机应用, 2021, 41(5): 1438-1444.
[12]	段友祥, 张含笑, 孙歧峰, 孙友凯. 基于拉普拉斯金字塔生成对抗网络的图像超分辨率重建算法[J]. 计算机应用, 2021, 41(4): 1020-1026.
[13]	卞鹏程, 郑忠龙, 李明禄, 何依然, 王天翔, 张大伟, 陈丽媛. 基于注意力融合网络的视频超分辨率重建[J]. 计算机应用, 2021, 41(4): 1012-1019.
[14]	王永金, 左羽, 吴恋, 崔忠伟, 赵晨洁. 基于注意力机制的图像超分辨率重建[J]. 计算机应用, 2021, 41(3): 845-850.
[15]	王海勇, 张开心, 管维正. 基于密集Inception的单图像超分辨率重建方法[J]. 《计算机应用》唯一官方网站, 2021, 41(12): 3666-3671.

真实复杂场景下基于残差收缩网络的单幅图像超分辨率方法

Single image super-resolution method based on residual shrinkage network in real complex scenes

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 26

相关文章 15

编辑推荐

Metrics