Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (4): 1277-1284.DOI: 10.11772/j.issn.1001-9081.2023040523
Special Issue: 多媒体计算与计算机仿真
• Multimedia computing and computer simulation • Previous Articles Next Articles
Yang LIU, Rong LIU(), Ke FANG, Xinyue ZHANG, Guangxu WANG
Received:
2023-05-05
Revised:
2023-08-18
Accepted:
2023-08-24
Online:
2023-12-04
Published:
2024-04-10
Contact:
Rong LIU
About author:
LIU Yang, born in 1999, M. S. candidate. His research interests include computer vision.Supported by:
通讯作者:
刘蓉
作者简介:
刘扬(1999—),男,湖南长沙人,硕士研究生,主要研究方向:计算机视觉基金资助:
CLC Number:
Yang LIU, Rong LIU, Ke FANG, Xinyue ZHANG, Guangxu WANG. Video super-resolution reconstruction network based on frame straddling optical flow[J]. Journal of Computer Applications, 2024, 44(4): 1277-1284.
刘扬, 刘蓉, 方可, 张心月, 王光旭. 基于帧间跨越光流的视频超分辨率重建网络[J]. 《计算机应用》唯一官方网站, 2024, 44(4): 1277-1284.
Add to citation manager EndNote|Ris|BibTeX
URL: https://www.joca.cn/EN/10.11772/j.issn.1001-9081.2023040523
模型 | 运行时间/ms | 参数量/106 | 浮点运算量/ GFLOPs | REDS4 | Vid4 | SPMC | UDM10 | ||||
---|---|---|---|---|---|---|---|---|---|---|---|
PSNR/dB | SSIM | PSNR/dB | SSIM | PSNR/dB | SSIM | PSNR/dB | SSIM | ||||
Bicubic | — | — | — | 26.14 | 0.72 | 21.80 | 0.52 | 23.29 | 0.64 | 28.47 | 0.82 |
EDVR-m | 156 | 3.3 | 200.0 | 28.11 | 0.80 | 23.54 | 0.69 | 27.27 | 0.75 | 37.42 | 0.93 |
DUF-VSR | 559 | 5.8 | 736.6 | 29.40 | 0.81 | 25.01 | 0.73 | 27.96 | 0.79 | 37.97 | 0.93 |
BasicVSR | 1 306 | 6.3 | 163.7 | 30.21 | 0.85 | 24.94 | 0.74 | 28.52 | 0.82 | 38.20 | 0.94 |
RawVSR | 1 610 | 4.5 | 356.9 | 30.20 | 0.86 | 25.04 | 0.75 | 28.54 | 0.84 | 38.44 | 0.96 |
OVSR | 1 152 | 1.8 | 110.0 | 30.17 | 0.85 | 25.15 | 0.79 | 28.57 | 0.83 | 38.18 | 0.94 |
本文模型 | 1 544 | 6.3 | 336.4 | 30.47 | 0.87 | 25.17 | 0.79 | 28.59 | 0.83 | 38.31 | 0.95 |
Tab. 1 Results comparison of different models on four datasets
模型 | 运行时间/ms | 参数量/106 | 浮点运算量/ GFLOPs | REDS4 | Vid4 | SPMC | UDM10 | ||||
---|---|---|---|---|---|---|---|---|---|---|---|
PSNR/dB | SSIM | PSNR/dB | SSIM | PSNR/dB | SSIM | PSNR/dB | SSIM | ||||
Bicubic | — | — | — | 26.14 | 0.72 | 21.80 | 0.52 | 23.29 | 0.64 | 28.47 | 0.82 |
EDVR-m | 156 | 3.3 | 200.0 | 28.11 | 0.80 | 23.54 | 0.69 | 27.27 | 0.75 | 37.42 | 0.93 |
DUF-VSR | 559 | 5.8 | 736.6 | 29.40 | 0.81 | 25.01 | 0.73 | 27.96 | 0.79 | 37.97 | 0.93 |
BasicVSR | 1 306 | 6.3 | 163.7 | 30.21 | 0.85 | 24.94 | 0.74 | 28.52 | 0.82 | 38.20 | 0.94 |
RawVSR | 1 610 | 4.5 | 356.9 | 30.20 | 0.86 | 25.04 | 0.75 | 28.54 | 0.84 | 38.44 | 0.96 |
OVSR | 1 152 | 1.8 | 110.0 | 30.17 | 0.85 | 25.15 | 0.79 | 28.57 | 0.83 | 38.18 | 0.94 |
本文模型 | 1 544 | 6.3 | 336.4 | 30.47 | 0.87 | 25.17 | 0.79 | 28.59 | 0.83 | 38.31 | 0.95 |
模型 | 运行时间/ms | 参数量/106 | 浮点运算量/ GFLOPs | REDS4-000 | REDS4-011 | REDS4-015 | REDS4-020 | ||||
---|---|---|---|---|---|---|---|---|---|---|---|
PSNR/dB | SSIM | PSNR/dB | SSIM | PSNR/dB | SSIM | PSNR/dB | SSIM | ||||
ModelX | 1 129 | 4.6 | 192.2 | 27.72 | 0.80 | 30.13 | 0.85 | 32.16 | 0.88 | 29.18 | 0.85 |
BasicVSR | 1 306 | 6.3 | 163.7 | 27.67 | 0.81 | 30.83 | 0.87 | 32.75 | 0.90 | 29.61 | 0.87 |
本文模型 | 1 544 | 6.3 | 336.4 | 27.83 | 0.82 | 31.13 | 0.88 | 33.12 | 0.91 | 29.82 | 0.88 |
Tab. 2 Performance comparison of different models on REDS4 dataset
模型 | 运行时间/ms | 参数量/106 | 浮点运算量/ GFLOPs | REDS4-000 | REDS4-011 | REDS4-015 | REDS4-020 | ||||
---|---|---|---|---|---|---|---|---|---|---|---|
PSNR/dB | SSIM | PSNR/dB | SSIM | PSNR/dB | SSIM | PSNR/dB | SSIM | ||||
ModelX | 1 129 | 4.6 | 192.2 | 27.72 | 0.80 | 30.13 | 0.85 | 32.16 | 0.88 | 29.18 | 0.85 |
BasicVSR | 1 306 | 6.3 | 163.7 | 27.67 | 0.81 | 30.83 | 0.87 | 32.75 | 0.90 | 29.61 | 0.87 |
本文模型 | 1 544 | 6.3 | 336.4 | 27.83 | 0.82 | 31.13 | 0.88 | 33.12 | 0.91 | 29.82 | 0.88 |
P | R | PSNR/dB | SSIM | 运行时间/ms | 参数量/106 | 浮点运算量/ GFLOPs |
---|---|---|---|---|---|---|
4 | 6 | 30.38 | 0.85 | 1624 | 7.1 | 439.4 |
6 | 6 | 30.26 | 0.83 | 1731 | 7.7 | 484.0 |
3 | 6 | 30.41 | 0.87 | 1558 | 6.8 | 405.8 |
3 | 4 | 30.17 | 0.83 | 1436 | 5.8 | 267.0 |
3 | 5 | 30.47 | 0.87 | 1544 | 6.3 | 336.4 |
Tab. 3 Different numbers of PRDB and RDB on performance on REDS4 dataset
P | R | PSNR/dB | SSIM | 运行时间/ms | 参数量/106 | 浮点运算量/ GFLOPs |
---|---|---|---|---|---|---|
4 | 6 | 30.38 | 0.85 | 1624 | 7.1 | 439.4 |
6 | 6 | 30.26 | 0.83 | 1731 | 7.7 | 484.0 |
3 | 6 | 30.41 | 0.87 | 1558 | 6.8 | 405.8 |
3 | 4 | 30.17 | 0.83 | 1436 | 5.8 | 267.0 |
3 | 5 | 30.47 | 0.87 | 1544 | 6.3 | 336.4 |
模型 | 5帧前向分支 | 3帧后向分支 | 预处理 | 长距离复用 | PSNR/dB | SSIM | 运行时间/ms | 参数量/106 | 浮点运算量/GFLOPs |
---|---|---|---|---|---|---|---|---|---|
Model1 | 29.92 | 0.81 | 1 066 | 3.7 | 125.2 | ||||
Model2 | √ | 30.00 | 0.81 | 1 282 | 4.5 | 183.1 | |||
Model3 | √ | √ | 30.25 | 0.83 | 1 481 | 5.4 | 270.7 | ||
Model4 | √ | √ | √ | 30.33 | 0.84 | 1 543 | 6.3 | 336.4 | |
本文模型 | √ | √ | √ | √ | 30.47 | 0.87 | 1 544 | 6.3 | 336.4 |
Tab. 4 Validation experimental results of different modules on REDS4 dataset
模型 | 5帧前向分支 | 3帧后向分支 | 预处理 | 长距离复用 | PSNR/dB | SSIM | 运行时间/ms | 参数量/106 | 浮点运算量/GFLOPs |
---|---|---|---|---|---|---|---|---|---|
Model1 | 29.92 | 0.81 | 1 066 | 3.7 | 125.2 | ||||
Model2 | √ | 30.00 | 0.81 | 1 282 | 4.5 | 183.1 | |||
Model3 | √ | √ | 30.25 | 0.83 | 1 481 | 5.4 | 270.7 | ||
Model4 | √ | √ | √ | 30.33 | 0.84 | 1 543 | 6.3 | 336.4 | |
本文模型 | √ | √ | √ | √ | 30.47 | 0.87 | 1 544 | 6.3 | 336.4 |
1 | SON S, LEE S, NAH S, et al. NTIRE 2021 challenge on video super-resolution [C]// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2021: 166-181. |
2 | ZHANG L, ZHANG H, SHEN H,et al. A super-resolution reconstruction algorithm for surveillance images[J]. Signal Processing,2010,90(3): 848-859. 10.1016/j.sigpro.2009.09.002 |
3 | SCHULTZ R R, STEVENSON R L. Extraction of high-resolution frames from video sequences[J]. IEEE Transactions on Image Processing, 1996, 5(6): 996-1011. 10.1109/83.503915 |
4 | CHAN K C K, WANG X, YU K, et al. BasicVSR: the search for essential components in video super-resolution and beyond [C]// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 4945-4954. 10.1109/cvpr46437.2021.00491 |
5 | TIAN Y, ZHANG Y, FU Y, et al. TDAN: temporally-deformable alignment network for video super-resolution [C]// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 3360-3369. 10.1109/cvpr42600.2020.00342 |
6 | DAI J, QI H, XIONG Y, et al. Deformable convolutional networks [C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 764-773. 10.1109/iccv.2017.89 |
7 | WANG X, CHAN K C K, YU K, et al. EDVR: video restoration with enhanced deformable convolutional networks [C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2019: 1954-1963. 10.1109/cvprw.2019.00247 |
8 | LI W, TAO X, GUO T,et al. MuCAN: multi-correspondence aggregation network for video super-resolution[C]// Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 335-351. 10.1007/978-3-030-58607-2_20 |
9 | JO Y, OH S W, KANG J, et al. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation [C]// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 3224-3232. 10.1109/cvpr.2018.00340 |
10 | CAO J, LI Y, ZHANG K, et al. Video super-resolution transformer [EB/OL]. [2023-05-01]. . |
11 | ISOBE T, JIA X, GU S, et al. Video super-resolution with recurrent structure-detail network [C]// Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 645-660. 10.1007/978-3-030-58610-2_38 |
12 | ISOBE T, ZHU F, JIA X, et al. Revisiting temporal modeling for video super-resolution [EB/OL]. [2023-05-01]. . |
13 | JIANG L, WANG N, DANG Q, et al. PP-MSVSR: multi-stage video super-resolution [EB/OL]. [2023-05-01]. . |
14 | ISOBE T, JIA X, TAO X, et al. Look back and forth: video super-resolution with explicit temporal difference modeling [C]// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 17411-17420. 10.1109/cvpr52688.2022.01689 |
15 | XIAO Y, YUAN Q, JIANG K, et al. Local-global temporal difference learning for satellite video super-resolution [EB/OL]. [2023-05-01]. . 10.1109/tcsvt.2023.3312321 |
16 | CHAN K C K, ZHOU S, XU X, et al. BasicVSR++: improving video super-resolution with enhanced propagation and alignment [C]// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 5972-5981. 10.1109/cvpr52688.2022.00588 |
17 | CHAN K C K, ZHOU S, XU X, et al. Investigating tradeoffs in real-world video super-resolution [C]// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2021: 5962-5971. 10.1109/cvpr52688.2022.00587 |
18 | LIANG J, CAO J, SUN G, et al. SwinIR: image restoration using swin transformer [C]// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 1833-1844. 10.1109/iccvw54120.2021.00210 |
19 | RANJAN A, BLACK M J. Optical flow estimation using a spatial pyramid network [C] // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 4161-4170. 10.1109/cvpr.2017.291 |
20 | SONG Y, ZHU Y, DU X. Dynamic residual dense network for image denoising [J]. Sensors, 2019, 19(17): 3809. 10.3390/s19173809 |
21 | WANG X, XIE L, DONG C, et al. Real-ESRGAN: training real-world blind super-resolution with pure synthetic data [C] // Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 1905-1914. 10.1109/iccvw54120.2021.00217 |
22 | YAMANAKA J, KUWASHIMA S, KURITA T. Fast and accurate image super resolution by deep CNN with skip connection and network in network [C]// Proceedings of the 24th International Conference Neural Information Processing. Cham: Springer, 2017: 217-225. 10.1007/978-3-319-70096-0_23 |
23 | CHEN Y, WU T. SATVSR: scenario adaptive transformer for cross scenarios video super-resolution [J]. Journal of Physics: Conference Series, 2023, 2456: 012028. 10.1088/1742-6596/2456/1/012028 |
24 | CABALLERO J, LEDIG C, AITKEN A, et al. Real-time video super-resolution with spatio-temporal networks and motion compensation [C] // Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 4778-4787. 10.1109/cvpr.2017.304 |
25 | LAI W-S, HUANG J-B, AHUJA N, et al. Fast and accurate image super-resolution with deep laplacian pyramid networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11): 2599-2613. 10.1109/tpami.2018.2865304 |
26 | NAH S, BAIK S, HONG S, et al. NTIRE 2019 challenge on video deblurring and super-resolution: dataset and study [C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2019: 1996-2005. 10.1109/cvprw.2019.00251 |
27 | HARIS M, SHAKHNAROVICH G, UKITA N. Recurrent back-projection network for video super-resolution [C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 3897-3906. 10.1109/cvpr.2019.00402 |
28 | KINGMA D P, BA J. Adam: a method for stochastic optimization [EB/OL]. [2023-05-01]. . |
29 | LOSHCHILOV I, HUTTER F. SGDR: stochastic gradient descent with warm restarts [EB/OL]. [2023-05-01]. . |
30 | 佟雨兵,张其善,祁云平.基于PSNR与SSIM联合的图像质量评价模型[J].中国图象图形学报,2006, 11(12): 1758-1763. 10.11834/jig.2006012307 |
TONG Y B, ZHANG Q S, QI Y P. Image quality assessing by combining PSNR with SSIM [J]. Journal of Image and Graphics, 2006, 11(12): 1758-1763. 10.11834/jig.2006012307 | |
31 | LIU X, SHI K, WANG Z, et al. Exploit camera raw data for video super-resolution via hidden Markov model inference [J]. IEEE Transactions on Image Processing, 2021, 30: 2127-2140. 10.1109/tip.2021.3049974 |
32 | YI P, WANG Z, JIANG K, et al. Omniscient video super-resolution [C]// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 4429-4438. 10.1109/iccv48922.2021.00439 |
[1] | Yuming ZHAO, Shenkai GU. Adversarial attack defense model with residual dense block self-attention mechanism and generative adversarial network [J]. Journal of Computer Applications, 2022, 42(3): 921-929. |
[2] | LIN Zhixing, WANG Like. Network situation prediction method based on deep feature and Seq2Seq model [J]. Journal of Computer Applications, 2020, 40(8): 2241-2247. |
[3] | LIU Shihao, HU Xuemin, JIANG Bohou, ZHANG Ruohan, KONG Li. Video translation model from virtual to real driving scenes based on generative adversarial dual networks [J]. Journal of Computer Applications, 2020, 40(6): 1621-1626. |
[4] | ZHANG Jiagang, LI Daping, YANG Xiaodong, ZOU Maoyang, WU Xi, HU Jinrong. Deformable medical image registration algorithm based on deep convolution feature optical flow [J]. Journal of Computer Applications, 2020, 40(6): 1799-1805. |
[5] | NIE Kehui, LIU Wenzhe, TONG Tong, DU Min, GAO Qinquan. Video compression artifact removal algorithm based on adaptive separable convolution network [J]. Journal of Computer Applications, 2019, 39(5): 1473-1479. |
[6] | LIAO Bin, WU Wen. Video shadow removal method using region matching guided by illumination transfer [J]. Journal of Computer Applications, 2019, 39(2): 556-563. |
[7] | CHEN Hongyu, DENG Dexiang, YAN Jia, FAN Ci'en. Image retrieval algorithm based on saliency semantic region weighting [J]. Journal of Computer Applications, 2019, 39(1): 136-142. |
[8] | ZHAO Heng, AN Weisheng, FU Weigang. Saliency detection algorithm of deep guidance [J]. Journal of Computer Applications, 2019, 39(1): 143-147. |
[9] | WEI Zhenyu, WEN Chang, XIE Kai, HE Jianbiao. Real-time face detection for mobile devices with optical flow estimation [J]. Journal of Computer Applications, 2018, 38(4): 1146-1150. |
[10] | HU Xuemin, YI Chonghui, CHEN Qin, CHEN Xi, CHEN Long. Abnormal crowd behavior detection based on motion saliency map [J]. Journal of Computer Applications, 2018, 38(4): 1164-1169. |
[11] | ZHU Jie, ZHANG Junsan, WU Shufang, DONG Yukun, LYU Lin. Multi-center convolutional feature weighting based image retrieval [J]. Journal of Computer Applications, 2018, 38(10): 2778-2781. |
[12] | DING Feifei, YANG Wenyuan. Video object segmentation via information entropy constraint [J]. Journal of Computer Applications, 2018, 38(10): 2782-2787. |
[13] | ZHU Mingmin, HU Maohai. Long-term visual object tracking algorithm based on correlation filter [J]. Journal of Computer Applications, 2017, 37(5): 1466-1470. |
[14] | WEI Wei, MA Rui, WANG Xiaofang. Quantitative detection of face location in videos [J]. Journal of Computer Applications, 2017, 37(3): 801-805. |
[15] | LI Dan, BAO Rong, SUN Jinping, XIAO Liqing, DANG Xiangying. Tracking method of multi-resolution LK optical flow combined with SURF [J]. Journal of Computer Applications, 2017, 37(3): 806-810. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||