基于深度体素流的模糊视频插帧方法

doi:10.11772/j.issn.1001-9081.2019081474

计算机应用 ›› 2020, Vol. 40 ›› Issue (3): 819-824.DOI: 10.11772/j.issn.1001-9081.2019081474

• 虚拟现实与多媒体计算 • 上一篇下一篇

基于深度体素流的模糊视频插帧方法

林传健^1,2, 邓炜³, 童同³, 高钦泉^1,2,3

1. 福州大学物理与信息工程学院, 福州 350116;
2. 福建省医疗器械与医药技术重点实验室(福州大学), 福州 350116;
3. 福建帝视信息科技有限公司, 福州 350116

收稿日期:2019-08-29 修回日期:2019-10-23 出版日期:2020-03-10 发布日期:2019-11-06
通讯作者: 高钦泉
作者简介:林传健(1994-),男,福建宁德人,硕士研究生,主要研究方向:计算机视觉、图像处理;邓炜(1992-),男,福建龙岩人,硕士,主要研究方向:计算机视觉、图像处理;童同(1986-),男,安徽安庆人,研究员,博士,主要研究方向:人工智能与计算机视觉、医学图像处理与分析;高钦泉(1986-),男,福建福清人,副研究员,博士,主要研究方向:人工智能与计算机视觉、医学图像处理与分析、计算机辅助手术导航。
基金资助:
国家自然科学基金资助项目（61802065）。

Blurred video frame interpolation method based on deep voxel flow

LIN Chuanjian^1,2, DENG Wei³, TONG Tong³, GAO Qinquan^1,2,3

1. College of Physics and Information Engineering, Fuzhou University, Fuzhou Fujian 350116, China;
2. Key Lab of Medical Instrumentation and Pharmaceutical Technology of Fujian Province(Fuzhou University), Fuzhou Fujian 350116, China;
3. Imperial Vision Technology Company Limited, Fuzhou Fujian 350116, China

Received:2019-08-29 Revised:2019-10-23 Online:2020-03-10 Published:2019-11-06
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61802065).

摘要/Abstract

摘要： 针对视频运动模糊严重影响插帧效果的情况，提出了一种新型的模糊视频插帧方法。首先，提出一种多任务融合卷积神经网络，该网络结构由两个模块组成：去模糊模块和插帧模块。其中，去模糊模块采用残差块堆叠的深度卷积神经网络（CNN），提取并学习深度模糊特征以实现两帧输入图像的运动模糊去除；插帧模块用于估计帧间的体素流，所得体素流将用于指导像素进行三线性插值以合成中间帧。其次，制作了大型模糊视频仿真数据集，并提出一种先分后合、由粗略至细致的训练策略，实验结果表明该策略促进了多任务网络有效收敛。最后，对比前沿的去模糊和插帧算法组合，实验指标显示所提方法合成中间帧时峰值信噪比最少提高1.41 dB，结构相似性提升0.020，插值误差降低1.99。视觉对比及重制序列展示表明，所提模型对于模糊视频有着显著的帧率上转换效果，即能够将两帧模糊视频帧端对端重制为清晰且视觉连贯的三帧视频帧。

关键词: 深度学习, 卷积神经网络, 去模糊, 体素流, 视频插帧

Abstract: Motion blur has an extremely negative effect on video frame interpolation. In order to handle this problem, a novel blurred video frame interpolation method was proposed. Firstly, a multi-task fusion convolutional neural network was proposed, which consists of a deblurring module and a frame interpolation module. In the deblurring module, based on the deep Convolutional Neural Network (CNN) with stack of ResBlocks, motion blur removal of two input frames was implemented by extracting and learning the deep blur features. And the frame interpolation module was used to estimate voxel flow between two consecutive frames after blur removal, then the obtained voxel flow was used to guide the trilinear interpolation of the pixels to synthesize the intermediate frame. Secondly, a large blurred video simulation dataset was made, and a “first separate and then combine” “from coarse to fine” training strategy was proposed, experimental results show that this strategy promotes the effective convergence of the multi-task fusion network. Finally, compared with the simple combination of the state-of-the-art deblurring and frame interpolation algorithms, experimental metrics show that the intermediate frame synthesized by the proposed method has the peak-to-noise ratio increased by 1.41 dB, the structural similarity improved by 0.020, and the interpolation error decreased by 1.99， at least. Visual comparison and reconstructed sequences show that the proposed model performs good frame rate up conversion effect for blurred videos, in other words, two blurred consecutive frames can be reconstructed end-to-end to three sharp and visually smooth frames by the model.

Key words: deep learning, Convolutional Neural Network (CNN), debluring, voxel flow, video frame interpolation

中图分类号:

TP391
TP183

林传健, 邓炜, 童同, 高钦泉. 基于深度体素流的模糊视频插帧方法[J]. 计算机应用, 2020, 40(3): 819-824.

LIN Chuanjian, DENG Wei, TONG Tong, GAO Qinquan. Blurred video frame interpolation method based on deep voxel flow[J]. Journal of Computer Applications, 2020, 40(3): 819-824.

参考文献

[1] DOSOVITSKIY A,FISCHER P,ILG E,et al. FlowNet:learning optical flow with convolutional networks[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE,2015:2758-2766.
[2] ILG E,MAYER N,SAIKIA T,et al. FlowNet 2.0:evolution of optical flow estimation with deep networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:1647-1655.
[3] NIKLAUS S,MAI L,LIU F. Video frame interpolation via adaptive convolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:2270-2279.
[4] NIKLAUS S,MAI L,LIU F. Video frame interpolation via adaptive separable convolution[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:261-270.
[5] LIU Z,YEH R A,TANG X,et al. Video frame synthesis using deep voxel flow[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:4473-4481.
[6] JIANG H,SUN D,JAMPANI V,et al. Super SloMo:high quality estimation of multiple intermediate frames for video interpolation[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:9000-9008.
[7] NIKLAUS S,LIU F. Context-aware synthesis for video frame interpolation[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:1701-1710.
[8] LIU Y L,LIAO Y T,LIN Y Y,et al. Deep video frame interpolation using cyclic frame generation[C]//Proceedings of the 2019 AAAI Conference on Artificial Intelligence. Palo Alto,CA:AAAI Press,2019:8794-8802.
[9] BAO W,LAI W-S,MA C,et al. Depth-aware video frame interpolation[EB/OL].[2019-05-23]. https://arxiv.org/pdf/1904.00830.pdf.
[10] WANG R,TAO D. Recent progress in image deblurring[EB/OL].[2019-05-23]. https://arxiv.org/pdf/1409.6838.pdf.
[11] KUPYN O,BUDZAN V,MYKHAILYCH M,et al. DeblurGAN:blind motion deblurring using conditional adversarial networks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:8183-8192.
[12] NAH S,KIM T H,MU LEE K. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:3883-3891.
[13] TAO X,GAO H,SHEN X,et al. Scale-recurrent network for deep image deblurring[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:8174-8182.
[14] RONNEBERGER O,FISCHER P,BROX T. U-Net:convolutional networks for biomedical image segmentation[C]//Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention,LNCS 9351. Cham:Springer,2015:234-241.
[15] TAO X, GAO H, LIAO R, et al. Detail-revealing deep video superresolution[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2017:4482-4490.
[16] PATHAK D,KRÄHENBÜHL P,DONAHUE J,et al. Context encoders:feature learning by inpainting[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:2536-2544.
[17] SIMO-SERRA E,IIZUKA S,SASAKI K,et al. Learning to simplify:fully convolutional networks for rough sketch cleanup[J]. ACM Transactions on Graphics,2016,35(4):No. 121.
[18] HE K,ZHANG X,REN S,et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:770-778.
[19] SU S,DELBRACIO M,WANG J,et al. Deep video deblurring for hand-held cameras[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:237-246.
[20] SOOMRO K,ZAMIR A R,SHAH M. UCF101:a dataset of 101 human actions classes from videos in the wild[EB/OL].[2019-05-23]. https://arxiv.org/pdf/1212.0402.pdf.
[21] KINGMA D P,BA J L. Adam:a method for stochastic optimization[EB/OL].[2019-05-23]. https://arxiv.org/pdf/1412.6980.pdf.
[22] BAKER S,SCHARSTEIN D,LEWIS J P,et al. A database and evaluation methodology for optical flow[J]. International Journal of Computer Vision,2011,92(1):1-31.
[23] KIM T,CHA M,KIM H,et al. Learning to discover cross-domain relations with generative adversarial networks[C]//Proceedings of the 34th International Conference on Machine Learning. New York:JMLR.org,2017:1857-1865.

基于深度体素流的模糊视频插帧方法

Blurred video frame interpolation method based on deep voxel flow

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	谢德峰, 吉建民. 融入句法感知表示进行句法增强的语义解析[J]. 计算机应用, 2021, 41(9): 2489-2495.
[2]	代雨柔, 杨庆, 张凤荔, 周帆. 基于自监督学习的社交网络用户轨迹预测模型[J]. 计算机应用, 2021, 41(9): 2545-2551.
[3]	陈成瑞, 孙宁, 何世彪, 廖勇. 面向C-V2X通信的基于深度学习的联合信道估计与均衡算法[J]. 计算机应用, 2021, 41(9): 2687-2693.
[4]	宋中山, 梁家锐, 郑禄, 刘振宇, 帖军. 基于双向门控尺度特征融合的遥感场景分类[J]. 计算机应用, 2021, 41(9): 2726-2735.
[5]	李康康, 张静. 基于注意力机制的多层次编码和解码的图像描述模型[J]. 计算机应用, 2021, 41(9): 2504-2509.
[6]	张永斌, 常文欣, 孙连山, 张航. 基于字典的域名生成算法生成域名的检测方法[J]. 计算机应用, 2021, 41(9): 2609-2614.
[7]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[8]	徐江浪, 李林燕, 万新军, 胡伏原. 结合目标检测的室内场景识别方法[J]. 计算机应用, 2021, 41(9): 2720-2725.
[9]	牟长宁, 王海鹏, 周丕宇, 侯鑫行. 基于图卷积神经网络的串联质谱从头测序[J]. 计算机应用, 2021, 41(9): 2773-2779.
[10]	王贺兵, 张春梅. 基于非对称卷积-压缩激发-次代残差网络的人脸关键点检测[J]. 计算机应用, 2021, 41(9): 2741-2747.
[11]	郑志强, 胡鑫, 翁智, 王雨禾, 程曦. 基于改进DenseNet的牛眼图像特征提取方法[J]. 计算机应用, 2021, 41(9): 2780-2784.
[12]	曾祥银, 郑伯川, 刘丹. 基于深度卷积神经网络和聚类的左右轨道线检测[J]. 计算机应用, 2021, 41(8): 2324-2329.
[13]	曹玉红, 徐海, 刘荪傲, 王紫霄, 李宏亮. 基于深度学习的医学影像分割研究综述[J]. 计算机应用, 2021, 41(8): 2273-2287.
[14]	秦斌斌, 彭良康, 卢向明, 钱江波. 司机分心驾驶检测研究进展[J]. 计算机应用, 2021, 41(8): 2330-2337.
[15]	黄程程, 董霄霄, 李钊. 基于二维Winograd算法的深流水线5×5卷积方法[J]. 计算机应用, 2021, 41(8): 2258-2264.