Video background completion with complexly moving camera

Abstract

Abstract:

Video background completion is attracting more and more attention. For videos captured by complexly moving camera, the problem is even more challenging. In order to solve the problem, a motion-guided optimizing algorithm was proposed to complete the spatio-temporal hole left by foreground object removal. First, to estimate the motion field in the hole, a global objective function was established, and a hierarchical iterative method was applied to find its optimal solution. Completion problem was then abstracted into a Markov Random Field (MRF) problem. Using motion field as the guidance, video background was completed by optimally assigning available pixels from other parts to the missing regions. Finally, traditional illumination transfer strategy was improved, and a new illumination adjusting method was proposed to eliminate the illumination inconsistency in the completed parts. This approach got good results on a variety of videos. Compared with previous methods, this approach works better in keeping spatio-temporal coherence, and can be applied on videos with complex background captured by complexly moving camera.

摘要：

视频背景修复问题正受到越来越多的关注,对于复杂运动的摄像机所拍摄的视频而言,该问题具有更高的难度。针对此问题,提出一种由运动场引导的优化算法,填补由于去掉前景物体所留下的视频体空洞。首先,为了估计视频空洞部分的运动场,构建全局目标方程并利用分层次迭代的方法求得其最优解；修复问题继而被抽象为马尔可夫随机场问题。将运动场作为引导,最优地从已知区域选择可用的像素修复视频的背景。最后,改进传统的光照迁移方法,提出一种亮度调整策略,消除修复区域光照不连续的现象。该算法在多种不同类型的视频上均取得良好的效果。与现有算法相比,该算法能更好地保证时空连续性,并能修复由复杂运动的摄像机所拍摄的、含有复杂背景的视频。

CLC Number:

TP391.41

XU Zhan CAO Zhe. Video background completion with complexly moving camera[J]. Journal of Computer Applications, 2014, 34(12): 3540-3544.

徐展曹哲. 复杂运动摄像机拍摄视频的背景修复技术[J]. 计算机应用, 2014, 34(12): 3540-3544.

References

[1] WEXLER Y, SHECHTMAN E, IRANI M. Space-time completion of video[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(3): 463-476.
[2] JIA Y T, HU S M, MARTIN R R. Video completion using tracking and fragment merging[J]. The Visual Computer, 2005, 21(8/9/10): 601-610.
[3] SHIRATORI T, MATSUSHITA Y, TANG X, et al. Video completion by motion field transfer[C]// Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2006, 1: 411-418.
[4] GRANADOS M, TOMPKIN J, KIM K, et al. How not to be seen-object removal from videos of crowded scenes[C]// Eurographics 2012: Computer Graphics Forum. Oxford: Blackwell Publishing, 2012: 219-228.
[5] GU Y, HAN J. Video restoration using exemplar-based inpainting[J]. Journal of Applied Sciences, 2010, 28(2): 163-169.(谷伊,韩军. 基于样本的图像修补方法在视频修复中的应用［J］.应用科学学报, 2010,28(2): 163-169.）
[6] XIAO C, LIU S, LIN C, et al. A global space-time optimization framework for video completion[J]. Journal of Computer-Aided Design and Computer Graphics, 2008, 20(9): 1204-1211.(肖春霞,刘舒,林成春,等. 基于时空全局优化的视频修复[J]. 计算机辅助设计与图形学学报, 2008,20(9): 1204-1211.)
[7] PATWARDHAN K A, SAPIRO G, BERTALMO M. Video inpainting under constrained camera motion[J]. IEEE Transactions on Image Processing, 2007, 16(2): 545-553.
[8] SHEN Y, LU F, CAO X, et al. Video completion for perspective camera under constrained motion[C]// Proceedings of the 18th International Conference on Pattern Recognition. Piscataway: IEEE, 2006, 3: 63-66.
[9] LIANG M, WANG Z. Video inpainting of camera parallel to scene and shaked[J]. Computer Engineering and Applications, 2011, 47(4): 176-180.(梁敏,王兆仲. 摄像机平行于场景运动且晃动下的视频修复[J]. 计算机工程与应用, 2011, 47(4): 176-180.)
[10] GRANADOS M, KIM K I, TOMPKIN J, et al. Background inpainting for videos with dynamic objects and a free-moving camera[M]. Heidelberg: Springer, 2012: 682-695.
[11] AGARWALA A, HERTZMANN A, SALESIN D H, et al. Keyframe-based tracking for rotoscoping and animation[J].ACM Transactions on Graphics, 2004, 23(3): 584-591.
[12] LIU C, FREEMAN W T, ADELSON E H, et al. Human-assisted motion annotation[C]// Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2008: 1-8.
[13] KWATRA V, SCHDL A, ESSA I, et al. GraphCut textures: image and video synthesis using graph cuts[J]. ACM Transactions on Graphics, 2003, 22(3): 277-286.
[14] BOYKOV Y, VEKSLER O, ZABIH R. Fast approximate energy minimization via graph cuts[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(11): 1222-1239.
[15] XIAO C, XIAO D, ZHANG L, et al. Efficient shadow removal using subregion matching illumination transfer[J]. Computer Graphics Forum, 2013, 32(7): 421-430.
[16] MANJUNATH B S, MA W Y. Texture features for browsing and retrieval of image data[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996, 18(8): 837-842.
[17] REINHARD E, ADHIKHMIN M, GOOCH B, et al. Color transfer between images[J]. IEEE Computer Graphics and Applications, 2001, 21(5): 34-41.