High-accuracy video image stabilization algorithm incorporating temporal and spatial saliency

doi:10.11772/j.issn.1001-9081.2021061061

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (8): 2564-2570.DOI: 10.11772/j.issn.1001-9081.2021061061

• Multimedia computing and computer simulation • Previous Articles Next Articles

High-accuracy video image stabilization algorithm incorporating temporal and spatial saliency

Lihua YIN(), Liang KANG, Wenhua ZHU

Engineering Training Center，Shanghai Polytechnic University，Shanghai 201209，China

Received:2021-06-24 Revised:2022-01-27 Accepted:2022-02-09 Online:2022-03-16 Published:2022-08-10
Contact: Lihua YIN
About author:YIN Lihua， born in 1988， Ph. D.， lecturer. Her research interests include computer vision， pattern recognition.
KANG Liang， born in 1980， Ph. D. associate professor. His research interests include digital image processing， machine learning.
ZHU Wenhua， born in 1968， Ph. D. professor. His research interests include virtual simulation， optimization algorithm.
Supported by:
Industry-University-Research Innovation Foundation of Science and Technology Development Center of Ministry of Education(2018C01059);National Natural Pre-Research Project of Shanghai Polytechnic University(EGD21QD16)

融入时空显著性的高精度视频稳像算法

尹丽华(), 康亮, 朱文华

上海第二工业大学工程训练中心，上海 201209

通讯作者: 尹丽华
作者简介:尹丽华（1988—），女，山东省泰安人，讲师，博士，主要研究方向：计算机视觉、模式识别；
康亮（1980—），男，黑龙江绥滨人，副教授，博士，主要研究方向：数字图像处理、机器学习；
朱文华（1968—），男，江苏无锡人，教授，博士，主要研究方向：虚拟仿真、优化算法。
基金资助:
教育部科技发展中心产学研创新基金资助项目(2018C01059);上海第二工业大学国家自然科学基金预研项目(EGD21QD16)

Abstract

Abstract:

In order to eliminate the interference of complicated moving foreground on the accuracy of video stabilization， and to combine with the unique advantages of temporal and spatial saliency in moving target detection， a high-accuracy video stabilization algorithm incorporating temporal and spatial saliency was proposed. In the proposed algorithm， the spatio-temporal saliency detection technology was used to identify and eliminate the moving targets. At the same time， the multi-grid motion paths were adopted for motion compensation. The proposed algorithm specifically includes： Speeded Up Robust Features （SURF） feature point extraction and matching， spatio-temporal saliency target detection， grid division and motion vector calculation， motion trajectory generation， multi-path smoothing， motion compensation and so on. Experimental results show that compared with traditional image stabilization algorithms， the proposed algorithm has outstanding performance in stability index. For videos with moving large range of foreground interference，the stability of the proposed algorithm is improved by about 9.6% compared with Robust Traffic Video Stabilization Method assisted by foreground feature trajectories （RTVSM）. For videos with multiple moving foreground interference，the stability of the proposed algorithm is improved by about 5.8% compared with Bundled-paths algorithm， which fully verify the proposed algorithm’s image stability advantage.

Key words: video image stabilization, temporal and spatial saliency, high-accuracy video, moving foreground, grid

摘要：

为剔除复杂运动前景对视频稳像精度的干扰，同时结合时空显著性在运动目标检测上的独特优势，提出一种融入时空显著性的高精度视频稳像算法。该算法一方面通过时空显著性检测技术识别出运动目标并对其进行剔除；另一方面，采用多网格的运动路径进行运动补偿。具体包括：SURF特征点提取和匹配、时空显著性目标检测、网格划分与运动矢量计算、运动轨迹生成、多路径平滑、运动补偿等环节。实验结果表明，相较于传统的稳像算法，所提算法在稳定度（Stability）指标方面表现突出。对于有大范围运动前景干扰的视频，所提算法比RTVSM（Robust Traffic Video Stabilization Method assisted by foreground feature trajectories）的Stability指标提高了约9.6%；对于有多运动前景干扰的视频，所提算法比Bundled-paths算法的Stability指标提高了约5.8%，充分说明了所提算法对于复杂场景的稳像优势。

关键词: 视频稳像, 时空显著性, 高精度视频, 运动前景, 网格

CLC Number:

TP391.41

Lihua YIN, Liang KANG, Wenhua ZHU. High-accuracy video image stabilization algorithm incorporating temporal and spatial saliency[J]. Journal of Computer Applications, 2022, 42(8): 2564-2570.

尹丽华, 康亮, 朱文华. 融入时空显著性的高精度视频稳像算法[J]. 《计算机应用》唯一官方网站, 2022, 42(8): 2564-2570.

Figures/Tables 9

Fig. 1 Example of temporal saliency map

Fig. 2 Example of spatial saliency map

Fig. 3 Motion trajectory generation

Fig. 4 Moving target detection results based on spatio-temporal saliency maps

Tab. 1 Comparison of moving target recognition accuracy with three saliency maps

显著图类型	Video1	Video2
时空显著图	95.13	94.25
时间显著图	88.46	87.92
空间显著图	80.97	81.84

Tab. 2 Moving target recognition results on different datasets

数据集	结果占比
数据集	满意	可接受	失败
Testing Set1（Video1）	0.82	0.14	0.04
Testing Set1（Video2）	0.74	0.13	0.13
Testing Set2（Video3）	0.80	0.15	0.05
Testing Set2（Video4）	0.81	0.16	0.03

Fig. 5 Image stabilization test videos

Fig. 6 Comparison of image stability accuracy of different algorithms in three videos

Fig. 7 Comparison of visual image stabilization results of different algorithms in three videos

References 18

1	GRUNDMANN M， KWATRA V， ESSA I， et al. Auto-directed video stabilization with robust L1 optimal camera paths ［C］// Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2011： 225-232. 10.1109/cvpr.2011.5995525
2	MATSUSHITA Y， OFEK E， GE W N， et al. Full-frame video stabilization with motion inpainting［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2006， 28（7）： 1150-1163. 10.1109/tpami.2006.141
3	尹丽华，李范鸣，刘士建.基于区域分割与融合的全景稳像算法［J］.红外与激光工程， 2018， 47（9）： No.926004. 10.3788/IRLA201847.0926004
	YIN L H， LI F M， LIU S J. Panoramic image stabilization algorithm based on region segmentation and fusion［J］. Infrared and Laser Engineering， 2018， 47（9）： No.926004. 10.3788/IRLA201847.0926004
4	佘建国，徐仁桐，陈宁.基于ORB和改进RANSAC算法的图像拼接技术［J］.江苏科技大学学报（自然科学版）， 2015， 29（2）： 164-169. 10.3969/j.issn.1673-4807.2015.02.012
	SHE J G， XU R T， CHEN N. Image stitching technology based on ORB and improved RANSAC algorithm［J］. Journal of Jiangsu University of Science and Technology （Natural Science Edition）， 2015， 29（2）： 164-169. 10.3969/j.issn.1673-4807.2015.02.012
5	邱家涛.电子稳像算法和视觉跟踪算法研究［D］.西安：西安电子科技大学， 2013： 15-54. 10.7666/d.Y2380385
	QIU J T. A study of electronic image stabilization algorithms and visual tracking algorithms［D］. Xi’an： Xidian University， 2013： 15-54. 10.7666/d.Y2380385
6	朱娟娟，范静，郭宝龙.抗前景干扰的自适应电子稳像算法［J］.光子学报， 2015， 44（6）： No.610002. 10.3788/gzxb20154406.0610002
	ZHU J J， FAN J， GUO B L. Adaptive electronic image stabilization algorithm resistant to foreground moving object［J］. Acta Photonica Sinica， 2015， 44（6）： No.610002. 10.3788/gzxb20154406.0610002
7	LIU S C， YUAN L， TAN P， et al. Bundled camera paths for video stabilization［J］. ACM Transactions on Graphics， 2013， 32（4）： No.78. 10.1145/2461912.2461995
8	谢亚晋，徐之海，冯华君，等.基于最小生成树与改进卡尔曼滤波器的实时电子稳像方法［J］.光子学报， 2018， 47（1）： No.110002. 10.3788/gzxb20184701.0110002
	XIE Y J， XU Z H， FENG H J， et al. Real-time video stabilization based on minimal spanning tree and modified Kalman filter［J］. Acta Photonica Sinica， 2018， 47（1）： No.110002. 10.3788/gzxb20184701.0110002
9	杨佳丽，来林静，张磊，等.李群流形上的在线视频稳像算法［J］.模式识别与人工智能， 2019， 32（4）： 298-305.
	YANG J L， LAI L J， ZHANG L， et al. Online video stabilization algorithm on Lie group manifold［J］. Pattern Recognition and Artificial Intelligence， 2019， 32（4）： 298-305.
10	ZHAO M D， LING Q. A robust traffic video stabilization method assisted by foreground feature trajectories［J］. IEEE Access， 2019， 7： 42921-42933. 10.1109/access.2019.2908186
11	WANG M， YANG G Y， LIN J K， et al. Deep online video stabilization with multi-grid warping transformation learning［J］. IEEE Transactions on Image Processing， 2019， 28（5）： 2283-2292. 10.1109/tip.2018.2884280
12	徐金婧，徐熙平.基于时空显著特征的运动目标检测方法研究［J］.长春理工大学学报（自然科学版）， 2020， 43（4）： 43-48.
	XU J J， XU X P. Research on moving target detection method based on time and space saliency features［J］. Journal of Changchun University of Science and Technology （Natural Science Edition）， 2020， 43（4）： 43-48.
13	张冬明，靳国庆，代锋，等.基于深度融合的显著性目标检测算法［J］.计算机学报， 2019， 42（9）： 2076-2086. 10.11897/SP.J.1016.2019.02076
	ZHANG D M， JIN G Q， DAI F， et al. Salient object detection based on deep fusion of hand-crafted features［J］. Chinese Journal of Computers， 2019， 42（9）： 2076-2086. 10.11897/SP.J.1016.2019.02076
14	冯思琴.基于时空特征的视觉显著性检测方法研究［D］.重庆：重庆邮电大学， 2019： 6-30.
	FENG S Q. Research on visual saliency detection based on spatio-temporal features［D］. Chongqing： Chongqing University of Posts and Telecommunications， 2019： 6-30
15	ZHAI Y， SHAH M. Visual attention detection in video sequences using spatiotemporal cues ［C］// Proceedings of the 14th ACM International Conference on Multimedia. New York： ACM， 2006： 815-824. 10.1145/1180639.1180824
16	黄晨，王建军，高昕，等.电子稳像中稳像质量评价方法研究［J］.激光与红外， 2013， 43（5）： 477-481. 10.3969/j.issn.1001-5078.2013.05.001
	HUANG C， WANG J J， GAO X， et al. Study on video stabilization performance assessment in electronic image stabilization［J］. Laser and Infrared， 2013， 43（5）： 477-481. 10.3969/j.issn.1001-5078.2013.05.001
17	LIU F， GLEICHER M， WANG J， et al. Subspace video stabilization［J］. ACM Transactions on Graphics， 2011， 30（1）： No.4. 10.1145/1899404.1899408
18	HUYNH L， CHOI J， MEDIONI G. Aerial implicit 3D video stabilization using epipolar geometry constraint ［C］// Proceedings of the 22nd International Conference on Pattern Recognition. Piscataway： IEEE， 2014： 3487-3492. 10.1109/icpr.2014.600

[1]	An YANG, Qun JIANG, Gang SUN, Jie YIN, Ying LIU. Power data analysis based on financial technical indicators [J]. Journal of Computer Applications, 2022, 42(3): 904-910.
[2]	ZENG Xiangyin, ZHENG Bochuan, LIU Dan. Detection of left and right railway tracks based on deep convolutional neural network and clustering [J]. Journal of Computer Applications, 2021, 41(8): 2324-2329.
[3]	SHANG Fangjian, LI Xin, Di ZHAI, LU Yang, ZHANG Donglei, QIAN Yuwen. Two-phase resource allocation technology for network slices in smart grid [J]. Journal of Computer Applications, 2021, 41(7): 2033-2038.
[4]	ZHANG Lihua, WANG Xinyi, HU Fangzhou, HUANG Yang, BAI Jiayi. Data sharing model of smart grid based on double consortium blockchains [J]. Journal of Computer Applications, 2021, 41(4): 963-969.
[5]	LONG Chaoqi, JIANG Yu, XIE Yu. Improved wavelet clustering algorithm based on peak grid [J]. Journal of Computer Applications, 2021, 41(4): 1122-1127.
[6]	Erchao LI, Kuankuan QI. Robot path planning based on B-spline curve and ant colony algorithm [J]. Journal of Computer Applications, 2021, 41(12): 3558-3564.
[7]	HUANG Zhongzhan, XU Shiming. Automatic generation algorithm of orthogonal grid based on recurrent neural network [J]. Journal of Computer Applications, 2020, 40(7): 2009-2015.
[8]	DING Hui, LI Lihong, YUAN Gang. Image registration algorithm combining GMS and VCS+GC-RANSAC [J]. Journal of Computer Applications, 2020, 40(4): 1138-1143.
[9]	CHENG Qiwei, CHEN Qimai, HE Chaobo, LIU Hai. Overlapping community detection method based on improved symmetric binary nonnegative matrix factorization [J]. Journal of Computer Applications, 2020, 40(11): 3203-3210.
[10]	MU Qi, TANG Yang, LI Zhanli, LI Hong'an. Fast stitching method for dense repetitive structure images based on grid-based motion statistics algorithm and optimal seam [J]. Journal of Computer Applications, 2020, 40(1): 239-244.
[11]	ZHAO Ruixiang, HOU Honghua, ZHANG Pengcheng, LIU Yi, TIAN Zhu, GUI Zhiguo. Welding ball edge bubble segmentation for ball grid array based on full convolutional network and K-means clustering [J]. Journal of Computer Applications, 2019, 39(9): 2580-2585.
[12]	GAO Jian, MAO Yingchi, LI Zhitao. Trajectory prediction based on Gauss mixture time series model [J]. Journal of Computer Applications, 2019, 39(8): 2261-2270.
[13]	ZHU Chengde, LI Zhiwei, WANG Kai, GAO Yan, GUO Hengchang. Image matching algorithm based on improved RANSAC-GMS [J]. Journal of Computer Applications, 2019, 39(8): 2396-2401.
[14]	TAN Xin, LI Xiaohui, LIU Zhenxing, DING Yuemin, ZHAO Min, WANG Qi. Sampling awareness weighted round robin scheduling algorithm in power grid [J]. Journal of Computer Applications, 2019, 39(7): 2061-2064.
[15]	HUO Zheng, ZHANG Kun, HE Ping, WU Yanbin. Crowdsourcing location data collection for local differential privacy [J]. Journal of Computer Applications, 2019, 39(3): 763-768.

High-accuracy video image stabilization algorithm incorporating temporal and spatial saliency

融入时空显著性的高精度视频稳像算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 18

Related Articles 15

Recommended Articles

Metrics