Object tracking based on instance segmentation and Pythagorean fuzzy decision-making

doi:10.11772/j.issn.1001-9081.2022050674

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (6): 1930-1937.DOI: 10.11772/j.issn.1001-9081.2022050674

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Object tracking based on instance segmentation and Pythagorean fuzzy decision-making

Yuanlong ZHAO¹, Yugang SHAN²(), Jie YUAN¹, Kangdi ZHAO¹

^1.School of Electrical Engineering，Xinjiang University，Urumqi Xinjiang 830017，China
^2.School of Education，Hubei University of Arts and Science，Xiangyang Hubei 441053，China

Received:2022-05-11 Revised:2023-02-15 Accepted:2023-02-15 Online:2023-06-08 Published:2023-06-10
Contact: Yugang SHAN
About author:ZHAO Yuanlong， born in 1994， M. S. candidate. His research interests include object tracking， object detection.
YUAN Jie， born in 1975， Ph. D.， associate professor. His research interests include robot control.
ZHAO Kangdi， born in 1995， M. S. candidate. His research interests include object detection.
Supported by:
National Natural Science Foundation of China(62263031);Natural Science Foundation of Xinjiang Uygur Autonomous Region(2022D01C53);Education Science Planning Foundation of Hubei(2021GA048);Industry-University Cooperative Education Project of Ministry of Education(202102602033);Science and Technology Program of Xiangyang （High-Tech Field）(2020ABH001799);“Double Hundred Action Plan” Project of Hubei University of Arts and Science(PYSB20202016);2021 Teachers’ Research Ability Training Fund of Hubei University of Arts and Science(2021KPGPSK08)

基于实例分割与毕达哥拉斯模糊决策的目标跟踪

赵元龙¹, 单玉刚²(), 袁杰¹, 赵康迪¹

^1.新疆大学电气工程学院，乌鲁木齐 830017
^2.湖北文理学院教育学院，湖北襄阳 441053

通讯作者: 单玉刚
作者简介:赵元龙（1994—），男，湖北襄阳人，硕士研究生，主要研究方向：目标跟踪、目标检测
单玉刚（1971—），男，辽宁沈阳人，讲师，博士，主要研究方向：模式识别Email：shanyg@21cn.com
袁杰（1975—），男，新疆乌鲁木齐人，副教授，博士，主要研究方向：机器人控制
赵康迪（1995—），男，河南南阳人，硕士研究生，主要研究方向：目标检测。
基金资助:
国家自然科学基金资助项目(62263031);新疆维吾尔自治区自然科学基金资助项目(2022D01C53);湖北省教育科学规划基金资助项目(2021GA048);教育部产学合作协同育人项目(202102602033);襄阳市科技计划项目（高新领域）(2020ABH001799);湖北文理学院“双百行动计划”项目(PYSB20202016);湖北文理学院2021年度教师科研能力培育基金资助项目(2021KPGPSK08)

Abstract

Abstract:

In order to solve the problems of scale change， similarity interference and occlusion in object tracking， an object tracking algorithm based on instance segmentation and Pythagorean fuzzy decision-making was proposed. Based on the instance segmentation network YOLACT++ （improved You Only Look At CoefficienTs）， three different matching methods were integrated to predict the tracking results for different scenes. At the same time， a template update mechanism based on Pythagorean fuzzy decision-making was proposed by which whether to update the object template and replace the matching method was determined according to the quality of the prediction results. Experimental results show that the proposed algorithm can track the video sequences with scale change， similarity interference， occlusion and other problems more accurately. Compared with SiamMask algorithm， the proposed algorithm has the regional similarity on DAVIS 2016 and DAVIS 2017 datasets increased by 12.3 and 15.3 percentage points， respectively， and the Expected Average Overlap rate （EAO） on VOT2016 and VOT2018 datasets increased by 4.2 and 4.1 percentage points， respectively. Meanwhile， the average tracking speed of the proposed algorithm is 32.00 frames per second， meeting real-time requirements.

Key words: image processing, object tracking, Pythagorean fuzzy decision-making, instance segmentation, Siamese network

摘要：

为了解决目标跟踪中的尺度变化、相似性干扰、遮挡等问题，提出一种基于实例分割与毕达哥拉斯模糊决策的目标跟踪算法。在实例分割网络YOLACT++ （improved You Only Look At CoefficienTs）的基础上，融合3种不同的匹配方式针对不同场景预测跟踪结果；同时提出一种基于毕达哥拉斯模糊决策的模板更新机制，即根据预测结果的质量作出是否更新目标模板和更换匹配方式的决定。实验结果表明，所提算法能够更准确地跟踪存在尺度变化、相似性干扰、遮挡等问题的视频序列。相较于SiamMask算法，所提算法在DAVIS 2016、DAVIS 2017数据集上的区域相似度分别提高了12.3、15.3个百分点，在VOT2016、VOT2018数据集上的预期平均重叠率（EAO）分别提高了4.2、4.1个百分点，且所提算法的平均跟踪速度为每秒32.00帧，满足实时性要求。

关键词: 图像处理, 目标跟踪, 毕达哥拉斯模糊决策, 实例分割, 孪生网络

CLC Number:

TP391.4

Yuanlong ZHAO, Yugang SHAN, Jie YUAN, Kangdi ZHAO. Object tracking based on instance segmentation and Pythagorean fuzzy decision-making[J]. Journal of Computer Applications, 2023, 43(6): 1930-1937.

赵元龙, 单玉刚, 袁杰, 赵康迪. 基于实例分割与毕达哥拉斯模糊决策的目标跟踪[J]. 《计算机应用》唯一官方网站, 2023, 43(6): 1930-1937.

Figures/Tables 11

References 31

1	李玺，查宇飞，张天柱，等. 深度学习的目标跟踪算法综述［J］. 中国图象图形学报， 2019， 24（12）： 2057-2080. 10.11834/jig.190372
	LI X， ZHA Y F， ZHANG T Z， et al. Survey of visual object tracking algorithms based on deep learning ［J］. Journal of Image and Graphics， 2019， 24（12）： 2057-2080. 10.11834/jig.190372
2	DANELLJAN M， HAGER G， KHAN F S， et al. Convolutional features for correlation filter based visual tracking ［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision Workshops. Piscataway： IEEE， 2015：621-629. 10.1109/iccvw.2015.84
3	BERTINETTO L， VALMADRE J， HENRIQUES J F， et al. Fully-convolutional Siamese networks for object tracking［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9914. Cham： Springer， 2016：850-865.
4	DANELLJAN M， ROBINSON A， KHAN F S， et al. Beyond correlation filters： learning continuous convolution operators for visual tracking ［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9909. Cham： Springer， 2016：472-488.
5	DANELLJAN M， BHAT G， KHAN F S， et al. ECO： efficient convolution operators for tracking［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017：6931-6939. 10.1109/cvpr.2017.733
6	NAM H， BAEK M， HAN B. Modeling and propagating CNNs in a tree structure for visual tracking ［EB/OL］. （2016-08-25）［2022-03-10］..
7	LI B， YAN J J， WU W， et al. High performance visual tracking with Siamese region proposal network ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8971-8980. 10.1109/cvpr.2018.00935
8	WANG Q， ZHANG L， BERTINETTO L， et al. Fast online object tracking and segmentation： a unifying approach ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019：1328-1338. 10.1109/cvpr.2019.00142
9	PENG X D， MA X L. Pythagorean fuzzy multi-criteria decision making method based on CODAS with new score function［J］. Journal of Intelligent and Fuzzy Systems， 2020， 38（3）：3307-3318. 10.3233/jifs-190043
10	BOLYA D， ZHOU C， XIAO F， et al. YOLACT++： better real-time instance segmentation［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 44（2）：1108-1121. 10.1109/tpami.2020.3014297
11	RGEY Z， NIKOS K. Learning to compare image patches via convolutional neural networks ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 4353-4361. 10.1109/cvpr.2015.7299064
12	PERAZZI F， PONT-TUSET J， McWILLIAMS B， et al. A benchmark dataset and evaluation methodology for video object segmentation ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016：724-732. 10.1109/cvpr.2016.85
13	PONT-TUSET J， PERAZZI F， CAELLES S， et al. The 2017 DAVIS challenge on video object segmentation ［EB/OL］. （2018-03-01）［2022-03-10］.. 10.1109/cvpr.2017.565
14	VOIGTLAENDER P， LEIBE B. Online adaptation of convolutional neural networks for video object segmentation ［C］// Proceedings of the 2017 British Machine Vision Conference. Durham： BMVA Press， 2017： No.116. 10.5244/c.31.116
15	CAELLES S， MANINIS K K， PONT-TUSET J， et al. One-shot video object segmentation［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017：5320-5329. 10.1109/cvpr.2017.565
16	MANINIS K K， CAELLES S， CHEN Y， et al. Video object segmentation without temporal information ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2019， 41（6）： 1515-1530. 10.1109/tpami.2018.2838670
17	PERAZZI F， KHOREVA A， BENENSON R， et al. Learning video object segmentation from static images ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 3491-3500. 10.1109/cvpr.2017.372
18	CHENG J C， TSAI Y H， HUNG W C， et al. Fast and accurate online video object segmentation via tracking parts［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018：7415-7424. 10.1109/cvpr.2018.00774
19	OH S W， LEE J Y， SUNKAVALLI K， et al. Fast video object segmentation by reference-guided mask propagation［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7376-7385. 10.1109/cvpr.2018.00770
20	YANG L J， WANG Y R， XIONG X H， et al. Efficient video object segmentation via network modulation ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018：6499-6507. 10.1109/cvpr.2018.00680
21	KRISTAN M， LEONARDIS A， MATAS J， et al. The visual object tracking VOT2016 challenge results ［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9914. Cham： Springer， 2016：777-823.
22	KRISTAN M， LEONARDIS A， MATAS J， et al. The sixth visual object tracking VOT2018 challenge results ［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11129. Cham： Springer， 2019： 3-53.
23	DANELLJAN M， BHAT G， KHAN F S， et al. ATOM： accurate tracking by overlap maximization［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019：4660-4669. 10.1109/cvpr.2019.00479
24	DAI K， WANG D， LU H， et al. Visual tracking via adaptive spatially-regularized correlation filters［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019：4670-4679. 10.1109/cvpr.2019.00480
25	ZHU Z， WANG Q， LI B， et al. Distractor-aware Siamese networks for visual object tracking ［C］// Proceedings of the 2018 International Conference on European Conference on Computer Vision. Berlin， Heidelberg： Springer， 2018： 101-117. 10.1007/978-3-030-01240-3_7
26	LI B， WU W， WANG Q， et al. SiamRPN++： evolution of siamese visual tracking with very deep networks ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020：4282-4291. 10.1109/cvpr.2019.00441
27	XU T， FENG Z H， WU X J， et al. Learning adaptive discriminative correlation filters via temporal consistency preserving spatial feature selection for robust visual object tracking［J］. IEEE Transactions on Image Processing， 2019， 28（11）：5596-5609. 10.1109/tip.2019.2919201
28	WANG G T， LUO C， XIONG Z W. Spmtracker： series-parallel matching for real-time visual object tracking ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3643-3652. 10.1109/cvpr.2019.00376
29	BHAT G， JOHNANDER J， DANELLJAN M， et al. Unveiling the power of deep tracking ［C］// Proceedings of the 2018 European Conference on Computer Vision. Berlin， Heidelberg： Springer-Verlag， 2018：483-498. 10.1007/978-3-030-01216-8_30
30	BAI S， HE Z， XU T-B， et al. Multi-hierarchical independent correlation filters for visual tracking［EB/OL］. （2018-11-26）［2022-03-11］. . 10.1109/icme46284.2020.9102759
31	XU T， FENG Z H， WU X J， et al. Joint group feature selection and discriminative filter learning for robust visual object tracking［C］// Proceedings of the 2019 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2019： 7950-796. 10.1109/iccv.2019.00804

算法	OL	M	DAVIS 2016			DAVIS 2017			运行速度/ （frame·s^-1）
算法	OL	M	J&F	J	F	J&F	J	F	运行速度/ （frame·s^-1）
OnAVOS	√	√	0.855 0	0.861	0.849	0.678 5	0.645	0.712	0.08
OSVOS	√	√	0.802 0	0.798	0.806	0.602 5	0.566	0.639	0.10
OSVOS^S	√	√	—	—	—	0.680 0	0.647	0.713	0.22
MSK	√	√	0.775 5	0.797	0.754	—	—	—	0.10
FAVOS	×	√	0.809 5	0.824	0.795	0.582 0	0.546	0.618	0.80
RGMP	×	√	0.817 5	0.815	0.820	0.667 0	0.648	0.689	8.00
OSMN	×	√	0.734 5	0.740	0.729	0.548 0	0.525	0.571	8.00
SiamMask	×	×	0.697 5	0.717	0.678	0.564 0	0.543	0.585	55.00
本文算法	×	×	0.836 5	0.840	0.833	0.690 5	0.696	0.685	32.00

算法	OL	M	DAVIS 2016			DAVIS 2017			运行速度/ （frame·s^-1）
算法	OL	M	J&F	J	F	J&F	J	F	运行速度/ （frame·s^-1）
OnAVOS	√	√	0.855 0	0.861	0.849	0.678 5	0.645	0.712	0.08
OSVOS	√	√	0.802 0	0.798	0.806	0.602 5	0.566	0.639	0.10
OSVOS^S	√	√	—	—	—	0.680 0	0.647	0.713	0.22
MSK	√	√	0.775 5	0.797	0.754	—	—	—	0.10
FAVOS	×	√	0.809 5	0.824	0.795	0.582 0	0.546	0.618	0.80
RGMP	×	√	0.817 5	0.815	0.820	0.667 0	0.648	0.689	8.00
OSMN	×	√	0.734 5	0.740	0.729	0.548 0	0.525	0.571	8.00
SiamMask	×	×	0.697 5	0.717	0.678	0.564 0	0.543	0.585	55.00
本文算法	×	×	0.836 5	0.840	0.833	0.690 5	0.696	0.685	32.00

数据集	算法	A	R	EAO
VOT2016	SiamMask	0.640	0.214	0.433
	ATOM	0.610	0.180	0.430
	ECO	0.550	0.200	0.375
	ASRCF	0.560	0.187	0.391
	C-COT	0.540	0.238	0.331
	TCNN	0.550	0.268	0.325
	SiamRPN	0.560	0.302	0.344
	DaSiamRPN	0.610	0.220	0.411
	SiamRPN++	0.640	0.200	0.464
	本文算法	0.620	0.142	0.475
VOT2018	SiamMask	0.610	0.276	0.380
	ATOM	0.590	0.204	0.401
	ECO	0.484	0.276	0.280
	LADCF	0.510	0.159	0.389
	SPM	0.580	0.300	0.338
	RCO	0.507	0.155	0.376
	UPDT	0.536	0.184	0.379
	MFT	0.505	0.140	0.386
	GFS-DCF	0.510	0.140	0.397
	本文算法	0.586	0.183	0.421

数据集	算法	A	R	EAO
VOT2016	SiamMask	0.640	0.214	0.433
	ATOM	0.610	0.180	0.430
	ECO	0.550	0.200	0.375
	ASRCF	0.560	0.187	0.391
	C-COT	0.540	0.238	0.331
	TCNN	0.550	0.268	0.325
	SiamRPN	0.560	0.302	0.344
	DaSiamRPN	0.610	0.220	0.411
	SiamRPN++	0.640	0.200	0.464
	本文算法	0.620	0.142	0.475
VOT2018	SiamMask	0.610	0.276	0.380
	ATOM	0.590	0.204	0.401
	ECO	0.484	0.276	0.280
	LADCF	0.510	0.159	0.389
	SPM	0.580	0.300	0.338
	RCO	0.507	0.155	0.376
	UPDT	0.536	0.184	0.379
	MFT	0.505	0.140	0.386
	GFS-DCF	0.510	0.140	0.397
	本文算法	0.586	0.183	0.421

模型	MaskIoU	Appearance	KF	DFPN1	DFPN2	EAO	J	跟踪速度/ （frame·s^-1）
MaskIoU_based	√					0.334	0.613	48
Appearance_based		√				0.386	0.667	19
MaskIoU+Appearance+DFPN1	√	√		√		0.403	0.696	32
本文模型	√	√	√	√	√	0.421	0.696	32

Object tracking based on instance segmentation and Pythagorean fuzzy decision-making

基于实例分割与毕达哥拉斯模糊决策的目标跟踪

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 31

Related Articles 15

Recommended Articles

Metrics

[1]	Wu XIONG, Congjun CAO, Xuefang SONG, Yunlong SHAO, Xusheng WANG. Handwriting identification method based on multi-scale mixed domain attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2225-2232.
[2]	Xun SUN, Ruifeng FENG, Yanru CHEN. Monocular 3D object detection method integrating depth and instance segmentation [J]. Journal of Computer Applications, 2024, 44(7): 2208-2215.
[3]	Yuanjiong LIU, Maozheng HE, Yibin HUANG, Cheng QIAN. Ship identification model based on ResNet50 and improved attention mechanism [J]. Journal of Computer Applications, 2024, 44(6): 1935-1941.
[4]	Ziwen SUN, Lizhi QIAN, Chuandong YANG, Yibo GAO, Qingyang LU, Guanglin YUAN. Survey of visual object tracking methods based on Transformer [J]. Journal of Computer Applications, 2024, 44(5): 1644-1654.
[5]	Zhiwen JING, Yujia ZHANG, Boting SUN, Hao GUO. Two-stage recommendation algorithm of Siamese graph convolutional neural network [J]. Journal of Computer Applications, 2024, 44(2): 469-476.
[6]	Chenhui CUI, Suzhen LIN, Dawei LI, Xiaofei LU, Jie WU. Infrared dim small target tracking method based on Siamese network and Transformer [J]. Journal of Computer Applications, 2024, 44(2): 563-571.
[7]	Cunyi LIAO, Yi ZHENG, Weijin LIU, Huan YU, Shouyin LIU. Decoupling-fusing algorithm for multiple tasks with autonomous driving environment perception [J]. Journal of Computer Applications, 2024, 44(2): 424-431.
[8]	Li ZENG, Jingru YANG, Gang HUANG, Xiang JING, Chaoran LUO. Survey on hypergraph application methods： issues， advances， and challenges [J]. Journal of Computer Applications, 2024, 44(11): 3315-3326.
[9]	Rui HUANG, Chaoqun ZHANG, Xuyi CHENG, Yan XING, Bao ZHANG. Incomplete instance guided aeroengine blade instance segmentation [J]. Journal of Computer Applications, 2024, 44(1): 167-174.
[10]	Junyu YANG, Yan DONG, Zhennan LONG, Xin YANG, Bin HAN. Rain detection algorithm based on event camera [J]. Journal of Computer Applications, 2023, 43(9): 2904-2909.
[11]	Kunting LU, Rongrong FEI, Xuande ZHANG. Remote sensing image pansharpening by convolutional neural network [J]. Journal of Computer Applications, 2023, 43(9): 2963-2969.
[12]	Junjian JIANG, Dawei LIU, Yifan LIU, Yougui REN, Zhibin ZHAO. Few-shot object detection algorithm based on Siamese network [J]. Journal of Computer Applications, 2023, 43(8): 2325-2329.
[13]	Li WANG, Shibin XUAN, Xuyang QIN, Ziwei LI. Multi-object tracking method based on dual-decoder Transformer [J]. Journal of Computer Applications, 2023, 43(6): 1919-1929.
[14]	Guangyao ZHANG, Chunfeng SONG. Pedestrian head tracking model based on full-body appearance features [J]. Journal of Computer Applications, 2023, 43(5): 1372-1377.
[15]	Yue CHI, Zhengping LI, Chao XU, Bo FENG. Highlight removal algorithm for medical endoscopic images [J]. Journal of Computer Applications, 2023, 43(4): 1278-1283.