Multi-UAV real-time tracking algorithm based on improved PP-YOLO and Deep-SORT

doi:10.11772/j.issn.1001-9081.2021071146

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (9): 2885-2892.DOI: 10.11772/j.issn.1001-9081.2021071146

• Multimedia computing and computer simulation • Previous Articles Next Articles

Multi-UAV real-time tracking algorithm based on improved PP-YOLO and Deep-SORT

Jun MA¹^,²(), Zhen YAO¹, Cuifeng XU¹^,², Shouhong CHEN¹^,²

^1.School of Electronic Engineering and Automation，Guilin University of Electronic Technology，Guilin Guangxi 541004，China
^2.Guangxi Key Laboratory of Automatic Detecting Technology and Instruments （Guilin University of Electronic Technology），Guilin Guangxi 541004，China

Received:2021-07-02 Revised:2021-09-14 Accepted:2021-09-15 Online:2021-09-24 Published:2022-09-10
Contact: Jun MA
About author:YAO Zhen， born in 1996， M. S. candidate. His research interests include deep learning， target detection， target tracking.
XU Cuifeng， born in 1977， M. S.， senior experimentalist. Her research interests include signal processing， digital image processing.
CHEN Shouhong， born in 1981， M. S.， senior experimentalist. His research interests include neural network， machine learning， computer-aided testing.
Supported by:
National Natural Science Foundation of China(61671008);Innovation Project of Graduate Education of Guilin University of Electronic Technology(2020YCXS095)

基于改进PP-YOLO和Deep-SORT的多无人机实时跟踪算法

马峻¹^,²(), 姚震¹, 徐翠锋¹^,², 陈寿宏¹^,²

^1.桂林电子科技大学电子工程与自动化学院，广西桂林 541004
^2.广西自动检测技术与仪器重点实验室（桂林电子科技大学），广西桂林 541004

通讯作者: 马峻
作者简介:姚震（1996—），男，陕西商南人，硕士研究生，主要研究方向：深度学习、目标检测、目标跟踪；
徐翠锋（1977—），女，黑龙江鸡西人，正高级实验师，硕士，主要研究方向：信号处理、数字图像处理；
陈寿宏（1981—），男，江西新干人，高级实验师，硕士，主要研究方向：神经网络、机器学习、计算机辅助测试。
基金资助:
国家自然科学基金资助项目(61671008);桂林电子科技大学研究生教育创新项目(2020YCXS095)

Abstract

Abstract:

The target size of the Unmanned Aerial Vehicle （UAV） is small， and the characteristics among multiple UAVs are not obvious. At the same time， the interference of birds and flying insects brings a huge challenge to the accurate detection and stable tracking of the UAV targets. Aiming at the problem of poor detection performance and unstable tracking of small target UAVs by using traditional target detection algorithms， a real-time tracking algorithm for multiple UAVs based on improved PaddlePaddle-YOLO （PP-YOLO） and Simple Online and Realtime Tracking with a Deep association metric （Deep-SORT） was proposed. Firstly， the squeeze-excitation module was integrated into PP-YOLO detection algorithm to achieve feature extraction and detection of UAV targets. Secondly， the Mish activation function was introduced into ResNet50-vd structure to solve the problem of vanishing gradient in the back propagation process and further improve the detection precision. Thirdly， Deep-SORT algorithm was used to track UAV targets in real time， and the backbone network that extracts appearance features was replaced with ResNet50， thereby improving the original network’s weak perceptual ability of small appearances. Finally， the loss function Margin Loss was introduced， which not only improved the class separability， but also strengthened the tightness within the class and the difference between classes. Experimental results show that the detection mean Average Precision （mAP） of the proposed algorithm is increased by 2.27 percentage points compared to that of the original PP-YOLO algorithm， and the tracking accuracy of the proposed algorithm is increased by 4.5 percentage points compared to that of the original Deep-SORT algorithm. The proposed algorithm has a tracking accuracy of 91.6%， can track multiple UAV targets within 600 m in real time， and effectively solves the problem of "frame loss" in the tracking process.

Key words: Unmanned Aerial Vehicle (UAV) detection, real-time tracking, squeeze-excitation module, Mish activation function, Margin Loss

摘要：

无人机（UAV）目标尺寸较小，多架无人机之间特征也不明显，且鸟类和飞虫的干扰给无人机目标的准确检测和稳定跟踪带来了巨大挑战。针对传统目标检测算法对小目标无人机检测性能差、跟踪不稳定的问题，提出一种基于改进PP-YOLO和Deep-SORT的多无人机实时跟踪算法。首先，将压缩-激励模块融入PP-YOLO检测算法中，以实现对无人机目标的特征提取和检测；其次，在ResNet50-vd结构中引入Mish激活函数，以解决反向传播过程中的梯度消失问题，并进一步提升检测精度；然后，采用Deep-SORT算法来实时跟踪无人机目标，并将提取外观特征的主干网络更换为ResNet50，从而改善原有网络对微小外观感知能力弱的状况；最后，引入损失函数Margin Loss，既提高了类别可分性，又加强了类内紧度和类间差异。实验结果表明，所提算法的检测平均精度均值（mAP）相比原始PP-YOLO算法提升了2.27个百分点，跟踪准确性相对于原始Deep-SORT算法提升了4.5个百分点。所提算法的跟踪准确性可达91.6%，能够实时跟踪600 m以内多架无人机目标，有效解决了跟踪过程中的“丢帧”问题。

关键词: 无人机检测, 实时跟踪, 压缩-激励模块, Mish激活函数, Margin Loss

CLC Number:

TP391.41

Jun MA, Zhen YAO, Cuifeng XU, Shouhong CHEN. Multi-UAV real-time tracking algorithm based on improved PP-YOLO and Deep-SORT[J]. Journal of Computer Applications, 2022, 42(9): 2885-2892.

马峻, 姚震, 徐翠锋, 陈寿宏. 基于改进PP-YOLO和Deep-SORT的多无人机实时跟踪算法[J]. 《计算机应用》唯一官方网站, 2022, 42(9): 2885-2892.

Figures/Tables 11

Fig. 1 Overall network structure

Fig. 2 Squeeze-excitation module structure

Fig. 3 ResNet50-vd structure with Mish activation function introduced

Fig. 4 Self-built UAV dataset

Fig. 5 Dataset of UAV cross-border identification

Fig. 6 Average precision curve

Fig. 7 Loss function curve

Fig. 8 Measure learning precision and recall

Tab. 1 Performance of common target detection algorithms on three datasets

算法	TIB-Net			Drone-vs-Bird			自制数据集（Homemade Dataset）
算法	mAP/%	Recall/%	FPS	mAP/%	Recall/%	FPS	mAP/%	Recall/%	FPS
SSD	87.28	92.35	12.1	90.25	91.05	13.5	90.32	92.34	14.3
Faster RCNN	89.31	91.18	15.2	90.12	92.20	18.2	90.41	92.75	17.6
YOLOv3	84.36	90.24	36.1	89.15	91.68	35.8	90.30	92.68	39.4
YOLOv4	85.44	90.35	35.2	89.61	92.35	37.5	90.47	90.05	40.2
PP-YOLO	87.12	91.29	48.8	89.86	92.61	45.3	92.82	93.28	47.2
PP-YOLO+Mish	88.25	93.17	47.6	90.36	92.89	44.5	93.36	93.75	45.9
本文算法	90.02	94.72	45.3	91.98	93.52	42.7	94.62	94.26	44.6

Tab. 2 Tracking results of the proposed algorithm in UAV videos

算法	MOTA/% $↑$	MOTP/% $↑$	FN/帧 $↓$	FP/帧 $↓$	IDSW/帧 $↓$	FPS $↑$
JDE^［29］	90.3	91.2	1 280	1 245	425	32.1
Deep-SORT^［8］	87.1	89.2	1 589	1 628	632	46.3
CenterTrack^［28］	90.9	92.3	1 306	1044	411	23.2
SiamRPN++^［30］	89.6	90.5	1 294	1 208	437	32.6
FairMOT^［22］	91.2	92.8	1 252	1 264	405	25.8
本文算法	91.6	92.6	1243	1 199	399	35.5

Tab. 2 Tracking results of the proposed algorithm in UAV videos

算法	MOTA/% $↑$	MOTP/% $↑$	FN/帧 $↓$	FP/帧 $↓$	IDSW/帧 $↓$	FPS $↑$
JDE^［29］	90.3	91.2	1 280	1 245	425	32.1
Deep-SORT^［8］	87.1	89.2	1 589	1 628	632	46.3
CenterTrack^［28］	90.9	92.3	1 306	1044	411	23.2
SiamRPN++^［30］	89.6	90.5	1 294	1 208	437	32.6
FairMOT^［22］	91.2	92.8	1 252	1 264	405	25.8
本文算法	91.6	92.6	1243	1 199	399	35.5

Fig. 9 Tracking results comparison

References 30

1	PAWEŁCZYK M Ł， WOJTYRA M. Real world object detection dataset for quadcopter unmanned aerial vehicle detection［J］. IEEE Access， 2020， 8： 174394-174409. 10.1109/access.2020.3026192
2	闫若怡，熊丹，于清华，等. 基于并行跟踪检测框架与深度学习的目标跟踪算法［J］. 计算机应用， 2019， 39（2）： 343-347. 10.11772/j.issn.1001-9081.2018061211
	YAN R Y， XIONG D， YU Q H， et al. Object tracking algorithm based on parallel tracking and detection framework and deep learning［J］. Journal of Computer Applications， 2019， 39（2）： 343-347. 10.11772/j.issn.1001-9081.2018061211
3	HAO J X， ZHOU Y M， ZHANG G S， et al. A review of target tracking algorithm based on UAV［C］// Proceedings of the 2018 IEEE International Conference on Cyborg and Bionic Systems. Piscataway： IEEE， 2018： 328-333. 10.1109/cbs.2018.8612263
4	HENRIQUES J F， CASEIRO R， MARTINS P， et al. Exploiting the circulant structure of tracking-by-detection with kernels［C］// Proceedings of the 2012 European Conference on Computer Vision， LNCS 7575. Cham： Springer， 2012： 702-715.
5	HENRIQUES J F， CASEIRO R， MARTINS P， et al. High-speed tracking with kernelized correlation filters［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2015， 37（3）： 583-596. 10.1109/tpami.2014.2345390
6	DANELLJAN M， HÄGER G， KHAN F S， et al. Discriminative scale space tracking［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017， 39（8）： 1561-1575. 10.1109/tpami.2016.2609928
7	BEWLEY A， GE Z Y， OTT L， et al. Simple online and realtime tracking［C］// Proceedings of the 2016 IEEE International Conference on Image Processing. Piscataway： IEEE， 2016： 3464-3468. 10.1109/icip.2016.7533003
8	WOJKE N， BEWLEY A， PAULUS D. Simple online and realtime tracking with a deep association metric［C］// Proceedings of the 2017 IEEE International Conference on Image Processing. Piscataway： IEEE， 2017： 3645-3649. 10.1109/icip.2017.8296962
9	LIU Y C， LIAO L C， WU H， et al. Trajectory and image-based detection and identification of UAV［J］. The Visual Computer， 2021， 37（7）： 1769-1780. 10.1007/s00371-020-01937-y
10	HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
11	DEMIR B， ERGUNAY S， NURLU G， et al. Real-time high-resolution omnidirectional imaging platform for drone detection and tracking［J］. Journal of Real-Time Image Processing， 2020， 17（5）： 1625-1635. 10.1007/s11554-019-00921-7
12	CRAYE C， ARDJOUNE S. Spatio-temporal semantic segmentation for drone detection［C］// Proceedings of the 16th IEEE International Conference on Advanced Video and Signal Based Surveillance. Piscataway： IEEE， 2019： 1-5. 10.1109/avss.2019.8909854
13	RONNEBERGER O， FISCHER P， BROX T. U-Net： convolutional networks for biomedical image segmentation［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9351. Cham： Springer， 2015： 234-241.
14	COLUCCIA A， FASCISTA A， SCHUMANN A， et al. Drone-vs-bird detection challenge at IEEE AVSS2019［C］// Proceedings of the 16th IEEE International Conference on Advanced Video and Signal Based Surveillance. Piscataway： IEEE， 2019： 1-7. 10.1109/avss.2019.8909876
15	HU Y Y， WU X J， ZHENG G D， et al. Object detection of UAV for anti-UAV based on improved YOLO v3［C］// Proceedings of the 2019 Chinese Control Conference. Piscataway： IEEE， 2019： 8386-8390. 10.23919/chicc.2019.8865525
16	REDMON J， FARHADI A. YOLOv3： an incremental improvement［EB/OL］. （2018-04-08）［2021-05-21］..
17	PHAM G N， NGUYEN P H. Drone detection experiment based on image processing and machine learning［J］. International Journal of Scientific and Technology Research， 2020， 9（2）： 2965-2971.
18	TORVIK B， OLSEN K E， GRIFFITHS H. Classification of birds and UAVs based on radar polarimetry［J］. IEEE Geoscience and Remote Sensing Letters， 2016， 13（9）： 1305-1309. 10.1109/lgrs.2016.2582538
19	COLUCCIA A， FASCISTA A， SCHUMANN A， et al. Drone vs. bird detection： deep learning algorithms and results from a grand challenge［J］. Sensors， 2021， 21（8）： No.2824. 10.3390/s21082824
20	SUN H， YANG J， SHEN J Q， et al. TIB-Net： drone detection network with tiny iterative backbone［J］. IEEE Access， 2020， 8： 130697-130707. 10.1109/access.2020.3009518
21	LONG X， DENG K P， WANG G Z， et al. PP-YOLO： an effective and efficient implementation of object detector［EB/OL］. （2020-08-03）［2021-05-22］..
22	ZHANG Y F， WANG C Y， WANG X G， et al. FairMOT： on the fairness of detection and re-identification in multiple object tracking［J］. International Journal of Computer Vision， 2021， 129（11）： 3069-3087. 10.1007/s11263-021-01513-4
23	金国栋，薛远亮，谭力宁，等. 基于双注意力混洗的无人机航拍目标跟踪算法［J/OL］. 北京航空航天大学学报. （2021-06-16）［2021-07-23］..
	JIN G D， QUE Y L， TAN L N， et al. UAV aerial object tracking algorithm based on dual-attention shuffling［J/OL］. Journal of Beijing University of Aeronautics and Astronautics. （2021-06-16）［2021-07-23］..
24	HU J， SHEN L， SUN G. Squeeze-and-excitation networks［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7132-7141. 10.1109/cvpr.2018.00745
25	SZANDAŁA T. Review and comparison of commonly used activation functions for deep neural networks［M］// BHOI A K， MALLICK P K， LIU C M， et al. Bio-inspired Neurocomputing， SCI 903. Singapore： Springer， 2021： 203-224.
26	ZHENG L， YANG Y， HAUPTMANN A G. Person re-identification： past， present and future［EB/OL］. （2016-10-10）［2021-05-28］..
27	DENG J K， GUO J， XUE N N， et al. ArcFace： additive angular margin loss for deep face recognition［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 4685-4694. 10.1109/cvpr.2019.00482
28	ZHOU X Y， KOLTUN V， KRÄHENBÜHL P. Tracking objects as points［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12349. Cham： Springer， 2020： 474-490.
29	WANG Z D， ZHENG L， LIU Y X， et al. Towards real-time multi-object tracking［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12356. Cham： Springer， 2020： 107-122.
30	LI B， WU W， WANG Q， et al. SiamRPN++： evolution of Siamese visual tracking with very deep networks［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 4277-4286. 10.1109/cvpr.2019.00441

Multi-UAV real-time tracking algorithm based on improved PP-YOLO and Deep-SORT

基于改进PP-YOLO和Deep-SORT的多无人机实时跟踪算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 30

Related Articles 1

Recommended Articles

Metrics