Pedestrian trajectory prediction based on multi-head soft attention graph convolutional network

doi:10.11772/j.issn.1001-9081.2022020207

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (3): 736-743.DOI: 10.11772/j.issn.1001-9081.2022020207

• Artificial intelligence • Previous Articles

Pedestrian trajectory prediction based on multi-head soft attention graph convolutional network

Tao PENG¹^,²^,³, Yalong KANG²^,³, Feng YU¹^,³(), Zili ZHANG²^,³, Junping LIU²^,³, Xinrong HU²^,³, Ruhan HE¹^,³, Li LI¹^,²

^1.Hubei Provincial Engineering Research Center for Intelligent Textile and Fashion （Wuhan Textile University），Wuhan Hubei 430200，China
^2.Engineering Research Center of Hubei Province for Clothing Information （Wuhan Textile University），Wuhan Hubei 430200，China
^3.School of Computer Science and Artificial Intelligence，Wuhan Textile University，Wuhan Hubei 430200，China

Received:2022-02-24 Revised:2022-05-17 Accepted:2022-05-19 Online:2022-08-16 Published:2023-03-10
Contact: Feng YU
About author:PENG Tao， born in 1981， Ph. D.， professor. His research interests include data reduction， pattern recognition， network security.
KANG Yalong， born in 1997， M. S. candidate. His research interests include computer vision.
ZHANG Zili， born in 1981， Ph. D.， lecturer. His research interests include image processing， computer vision.
LIU Junping， born in 1980， Ph. D.， associate professor. His research interests include computer vision.
HU Xinrong， born in 1973， Ph. D.， professor. Her research interests include graphics and image processing.
HE Ruhan， born in 1974， Ph. D.， professor. His research interests include machine learning， artificial intelligence.
LI Li， born in 1982， Ph. D.， associate professor. Her research interests include machine vision， optical nondestructive testing.
Supported by:
National Natural Science Foundation of China(61901308);Youth Project of Education Department of Hubei Province(Q201316);Key Project of Scientific Research Plan of Education Department of Hubei Province(D20191708)

基于多头软注意力图卷积网络的行人轨迹预测

彭涛¹^,²^,³, 康亚龙²^,³, 余锋¹^,³(), 张自力²^,³, 刘军平²^,³, 胡新荣²^,³, 何儒汉¹^,³, 李丽¹^,²

^1.纺织服装智能化湖北省工程研究中心(武汉纺织大学), 武汉 430200
^2.湖北省服装信息化工程技术研究中心(武汉纺织大学), 武汉 430200
^3.武汉纺织大学计算机与人工智能学院, 武汉 430200

通讯作者: 余锋
作者简介:彭涛（1981—），男，湖北武汉人，教授，博士，CCF会员，主要研究方向：数据简化、模式识别、网络安全
康亚龙（1997—），男，湖北孝感人，硕士研究生，CCF会员，主要研究方向：计算机视觉
余锋（1989—），男，湖北武汉人，讲师，博士，CCF会员，主要研究方向：医学图像处理、光学成像
张自力（1981—），男，湖北武汉人，讲师，博士，CCF会员，主要研究方向：图像处理、计算机视觉
刘军平（1980—），男，湖北武汉人，副教授，博士，CCF会员，主要研究方向：计算机视觉
胡新荣（1973—），女，湖北武汉人，教授，博士，CCF会员，主要研究方向：图形图像处理
何儒汉（1974—），男，湖北武汉人，教授，博士，CCF会员，主要研究方向：机器学习、人工智能
李丽（1982—），女，湖北武汉人，副教授，博士，CCF会员，主要研究方向：机器视觉、光学无损检测。
基金资助:
国家自然科学基金资助项目(61901308);湖北省教育厅青年项目(Q201316);湖北省教育厅科研计划重点项目(D20191708)

Abstract

Abstract:

The complexity of pedestrian interaction is a challenge for pedestrian trajectory prediction， and the existing algorithms are difficult to capture meaningful interaction information between pedestrians， which cannot intuitively model the interaction between pedestrians. To address this problem， a multi-head soft attention graph convolutional network was proposed. Firstly， a Multi-head Soft ATTention （MS ATT） combined with involution network was used to extract sparse spatial adjacency matrix and sparse temporal adjacency matrix from spatial and temporal graph inputs respectively to generate sparse spatial directed graph and sparse temporal directed graph. Then， a Graph Convolutional Network （GCN） was used to learn interaction and motion trend features from sparse spatial and sparse temporal directed graphs. Finally， the learned trajectory features were input into a Temporal Convolutional Network （TCN） to predict double Gaussian distribution parameters， thereby generating the predicted pedestrian trajectories. Experiments on Eidgenossische Technische Hochschule （ETH） and University of CYprus （UCY） datasets show that， compared with Space-time sOcial relationship pooling pedestrian trajectory Prediction Model （SOPM）， the proposed algorithm reduces the Average Displacement Error （ADE） by 2.78%， and compared to Sparse Graph Convolution Network （SGCN）， the proposed algorithm reduces the Final Displacement Error （FDE） by 16.92%.

Key words: Multi-head Soft ATTention (MS ATT), channel attention, spatial attention, involution network, Graph Convolutional Network (GCN), pedestrian trajectory prediction

摘要：

行人间交互作用的复杂性给行人轨迹预测带来了挑战，且现有算法难以捕获行人间有意义的交互信息，不能直观地建模行人间的交互作用。针对以上问题，提出多头软注意力图卷积网络。首先利用多头软注意力（MS ATT）结合内卷网络Involution分别从空间图和时间图输入中提取稀疏空间和稀疏时间邻接矩阵，生成稀疏空间和稀疏时间有向图；然后，利用图卷积网络（GCN）从稀疏空间和稀疏时间有向图中学习交互作用与运动趋势特征；最后，将学习到的轨迹特征输入时间卷积网络（TCN）以预测双高斯分布参数，生成行人预测轨迹。在ETH和UCY数据集上的实验结果表明：相较于空时社交关系池化行人轨迹预测模型（SOPM），所提算法的平均位移误差（ADE）降低了2.78%；相较于稀疏图卷积网络（SGCN），所提算法的最终位移误差（FDE）降低了16.92%。

关键词: 多头软注意力, 通道注意力, 空间注意力, 内卷, 图卷积网络, 行人轨迹预测

CLC Number:

TP391

Tao PENG, Yalong KANG, Feng YU, Zili ZHANG, Junping LIU, Xinrong HU, Ruhan HE, Li LI. Pedestrian trajectory prediction based on multi-head soft attention graph convolutional network[J]. Journal of Computer Applications, 2023, 43(3): 736-743.

彭涛, 康亚龙, 余锋, 张自力, 刘军平, 胡新荣, 何儒汉, 李丽. 基于多头软注意力图卷积网络的行人轨迹预测[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 736-743.

Figures/Tables 10

Fig. 1 Structure of multi-head soft attention

Fig. 2 Structure of sparse involution learning

Fig. 3 Structure of multi-head soft attention graph convolution network

Tab. 1 ADE， FDE indicators of different algorithms

算法	ETH		HOTEL		UNIV		ZARA1		ZARA2		平均值
算法	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE
S-LSTM	1.09	2.35	0.79	1.76	0.67	1.40	0.47	1.00	0.56	1.17	0.72	1.54
S-GAN	0.87	1.62	0.67	1.37	0.76	1.52	0.35	0.68	0.42	0.84	0.61	1.21
SoPhie	0.70	1.43	0.76	1.67	0.54	1.24	0.30	0.63	0.38	0.78	0.51	1.15
PITF	0.73	1.65	0.30	0.59	0.60	1.27	0.38	0.81	0.31	0.68	0.46	1.00
S-BIGAT	0.69	1.29	0.49	1.01	0.55	1.32	0.30	0.62	0.36	0.75	0.48	1.00
GAT	0.68	1.29	0.68	1.40	0.57	1.29	0.29	0.60	0.37	0.75	0.52	1.07
SSTGCNN	0.64	1.11	0.49	0.85	0.44	0.79	0.34	0.53	0.30	0.48	0.44	0.75
RSBG	0.80	1.53	0.33	0.64	0.59	1.25	0.40	0.86	0.30	0.65	0.48	0.99
STAR	0.56	1.11	0.26	0.50	0.52	1.15	0.41	0.90	0.31	0.71	0.41	0.87
SOPM	0.61	1.27	0.40	0.81	0.34	0.68	0.23	0.49	0.21	0.45	0.36	0.74
SA-GAN	0.72	1.28	0.50	1.01	0.58	1.19	0.42	0.83	0.39	0.85	0.52	1.03
TP-GCN	0.74	1.24	0.28	0.51	0.50	1.07	0.33	0.71	0.28	0.61	0.43	0.83
SGCN	0.63	1.03	0.32	0.55	0.37	0.70	0.29	0.53	0.25	0.45	0.37	0.65
本文算法	0.60	0.85	0.26	0.37	0.36	0.64	0.29	0.43	0.23	0.40	0.35	0.54

Fig. 4 Training loss and validation loss

Tab.2 Parameters and reasoning time of algorithms

算法	参数量/10³	推理时间/s
S-LSTM	264.000	1.178 9
SR-LSTM	64.900	0.157 8
S-GAN	46.300	0.096 8
PITF	360.300	0.114 5
SSTGCNN	7.600	0.002 0
SGCN	25.369	0.003 0
本文算法	18.184	0.003 0

Tab. 3 Ablation experimental results of different modules

模块	ETH		HOTEL		UNIV		ZARA1		ZARA2		平均值
模块	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE
MS ATT	0.64	0.92	0.29	0.42	0.37	0.72	0.27	0.49	0.24	0.42	0.36	0.60
Involution	0.65	1.03	0.25	0.42	0.37	0.64	0.28	0.46	0.23	0.40	0.36	0.59
本文算法	0.60	0.85	0.26	0.37	0.36	0.64	0.29	0.43	0.23	0.40	0.35	0.54

Tab. 4 ADE/FDE indicators of ablation experiments of different ε

$ε$	ETH		HOTEL		UNIV		ZARA1		ZARA2		平均值
$ε$	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE
0.00	0.63	1.00	0.35	0.65	0.39	0.72	0.28	0.51	0.24	0.44	0.38	0.66
0.25	0.66	1.12	0.31	0.54	0.38	0.72	0.27	0.46	0.24	0.42	0.37	0.65
0.50	0.60	0.85	0.26	0.37	0.36	0.64	0.29	0.43	0.23	0.40	0.35	0.54
0.75	0.68	1.30	0.32	0.57	0.38	0.70	0.28	0.50	0.23	0.43	0.38	0.70
1.00	0.70	1.41	0.34	0.61	0.38	0.72	0.27	0.48	0.23	0.41	0.38	0.73

Tab. 4 ADE/FDE indicators of ablation experiments of different ε

$ε$	ETH		HOTEL		UNIV		ZARA1		ZARA2		平均值
$ε$	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE
0.00	0.63	1.00	0.35	0.65	0.39	0.72	0.28	0.51	0.24	0.44	0.38	0.66
0.25	0.66	1.12	0.31	0.54	0.38	0.72	0.27	0.46	0.24	0.42	0.37	0.65
0.50	0.60	0.85	0.26	0.37	0.36	0.64	0.29	0.43	0.23	0.40	0.35	0.54
0.75	0.68	1.30	0.32	0.57	0.38	0.70	0.28	0.50	0.23	0.43	0.38	0.70
1.00	0.70	1.41	0.34	0.61	0.38	0.72	0.27	0.48	0.23	0.41	0.38	0.73

Fig. 5 Visual representation of trajectories

Fig. 6 Visual representation of actual scenes

References 30

1	LARGE F， VASQUEZ D， FRAICHARD T， et al. Avoiding cars and pedestrians using velocity obstacles and motion prediction［C］// Proceedings of the 2004 IEEE Intelligent Vehicle Symposium. Piscataway： IEEE， 2004： 375-379.
2	HELBING D， MOLNÁR P. Social force model for pedestrian dynamics［J］. Physical Review. E， Statistical Physics， Plasmas， Fluids， and Related Interdisciplinary Topics， 1995， 51（5）： 4282-4286. 10.1103/physreve.51.4282
3	KELLER C G， GAVRILA D M. Will the pedestrian cross？ a study on pedestrian path prediction［J］. IEEE Transactions on Intelligent Transportation Systems， 2014， 15（2）： 494-506. 10.1109/tits.2013.2280766
4	KOOIJ J F P， SCHNEIDER N， FLOHR F， et al. Context-based pedestrian path prediction［C］// Proceedings of the 2014 European Conference on Computer Vision， LNCS 8694. Cham： Springer， 2014： 618-633.
5	ALAHI A， GOEL K， RAMANATHAN V， et al. Social LSTM： human trajectory prediction in crowded spaces［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016：961-971. 10.1109/cvpr.2016.110
6	GUPTA A， JOHNSON J， LI F F， et al. Social GAN： socially acceptable trajectories with generative adversarial networks［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018：2255-2264. 10.1109/cvpr.2018.00240
7	MOHAMED A， QIAN K， ELHOSEINY M， et al. Social-STGCNN： a social spatio-temporal graph convolutional neural network for human trajectory prediction［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020：14412-14420. 10.1109/cvpr42600.2020.01443
8	SHI L S， WANG L， LONG C J， et al. SGCN： sparse graph convolution network for pedestrian trajectory prediction［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 8990-9009. 10.1109/cvpr46437.2021.00888
9	LI D， HU J， WANG C H， et al. Involution： inverting the inherence of convolution for visual recognition［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 12316-12325. 10.1109/cvpr46437.2021.01214
10	BAI S J， KOLTER J Z， KOLTUN V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling［EB/OL］. （2018-04-19）［2021-12-22］..
11	SADEGHIAN A， KOSARAJU V， SADEGHIAN A， et al. SoPhie： an attentive GAN for predicting paths compliant to social and physical constraints［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019：1349-1358. 10.1109/cvpr.2019.00144
12	LIANG J W， JIANG L， NIEBLES J C， et al. Peeking into the future： predicting future person activities and locations in videos［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019：5718-5727. 10.1109/cvprw.2019.00358
13	KOSARAJU V， SADEGHIAN A， MARTÍN-MARTÍN R， et al. Social-BiGAT： Multimodal trajectory forecasting using bicycle-GAN and graph attention networks［C/OL］// Proceedings of the 33rd Conference on Neural Information Processing System. ［2021-12-17］..
14	VELIČKOVIĆ P， CUCURULL G， CASANOVA A， et al. Graph attention networks［EB/OL］. （2018-02-04）［2021-12-23］..
15	SUN J H， JIANG Q H， LU C W. Recursive social behavior graph for trajectory prediction［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 657-666. 10.1109/cvpr42600.2020.00074
16	YU C J， MA X， REN J W， et al. Spatio-temporal graph transformer networks for pedestrian trajectory prediction［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12357. Cham： Springer， 2020： 507-523.
17	王天保，刘昱，郭继昌，等. 图卷积神经网络行人轨迹预测算法［J］. 哈尔滨工业大学学报， 2021， 53（2）：53-60. 10.11918/202006051
	WANG T B， LIU Y， GUO J C， et al. Pedestrian trajectory prediction algorithm based on graph convolutional network［J］. Journal of Harbin Institute of Technology， 2021， 53（2）：53-60. 10.11918/202006051
18	张志远，刁英华. 结合社会特征和注意力的行人轨迹预测模型［J］. 西安电子科技大学学报， 2020， 47（1）：10-17， 79. 10.19665/j.issn1001-2400.2020.01.002
	ZHANG Z Y， DIAO Y H. Pedestrian trajectory prediction model with social features and attention［J］. Journal of Xidian University， 2020， 47（1）：10-17， 79. 10.19665/j.issn1001-2400.2020.01.002
19	毛琳，巩欣飞，杨大伟，等. 空时社交关系池化行人轨迹预测模型［J］. 计算机辅助设计与图形学学报， 2020， 32（12）：1918-1925. 10.3724/sp.j.1089.2020.18236
	MAO L， GONG X F， YANG D W， et al. Space-time social relationship pooling pedestrian trajectory prediction model［J］. Journal of Computer-Aided Design and Graphics， 2020， 32（12）：1918-1925. 10.3724/sp.j.1089.2020.18236
20	李琳辉，周彬，连静，等. 基于社会注意力机制的行人轨迹预测方法研究［J］. 通信学报， 2020， 41（6）：175-183. 10.11959/j.issn.1000-436x.2020100
	LI L H， ZHOU B， LIAN J， et al. Research on pedestrian trajectory prediction method based on social attention mechanism［J］. Journal on Communication， 2020， 41（6）：175-183. 10.11959/j.issn.1000-436x.2020100
21	程媛，迟荣华，黄少滨，等. 基于非参数密度估计的不确定轨迹预测方法［J］. 自动化学报， 2019， 45（4）：787-798. 10.16383/j.aas.2018.c170419
	CHENG Y， CHI R H， HUANG S B， et al. Uncertain trajectory prediction method using non-parametric density estimation［J］. Acta Automatica Sinica， 2019， 45（4）：787-798. 10.16383/j.aas.2018.c170419
22	KIPF T N， WELLING M. Semi-supervised classification with graph convolutional networks［EB/OL］. （2017-02-22）［2021-12-22］.. 10.48550/arXiv.1609.02907
23	YAN S J， XIONG Y J， LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2018：7444-7452. 10.1609/aaai.v32i1.12328
24	WOO S， PARK J， LEE J Y， et al. CBAM： convolutional block attention module［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11211. Cham： Springer， 2018：3-19.
25	PELLEGRINI S， ESS A， SCHINDLER K， et al. You'll never walk alone： modeling social behavior for multi-target tracking［C］// Proceedings of the IEEE 12th International Conference on Computer Vision. Piscataway： IEEE， 2009： 261-268. 10.1109/iccv.2009.5459260
26	LERNER A， CHRYSANTHOU Y， LISCHINSKI D. Crowds by example［J］. Computer Graphics Forum， 2007， 26（3）： 655-664. 10.1111/j.1467-8659.2007.01089.x
27	RAKSINCHAROENSAK P， HASEGAWA T， NAGAI M. Motion planning and control of autonomous driving intelligence system based on risk potential optimization framework［J］. International Journal of Automotive Engineering， 2016， 7（AVEC14）：53-60. 10.20485/jsaeijae.7.avec14_53
28	HE K M， ZHANG X Y， REN S Q， et al. Delving deep into rectifiers： surpassing human-level performance on ImageNet classification［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 1026-1034. 10.1109/iccv.2015.123
29	KINGMA D P， BA J L. Adam： a method for stochastic optimization［EB/OL］. （2017-01-30）［2021-12-22］..
30	ZHANG P， OUYANG W L， ZHANG P F， et al. SR-LSTM： state refinement for LSTM towards pedestrian trajectory prediction［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 12077-12086. 10.1109/cvpr.2019.01236

[1]	Yingmao YAO, Xiaoyan JIANG. Video-based person re-identification method based on graph convolution network and self-attention graph pooling [J]. Journal of Computer Applications, 2023, 43(3): 728-735.
[2]	Ruoying WANG, Fan LYU, Liuqing ZHAO, Fuyuan HU. Floorplan generation algorithm integrating user requirements and boundary constraints [J]. Journal of Computer Applications, 2023, 43(2): 575-582.
[3]	Yating SU, Cuixiang LIU. Three-dimensional human reconstruction model based on high-resolution net and graph convolutional network [J]. Journal of Computer Applications, 2023, 43(2): 583-588.
[4]	Li’an ZHU, Hong ZHANG. Nonhomogeneous image dehazing based on dual-branch conditional generative adversarial network [J]. Journal of Computer Applications, 2023, 43(2): 567-574.
[5]	Lining YUAN, Zhao LIU. Graph representation learning by autoencoder with one-shot aggregation [J]. Journal of Computer Applications, 2023, 43(1): 8-14.
[6]	Zanxia QIANG, Xianfu BAO. Residual attention deraining network based on convolutional long short-term memory [J]. Journal of Computer Applications, 2022, 42(9): 2858-2864.
[7]	Wanjun LIU, Jiaming WANG, Haicheng QU, Libing DONG, Xinyu CAO. Music genre classification algorithm based on attention spectral-spatial feature [J]. Journal of Computer Applications, 2022, 42(7): 2072-2077.
[8]	Bo LIU, Linbo QING, Zhengyong WANG, Mei LIU, Xue JIANG. Group activity recognition based on partitioned attention mechanism and interactive position relationship [J]. Journal of Computer Applications, 2022, 42(7): 2052-2057.
[9]	Juan WANG, Xuliang YUAN, Minghu WU, Liquan GUO, Zishan LIU. Real-time semantic segmentation method based on squeezing and refining network [J]. Journal of Computer Applications, 2022, 42(7): 1993-2000.
[10]	Rongyuan CHEN, Jianmin YAO, Qun YAN, Zhixian LIN. Video playback speed recognition based on deep neural network [J]. Journal of Computer Applications, 2022, 42(7): 2043-2051.
[11]	Guangkai LIAO, Zheng ZHANG, Zhiguo SONG. Convolutional network-based vehicle re-identification combining wavelet features and attention mechanism [J]. Journal of Computer Applications, 2022, 42(6): 1876-1883.
[12]	Zhen QU, Kunting LI, Zhixi FENG. Remote sensing image scene classification based on effective channel attention [J]. Journal of Computer Applications, 2022, 42(5): 1431-1439.
[13]	Mo LI, Tianliang LU, Ziheng XIE. Android malware family classification method based on code image integration [J]. Journal of Computer Applications, 2022, 42(5): 1490-1499.
[14]	Pengwei LIU, Yuan GAO, Pinle QIN, Zhe YIN, Lifang WANG. Medical MRI image super-resolution reconstruction based on multi-receptive field generative adversarial network [J]. Journal of Computer Applications, 2022, 42(3): 938-945.
[15]	Hong LI, Junying ZOU, Xicheng TAN, Guiyang LI. Multi-attention fusion network for medical image segmentation [J]. Journal of Computer Applications, 2022, 42(12): 3891-3899.

Pedestrian trajectory prediction based on multi-head soft attention graph convolutional network

基于多头软注意力图卷积网络的行人轨迹预测

RichHTML

PDF

PDF (Mobile)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 30

Related Articles 15

Recommended Articles

Metrics