Data augmentation method for abnormal elevator passenger behaviors based on dynamic graph convolutional network

doi:10.11772/j.issn.1001-9081.2024101445

Journal of Computer Applications ›› 2025, Vol. 45 ›› Issue (10): 3187-3194.DOI: 10.11772/j.issn.1001-9081.2024101445

• Artificial intelligence • Previous Articles

Data augmentation method for abnormal elevator passenger behaviors based on dynamic graph convolutional network

Shixiong KUANG, Junbo YAO, Jiawei LU, Qibing WANG, Gang XIAO()

College of Mechanical and Electrical Engineering，China Jiliang University，Hangzhou Zhejiang 310018，China

Received:2024-10-14 Revised:2024-12-30 Online:2025-01-06 Published:2025-10-10
Contact: Gang XIAO
About author:KUANG Shixiong， born in 2000， M. S. candidate. His research interests include data augmentation， pattern recognition.
YAO Junbo， born in 2000， M. S. candidate. His research interests include signal processing， fault diagnosis.
LU Jiawei， born in 1981， Ph. D. candidate， associate professor. His research interests include knowledge graph， deep learning.
WANG Qibing， born in 1978， Ph. D.， professor. His research interests include digital design and manufacturing， intelligence of special equipment.
XIAO Gang， born in 1965， Ph. D.， professor. His research interests include fault diagnosis， intelligent manufacturing.
Supported by:
Zhejiang Province “Vanguard” Research and Development Breakthrough Program(2023C01022);Zhejiang Province “Leading Sparrow” Research and Development Breakthrough Program(2023C01215);Huzhou City Key Research and Development Science and Technology Program(2022ZD2019)

基于动态图卷积网络的电梯乘客异常行为数据增强方法

况世雄, 姚俊波, 陆佳炜, 王琪冰, 肖刚()

中国计量大学机电工程学院，杭州 310018

通讯作者: 肖刚
作者简介:况世雄（2000—），男，湖北武汉人，硕士研究生，主要研究方向：数据增强、模式识别
姚俊波（2000—），男，浙江宁波人，硕士研究生，主要研究方向：信号处理、故障诊断
陆佳炜（1981—），男，浙江湖州人，副教授，博士研究生，主要研究方向：知识图谱、深度学习
王琪冰（1978—），男，浙江绍兴人，教授，博士，主要研究方向：数字化设计与制造、特种设备智能化
肖刚（1965—），男，浙江上虞人，教授，博士，CCF会员，主要研究方向：故障诊断、智能制造。 Email:xg@zjut.edu.cn
基金资助:
浙江省“尖兵”研发攻关计划项目(2023C01022);浙江省“领雁”研发攻关计划项目(2023C01215);湖州市重点研发科技计划项目(2022ZD2019)

Abstract

Abstract:

The problems of low accuracy and poor generalization performance in recognizing abnormal behaviors of elevator passengers are due to the lack of sufficient diverse abnormal behavior data. To address these issues， a Dynamic Graph Convolutional Network-based Behavior data Augmentation （DGCN-BA） method was proposed. Firstly， a dynamic graph convolutional network was constructed to capture spatial relationships and motion correlations among different human joints in the behaviors of elevator passengers. Secondly， these features were utilized to enhance pose data， thereby generating richer and more reasonable pose sequences. Finally， the pose sequences were used to construct human actions in a virtual elevator scene， and lot of abnormal behavior video data for elevator passengers were generated. To validate the effectiveness of DGCN-BA， experiments were conducted on public datasets Human3.6M， 3DHP， MuPoTS-3D， and a self-constructed dataset. Experimental results show that compared to data augmentation methods JMDA （Joint Mixing Data Augmentation） and DDPMs （Denoising Diffusion Probabilistic Models）， DGCN-BA reduces the Mean Per Joint Position Error （MPJPE） on the Human3.6M dataset by 2.9 mm and 1.5 mm， respectively. It can be seen that DGCN-BA can complete pose estimation tasks more effectively， generates diverse and reasonable abnormal behavior data， and improves the recognition effect of video-based elevator passenger abnormal behaviors significantly.

Key words: graph convolutional network, data augmentation, behavior recognition, pose estimation, elevator passenger

摘要：

目前，因缺乏足够多样化的异常行为数据，电梯乘客异常行为的识别方法存在准确率不高和泛化性能较差的问题。为解决这个问题，提出一种基于动态图卷积网络的行为数据增强方法（DGCN-BA）。首先，构建一种动态图卷积网络，用于捕捉电梯乘客行为中不同人体关节之间的空间关系和运动关联性；其次，利用这些特征进行姿势增强，获取更丰富和合理的姿势序列；最后，利用姿势序列在虚拟电梯场景中构建人物动作，生成大量电梯乘客异常行为视频数据。在公开数据集Human3.6M、3DHP和MuPoTS-3D，以及自建数据集上验证DGCN-BA的有效性。实验结果表明，相较于JMDA（Joint Mixing Data Augmentation）、DDPMs（Denoising Diffusion Probabilistic Models）数据增强方法，DGCN-BA在Human3.6M数据集上的平均每个关节位置误差（MPJPE）分别降低了2.9 mm和1.5 mm。可见，DGCN-BA能够更有效完成姿势估计任务，生成合理多样的异常行为数据，从而明显改善基于视频的电梯乘客异常行为识别效果。

关键词: 图卷积网络, 数据增强, 行为识别, 姿势估计, 电梯乘客

CLC Number:

TP391.41

Shixiong KUANG, Junbo YAO, Jiawei LU, Qibing WANG, Gang XIAO. Data augmentation method for abnormal elevator passenger behaviors based on dynamic graph convolutional network[J]. Journal of Computer Applications, 2025, 45(10): 3187-3194.

况世雄, 姚俊波, 陆佳炜, 王琪冰, 肖刚. 基于动态图卷积网络的电梯乘客异常行为数据增强方法[J]. 《计算机应用》唯一官方网站, 2025, 45(10): 3187-3194.

Figures/Tables 12

Fig. 1 Structure of human skeletal keypoints

Fig. 2 Spatial correlation of human joints

Fig. 3 Temporal correlation of human joints

Fig. 4 Structure of dynamic graph convolutional network

Fig. 5 Pose augmentation operation

Fig. 6 Virtual elevator scene

Tab. 1 Pose estimation results of different data augmentation methods based on GraFormer

增强方法	MPJPE
增强方法	指引	讨论	吃饭	问候	打电话	拍照	摆姿势	购物	坐下	坐着	抽烟	等待	遛狗	走路	一起走	平均
无	45.2	50.8	48.0	50.0	54.9	65.0	48.2	47.1	60.2	70.0	51.6	48.7	54.1	39.7	43.1	51.8
Skelbumentations	41.9	45.9	43.2	47.5	51.3	57.8	43.9	45.7	57.0	61.9	47.7	46.8	49.0	34.6	42.8	47.8
JMDA	40.0	46.8	41.7	45.8	49.4	58.4	41.4	46.0	53.4	59.3	47.8	44.9	48.3	31.8	43.0	46.5
DDPMs	37.9	44.2	42.1	46.1	48.9	56.5	40.2	44.3	52.8	57.9	45.5	45.8	44.6	29.9	39.7	45.1
DGCN-BA	37.3	43.1	42.9	43.9	47.4	57.9	37.2	43.9	50.0	53.2	44.3	42.0	45.1	26.8	38.6	43.6

Tab. 2 Pose estimation results before and after using DGCN-BA on various methods

方法	3DHP				MuPoTS-3D
	MPJPE		PA-MPJPE		MPJPE		PA-MPJPE
	未使用	使用	未使用	使用	未使用	使用	未使用	使用
SemGCN	110.3	99.7	74.2	66.5	160.3	148.7	112.0	102.6
Baseline	102.5	91.8	72.9	67.4	129.6	113.5	78.7	69.3
TCN	100.3	90.9	71.5	63.1	135.2	118.9	87.1	76.7

Tab. 3 Performance of different combinations of dynamic spatio-temporal graph convolution modules

方法	$Z k$	$Z k'$	Human3.6M
方法	$Z k$	$Z k'$	MPJPE	PA-MPJPE
实验组A			51.0	36.7
实验组B	√		48.4	33.2
实验组C		√	45.2	31.0
DGCN-BA	√	√	43.6	29.8

Tab. 3 Performance of different combinations of dynamic spatio-temporal graph convolution modules

方法	$Z k$	$Z k'$	Human3.6M
方法	$Z k$	$Z k'$	MPJPE	PA-MPJPE
实验组A			51.0	36.7
实验组B	√		48.4	33.2
实验组C		√	45.2	31.0
DGCN-BA	√	√	43.6	29.8

Tab. 4 Performance of different combinations of pose augmentation modules

方法	Ⅰ	Ⅱ	Ⅲ	Human3.6M		MuPoTS-3D
方法	Ⅰ	Ⅱ	Ⅲ	MPJPE	PA-MPJPE	MPJPE	PA-MPJPE
实验组D				51.8	37.1	82.7	69.1
实验组E	√		√	45.5	30.8	76.3	63.2
实验组F		√	√	47.0	32.7	78.1	65.0
实验组G	√	√		46.5	31.6	74.7	60.6
DGCN-BA	√	√	√	43.6	29.8	72.9	59.5

Fig. 7 Data augmentation for abnormal behaviors of elevator passengers

Tab. 5 Behavior recognition results before and after using data augmentation framework on various methods

方法	准确率
	跌倒		扒门		踹门		暴力事件		平均
	原始	增强后	原始	增强后	原始	增强后	原始	增强后	原始	增强后
文献［33］方法	96.2	97.9	93.4	95.1	91.9	93.6	89.3	92.2	92.7	94.7
文献［34］方法	96.9	98.1	95.1	96.0	94.7	96.2	90.9	94.5	94.4	96.2
文献［35］方法	97.9	99.0	95.4	96.5	97.8	98.3	95.7	97.0	96.7	97.7

References 35

[1]	张连学，杜平虎.试谈电梯使用环节及相关环节的安全责任［J］. 中国电梯， 2020， 31（13）：40-43.
	ZHANG L X， DU P H. Discussion on safety responsibility of elevator use and related links［J］. China Elevator， 2020， 31（13）： 40-43.
[2]	ARROYO R， YEBES J J， BERGASA L M， et al. Expert video-surveillance system for real-time detection of suspicious behaviors in shopping malls［J］. Expert systems with Applications， 2015， 42（21）： 7991-8005.
[3]	吕蕾，庞辰. 基于图卷积网络的人体骨架行为识别方法综述［J］. 山东师范大学学报（自然科学版）， 2024， 39（3）：210-232.
	LYU L， PANG C. A review of human skeleton action recognition methods based on graph convolutional network［J］. Journal of Shandong Normal University （Natural Sciences Edition）， 2024， 39（3）： 210-232.
[4]	REN B， LIU M， DING R， et al. A survey on 3D skeleton-based action recognition using learning method［J］. Cyborg and Bionic Systems， 2024， 5： No.0100.
[5]	XU Q， ZHENG W， SONG Y， et al. Scene image and human skeleton-based dual-stream human action recognition［J］. Pattern Recognition Letters， 2021， 148： 136-145.
[6]	DAI C， LU S， LIU C， et al. A light-weight skeleton human action recognition model with knowledge distillation for edge intelligent surveillance applications［J］. Applied Soft Computing， 2024， 151： No.111166.
[7]	YANG M， WU C， GUO Y， et al. Transformer-based deep learning model and video dataset for unsafe action identification in construction projects［J］. Automation in Construction， 2023， 146： No.104703.
[8]	ALI R， HUTOMO I S， VAN L D， et al. A skeleton-based view-invariant framework for human fall detection in an elevator［C］// Proceedings of the 2022 IEEE International Conference on Industrial Technology. Piscataway： IEEE， 2022： 1-6.
[9]	LAN S， JIANG S， LI G. An elevator passenger behavior recognition method based on two-stream convolution neural network［J］. Journal of Physics： Conference Series， 2021， 1955： No.012089.
[10]	YANG J， WAN L， XU W， et al. 3D human pose estimation from a single image via exemplar augmentation［J］. Journal of Visual Communication Image Representation， 2019， 59： 371-379.
[11]	DU S， YUAN Z， LAI P， et al. JoyPose： jointly learning evolutionary data augmentation and anatomy-aware global-local representation for 3D human pose estimation［J］. Pattern Recognition Letters， 2024， 147： No.110116.
[12]	ALHAIJA H ABU， MUSTIKOVELA S K， MESCHEDER L， et al. Augmented reality meets computer vision： efficient data generation for urban driving scenes［J］. International Journal of Computer Vision， 2018， 126（9）： 961-972.
[13]	PURKAIT P， ZHAO C， ZACH C. SPP-Net： deep absolute pose regression with synthetic views［EB/OL］. ［2024-10-11］..
[14]	LI S， KE L， PRATAMA K， et al. Cascaded deep monocular 3D human pose estimation with evolutionary training data［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 6172-6182.
[15]	GONG K， ZHANG J， FENG J. PoseAug： a differentiable pose augmentation framework for 3D human pose estimation［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 8571-8580.
[16]	ZHANG C， ZHU L， ZHANG S， et al. PAC-GAN： an effective pose augmentation scheme for unsupervised cross-view person re-identification［J］. Neurocomputing， 2020， 387： 22-39.
[17]	SHAH A， ROY A， SHAH K， et al. HaLP： hallucinating latent positives for skeleton-based self-supervised learning of actions［C］// Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2023： 18846-18856.
[18]	ZHANG J， WANG Y， ZHOU Z， et al. Learning dynamical human-joint affinity for 3D pose estimation in videos［J］. IEEE Transactions on Image Processing， 2021， 30： 7914-7925.
[19]	WANDT B， ROSENHAHN B. RepNet： weakly supervised training of an adversarial reprojection network for 3D human pose estimation［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 7774-7783.
[20]	AKHTER I， BLACK M J. Pose-conditioned joint angle limits for 3D human pose reconstruction［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 1446-1455.
[21]	LI R， LI X， HENG P A， et al. PointAugment： an auto-augmentation framework for point cloud classification［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 6377-6386.
[22]	IONESCU C， PAPAVA D， OLARU V， et al. Human3.6M： large scale datasets and predictive methods for 3D human sensing in natural environments［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2014， 36（7）： 1325-1339.
[23]	MEHTA D， RHODIN H， CASAS D， et al. Monocular 3D human pose estimation in the wild using improved CNN supervision［C］// Proceedings of the 2017 International Conference on 3D Vision. Piscataway： IEEE， 2017： 506-516.
[24]	MEHTA D， SOTNYCHENKO O， MUELLER F， et al. Single-shot multi-person 3D pose estimation from monocular RGB［C］// Proceedings of the 2018 International Conference on 3D Vision. Piscataway： IEEE， 2018： 120-130.
[25]	CHEN Y， WANG Z， PENG Y， et al. Cascaded pyramid network for multi-person pose estimation［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7103-7112.
[26]	ZHAO W， WANG W， TIAN Y. GraFormer： graph-oriented transformer for 3D pose estimation［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2022： 20406-20415.
[27]	CORMIER M， SCHMID Y， BEYERER J. Enhancing skeleton-based action recognition in real-world scenarios through realistic data augmentation［C］// Proceedings of the 2024 IEEE/CVF Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2024： 300-309.
[28]	XIANG L， WANG Z. Joint mixing data augmentation for skeleton-based action recognition［J］. ACM Transactions on Multimedia Computing， Communications and Applications， 2025， 21（4）： No.108.
[29]	JIANG Y， CHEN H， KO H. Spatial-temporal transformer-guided diffusion based data augmentation for efficient skeleton-based action recognition［EB/OL］. ［2024-11-20］..
[30]	ZHAO L， PENG X， TIAN Y， et al. Semantic graph convolutional networks for 3D human pose regression［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3420-3430.
[31]	MARTINEZ J， HOSSAIN R， ROMERO J， et al. A simple yet effective baseline for 3D human pose estimation［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2659-2668.
[32]	PAVLLO D， FEICHTENHOFER C， GRANGIER D， et al. 3D human pose estimation in video with temporal convolutions and semi-supervised training［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 7745-7754.
[33]	SHI Y， GUO B， XU Y， et al. Recognition of abnormal human behavior in elevators based on CNN［C］// Proceedings of the 26th International Conference on Automation and Computing. Piscataway： IEEE， 2021： 1-6.
[34]	LEI J， SUN W， FANG Y， et al. A model for detecting abnormal elevator passenger behavior based on video classification［J］. Electronics， 2024， 13（13）： No.2472.
[35]	SUN Z， XU B， WU D， et al. A real-time video surveillance and state detection approach for elevator cabs［C］// Proceedings of the 2019 International Conference on Control， Automation and Information Sciences. Piscataway： IEEE， 2019： 1-6.

Data augmentation method for abnormal elevator passenger behaviors based on dynamic graph convolutional network

基于动态图卷积网络的电梯乘客异常行为数据增强方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 35

Related Articles 15

Recommended Articles

Metrics

[1]	Chao SHI, Yuxin ZHOU, Qian FU, Wanyu TANG, Ling HE, Yuanyuan LI. Action recognition algorithm for ADHD patients using skeleton and 3D heatmap [J]. Journal of Computer Applications, 2025, 45(9): 3036-3044.
[2]	Chuang WANG, Lu YU, Jianwei CHEN, Cheng PAN, Wenbo DU. Review of open set domain adaptation [J]. Journal of Computer Applications, 2025, 45(9): 2727-2736.
[3]	Yiming LIANG, Jing FAN, Wenze CHAI. Multi-scale feature fusion sentiment classification based on bidirectional cross attention [J]. Journal of Computer Applications, 2025, 45(9): 2773-2782.
[4]	Biao ZHAO, Yuhua QIN, Rongkun TIAN, Yuehang HU, Fangrui CHEN. Dependency type and distance enhanced aspect based sentiment analysis model [J]. Journal of Computer Applications, 2025, 45(8): 2507-2514.
[5]	Chaoying JIANG, Qian LI, Ning LIU, Lei LIU, Lizhen CUI. Readmission prediction model based on graph contrastive learning [J]. Journal of Computer Applications, 2025, 45(6): 1784-1792.
[6]	Ziliang LI, Guangli ZHU, Yulei ZHANG, Jiajia LIU, Yixuan JIAO, Shunxiang ZHANG. Aspect-based sentiment analysis model integrating syntax and sentiment knowledge [J]. Journal of Computer Applications, 2025, 45(6): 1724-1731.
[7]	Daoquan LI, Zheng XU, Sihui CHEN, Jiayu LIU. Network traffic classification model integrating variational autoencoder and AdaBoost-CNN [J]. Journal of Computer Applications, 2025, 45(6): 1841-1848.
[8]	Yufei LONG, Yuchen MOU, Ye LIU. Multi-source data representation learning model based on tensorized graph convolutional network and contrastive learning [J]. Journal of Computer Applications, 2025, 45(5): 1372-1378.
[9]	Quan WANG, Qixiang LU, Pei SHI. Multi-graph diffusion attention network for traffic flow prediction [J]. Journal of Computer Applications, 2025, 45(5): 1472-1479.
[10]	Xueying LI, Kun YANG, Guoqing TU, Shubo LIU. Adversarial sample generation method for time-series data based on local augmentation [J]. Journal of Computer Applications, 2025, 45(5): 1573-1581.
[11]	Man CHEN, Xiaojun YANG, Huimin YANG. Pedestrian trajectory prediction based on graph convolutional network and endpoint induction [J]. Journal of Computer Applications, 2025, 45(5): 1480-1487.
[12]	Renjie TIAN, Mingli JING, Long JIAO, Fei WANG. Recommendation algorithm of graph contrastive learning based on hybrid negative sampling [J]. Journal of Computer Applications, 2025, 45(4): 1053-1060.
[13]	Haitao SUN, Jiayu LIN, Zuhong LIANG, Jie GUO. Data augmentation technique incorporating label confusion for Chinese text classification [J]. Journal of Computer Applications, 2025, 45(4): 1113-1119.
[14]	Weichao DANG, Chujun SONG, Gaimei GAO, Chunxia LIU. Multi-behavior recommendation based on cascading residual graph convolutional network [J]. Journal of Computer Applications, 2025, 45(4): 1223-1231.
[15]	Shiyue GUO, Jianwu DANG, Yangping WANG, Jiu YONG. 3D hand pose estimation combining attention mechanism and multi-scale feature fusion [J]. Journal of Computer Applications, 2025, 45(4): 1293-1299.