Action recognition algorithm for ADHD patients using skeleton and 3D heatmap

doi:10.11772/j.issn.1001-9081.2024091304

Journal of Computer Applications ›› 2025, Vol. 45 ›› Issue (9): 3036-3044.DOI: 10.11772/j.issn.1001-9081.2024091304

• Frontier and comprehensive applications • Previous Articles

Action recognition algorithm for ADHD patients using skeleton and 3D heatmap

Chao SHI¹, Yuxin ZHOU¹, Qian FU¹, Wanyu TANG¹, Ling HE¹, Yuanyuan LI²()

^1.College of Biomedical Engineering，Sichuan University，Chengdu Sichuan 610065，China
^2.Mental Health Center of West China Hospital，Sichuan University，Chengdu Sichuan 610041，China

Received:2024-09-14 Revised:2025-01-15 Accepted:2025-01-24 Online:2025-03-24 Published:2025-09-10
Contact: Yuanyuan LI
About author:SHI Chao， born in 1997， M. S. candidate. His research interests include image processing.
ZHOU Yuxin， born in 2003. Her research interests include artificial intelligence.
FU Qian， born in 2001， M. S. candidate. Her research interests include deep learning.
TANG Wanyu， born in 2001， M.S. candidate. His research interests include video image processing.
HE Ling， born in 1981， Ph. D.， associate professor. Her research interests include medical signal processing， medical image processing.
Supported by:
Science and Technology Program of Sichuan Province(2023YFS0290)

基于骨架和3D热图的注意缺陷多动障碍患者动作识别算法

石超¹, 周昱昕¹, 扶倩¹, 唐万宇¹, 何凌¹, 李元媛²()

^1.四川大学生物医学工程学院，成都 610065
^2.四川大学华西医院心理卫生中心，成都 610041

通讯作者: 李元媛
作者简介:石超（1997—），男，贵州铜仁人，硕士研究生，主要研究方向：图像处理
周昱昕（2003—），女，广西桂林人，主要研究方向：人工智能
扶倩（2001—），女，四川成都人，硕士研究生，主要研究方向：深度学习
唐万宇（2001—），男，四川自贡人，硕士研究生，主要研究方向：视频图像处理
何凌（1981—），女，四川成都人，副教授，博士，主要研究方向：医学信号处理、医学图像处理
基金资助:
四川省科技计划项目(2023YFS0290)

Abstract

Abstract:

Attention Deficit Hyperactivity Disorder （ADHD） is a neurodevelopmental disorder common in childhood， characterized by inattention， hyperactivity， and impulsivity， often exhibiting specific motion patterns. Traditional action recognition algorithms have problems such as low recognition accuracy and slow response when handling these specific actions. To address these issues， an action recognition algorithm for ADHD patients using skeleton and 3D heatmap was proposed， and spatial relationships between joints were represented using Gaussian distribution precisely， which preserved spatio-temporal information effectively. To overcome the limitations of single-modal data， a multimodal integration method based on skeleton and 3D heatmap was introduced. At the same time， the output features of Short 3D-CNN （3D Convolutional Neural Network） and Adaptive Graph Convolutional Network （AGCN） were fused to fully exploit the advantages of both modalities， thereby improving action recognition performance. Experimental results on the ADHD patient dataset collected by Mental Health Center of West China Hospital， Sichuan University， show that the proposed algorithm achieves the Top-1 recognition accuracy of 0.860 4 and the Top-5 recognition accuracy of 0.987 3 for eight different types of actions. Additionally， an automatic ADHD classification algorithm based on action types was proposed， which classified ADHD into head and facial action type， trunk action type， and limb action type， achieving the recognition accuracy of 75% and the response time of 5 seconds. Compared with two-stream AGCN （2s-AGCN） and PoseConv3D， the proposed algorithm demonstrates higher recognition accuracy in complex action scenarios， providing a new technical approach for personalized ADHD intervention.

Key words: Attention Deficit Hyperactivity Disorder (ADHD), action recognition, skeleton data, Graph Convolutional Network (GCN), 3D Convolutional Neural Network (CNN)

摘要：

注意缺陷多动障碍（ADHD）是一种常见于儿童期的神经发育障碍，以注意力不集中、多动和冲动为主要特征，常表现出特定的动作模式。传统的动作识别算法在处理这些特定动作时存在识别准确率低和响应慢等问题。为解决这些问题，提出基于骨架和3D热图的注意缺陷多动障碍患者动作识别算法，并通过高斯分布精确地表示关节点间的空间关系，以有效地保留时空信息。针对单一模态数据的限制，引入基于骨架和3D热图的多模态集成方法。同时，通过融合Short 3D-CNN（3D Convolutional Neural Network）和自适应图卷积网络（AGCN）的输出特征，充分利用两种模态数据的优势，从而提升动作识别性能。在四川大学华西医院心理卫生中心采集的ADHD患者数据集上的实验结果表明，对于8种不同类型的动作，所提算法的Top-1识别准确率为0.860 4，Top-5识别准确率为0.987 3。此外，提出基于动作类型的ADHD自动分型算法，该算法将ADHD分型为头面部体动型、躯干体动型和四肢体动型，它的识别准确率为75%，响应时间为5 s。与2s-AGCN（two-stream AGCN）和PoseConv3D相比，所提算法在复杂动作场景下具有更高的识别精度，为ADHD的个性化干预提供了新的技术手段。

关键词: 注意缺陷多动障碍, 动作识别, 骨架数据, 图卷积网络, 3D卷积神经网络

CLC Number:

TP391

Chao SHI, Yuxin ZHOU, Qian FU, Wanyu TANG, Ling HE, Yuanyuan LI. Action recognition algorithm for ADHD patients using skeleton and 3D heatmap[J]. Journal of Computer Applications, 2025, 45(9): 3036-3044.

石超, 周昱昕, 扶倩, 唐万宇, 何凌, 李元媛. 基于骨架和3D热图的注意缺陷多动障碍患者动作识别算法[J]. 《计算机应用》唯一官方网站, 2025, 45(9): 3036-3044.

Figures/Tables 12

Fig. 1 Flow of proposed algorithm

Tab. 1 Input and output dimension changes for each module

模块/步骤	输入维度	输出维度	说明
视频输入	$(T, H, W, C)$	$(T, H, W, C)$	T=60为帧数，H=W=224、C=3分别为帧的高度、宽度和通道数
2D骨架提取	$(T, H, W, C)$	$(T, P, H, W)$	P=17/25/133为关键点数量
3D热图构建	$(T, P, H, W)$	$(K, T, H, W)$	K=17/25/133，即每个关节对应一个高斯热图
3D-CNN特征提取	$(K, T, H, W)$	$(T, D 1)$	提取视频的时空特征，D₁=256为特征维度
GCN特征提取	$(T, K, 2)$	$(T, D 2)$	提取骨架序列的时空特征，D₂=256为特征维度
MLFF-1特征融合	$(T, D 1) + (T, D 2)$	$(T, D 3)$	对3D-CNN特征和GCN特征进行加权融合，其中，D₃=256
MLFF-2特征融合	$C o n c a t (T, D 1) + (T, D 2)$	$(T, D 4)$	对3D-CNN特征和GCN特征进行拼接，其中，D₄=256
MLFF-3特征融合	$T r a n s (T, D 1) + (T, D 2)$	$(D 5)$	D₅=256为将特征进行拼接并使用Transformer进行进一步融合
分类输出	$(D 5)$	8	输出ADHD动作类别

Tab. 1 Input and output dimension changes for each module

模块/步骤	输入维度	输出维度	说明
视频输入	$(T, H, W, C)$	$(T, H, W, C)$	T=60为帧数，H=W=224、C=3分别为帧的高度、宽度和通道数
2D骨架提取	$(T, H, W, C)$	$(T, P, H, W)$	P=17/25/133为关键点数量
3D热图构建	$(T, P, H, W)$	$(K, T, H, W)$	K=17/25/133，即每个关节对应一个高斯热图
3D-CNN特征提取	$(K, T, H, W)$	$(T, D 1)$	提取视频的时空特征，D₁=256为特征维度
GCN特征提取	$(T, K, 2)$	$(T, D 2)$	提取骨架序列的时空特征，D₂=256为特征维度
MLFF-1特征融合	$(T, D 1) + (T, D 2)$	$(T, D 3)$	对3D-CNN特征和GCN特征进行加权融合，其中，D₃=256
MLFF-2特征融合	$C o n c a t (T, D 1) + (T, D 2)$	$(T, D 4)$	对3D-CNN特征和GCN特征进行拼接，其中，D₄=256
MLFF-3特征融合	$T r a n s (T, D 1) + (T, D 2)$	$(D 5)$	D₅=256为将特征进行拼接并使用Transformer进行进一步融合
分类输出	$(D 5)$	8	输出ADHD动作类别

Fig. 2 Flow of human skeleton extraction algorithm

Fig. 3 Flow of Short-3D-CNN algorithm

Fig. 4 Adaptive graph convolutional block

Fig. 5 Training loss and Top-1 accuracy curve

Tab. 2 Action type recognition accuracy of 3D-GCN for ADHD patients under different fusion strategies

MLFF	Top-1准确率	Top-5准确率
MLFF-1（1∶1）	0.856 0	0.987 9
MLFF-1（2∶1）	0.859 8	0.986 7
MLFF-1（1∶2）	0.838 8	0.987 8
MLFF-2	0.854 1	0.987 9
MLFF-3	0.860 4	0.987 3

Tab. 3 Action type recognition performance for ADHD patients of different deep learning algorithms

算法	Top-1准确率	Top-5准确率	参数量/10⁶
ST-GCN^［27］	0.811 5	0.994 8	3.10
MS-G3D^［23］	0.842 8	0.989 2	14.28
CTR-GCN^［22］	0.842 6	0.986 7	1.95
AGCN^［28］	0.832 5	0.980 3	2.80
ST-GCN++^［24］	0.814 7	0.984 8	1.40
2s-AGCN^［28］	0.843 1	0.991 8	3.50
PoseConv3D^［26］	0.847 1	0.991 1	2.00
3D-GCN	0.860 4	0.987 3	2.46

Fig. 6 Performance of three algorithms on different action types in ADHD Dataset

Fig. 7 Visualization of categories in ADHD action dataset

Fig. 8 Visualization of eight types of actions of ADHD patients using t-SNE

Tab. 4 Comparison of action recognition performance of different deep learning algorithms on NTU RGB+D 60 dataset

算法	Top-1准确率	Top-5准确率	参数量/10⁶
ST-GCN^［27］	0.889 5	0.987 8	3.10
CTR-GCN^［23］	0.896 0	0.989 3	1.95
MS-G3D^［23］	0.913 0	0.993 8	14.28
ST-GCN++^［24］	0.892 6	0.984 8	1.39
AGCN^［28］	0.886 0	0.985 1	3.50
2s-AGCN^［28］	0.919 5	0.992 6	3.50
PoseConv3D^［26］	0.934 7	0.995 4	2.00
3D-GCN	0.942 4	0.989 1	2.46

References 28

[1]	ZHENG Y， LI R， LI S， et al. A review on serious games for ADHD［EB/OL］. ［2024-08-10］. .
[2]	赵健翔，吴振起，王雪峰，等. 基于机器学习的注意力缺陷多动障碍风险预测研究［J］. 中国中西医结合儿科学， 2024， 16（2）： 130-136.
	ZHAO J X， WU Z Q， WANG X F， et al. Risk prediction of attention deficit hyperactivity disorder based on machine learning ［J］. Chinese Pediatrics of Integrated Traditional and Western Medicine， 2024， 16（2）： 130-136.
[3]	卜晓鸥，王耀，杜亚雯，等. 机器学习在发展性阅读障碍儿童早期筛查中的应用［J］. 心理科学进展， 2023， 31（11）： 2092-2105.
	BU X O， WANG Y， DU Y W， et al. Application of machine learning in early screening of children with dyslexia ［J］. Advances in Psychological Science， 2023， 31（11）： 2092-2105.
[4]	罗杰，何凡，郑毅. 视频游戏应用于儿童注意缺陷多动障碍评估与治疗的系统性综述［J］. 发育医学电子杂志， 2023， 11（6）： 401-410.
	LUO J， HE F， ZHENG Y. Video games for the assessment and treatment of attention deficit hyperactivity disorder in children： a systematic review ［J］. Journal of Developmental Medicine （Electronic Version）， 2023， 11（6）： 401-410.
[5]	向维. ADHD诊疗新进展综述［J］. 临床医学前沿， 2023， 5（6）： 138-140.
	XIANG W. A review on new progress in ADHD diagnosis and treatment ［J］. Frontiers of Clinical Medicine， 2023， 5（6）： 138-140.
[6]	JAISWAL S， VALSTAR M F， GILLOTT A， et al. Automatic detection of ADHD and ASD from expressive behaviour in RGBD data ［C］// Proceedings of the 12th IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway： IEEE， 2017： 762-769.
[7]	DENG S， PRASSE P， REICH D R， et al. Detection of ADHD based on eye movements during natural viewing ［C］// Proceedings of the 2022 European Conference on Machine Learning and Knowledge Discovery in Databases， LNCS 13718. Cham： Springer， 2023： 403-418.
[8]	OUYANG C S， CHIU Y H， CHIANG C T， et al. Evaluating therapeutic effects of ADHD medication objectively by movement quantification with a video-based skeleton analysis ［J］. International Journal of Environmental Research and Public Health， 2021， 18（17）： No.9363.
[9]	DEY S， RAO A R， SHAH M. Exploiting the brain’s network structure for automatic identification of ADHD subjects ［EB/OL］. ［2024-05-12］. .
[10]	SIMS C. Highly accurate FMRI ADHD classification using time distributed multi modal 3D-CNNs ［EB/OL］. ［2024-02-13］..
[11]	ULUYAGMUR-OZTURK M， ARMAN A R， YILMAZ S S， et al. ADHD and ASD classification based on emotion recognition data［C］// Proceedings of the 15th IEEE International Conference on Machine Learning and Applications. Piscataway： IEEE， 2016： 810-813.
[12]	AMADO-CABALLERO P， CASASECA-DE-LA-HIGUERA P， ALBEROLA-LOPEZ S， et al. Objective ADHD diagnosis using convolutional neural networks over daily-life activity records ［J］. IEEE Journal of Biomedical and Health Informatics， 2020， 24（9）： 2690-2700.
[13]	ZHANG-JAMES Y， RAZAVI A S， HOOGMAN M， et al. Machine learning and MRI-based diagnostic models for ADHD： are we there yet？［J］. Journal of Attention Disorders， 2023， 27（4）： 335-353.
[14]	ALCHALABI A E， SHIRMOHAMMADI S， EDDIN A N， et al. FOCUS： detecting ADHD patients by an EEG-based serious game［J］. IEEE Transactions on Instrumentation and Measurement， 2018， 67（7）： 1512-1520.
[15]	PENG J， DEBNATH M， BISWAS A K. Efficacy of novel summation-based synergetic artificial neural network in ADHD diagnosis ［J］. Machine Learning with Applications， 2021， 6： No.100120.
[16]	OCHAB J K， GERC K， FAFROWICZ M， et al. Classifying attention deficit hyperactivity disorder in children with non-linearities in actigraphy ［EB/OL］. ［2024-05-03］..
[17]	CHOI M T， YEOM J， SHIN Y， et al. Robot-assisted ADHD screening in diagnostic process ［J］. Journal of Intelligent and Robotic Systems， 2019， 95（2）： 351-363.
[18]	ANDRIKOPOULOS D， VASSILIOU G， FATOUROS P， et al. Machine learning-enabled detection of attention-deficit/hyperactivity disorder with multimodal physiological data： a case-control study ［J］. BMC Psychiatry， 2024， 24： No.547.
[19]	ALSHARIF N， AL-ADHAILEH M H， ALSUBARI S N， et al. ADHD diagnosis using text features and predictive machine learning and deep learning algorithms ［J］. Journal of Disability Research， 2024， 3（7）： No.0082.
[20]	LIU J， SHAHROUDY A， XU D， et al. Skeleton-based action recognition using spatio-temporal LSTM network with trust gates［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018， 40（12）： 3007-3021.
[21]	吕艾为. 基于机器学习的ADHD脑机接口辅助诊断的相关问题研究［D］. 鞍山：辽宁科技大学， 2022.
	LYU A W. Research on related problems of ADHD brain-computer interface aided diagnosis based on machine learning ［D］. Anshan： University of Science and Technology Liaoning， 2022.
[22]	CHEN Y， ZHANG Z， YUAN C， et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2021： 13339-13348.
[23]	LIU Z， ZHANG H， CHEN Z， et al. Disentangling and unifying graph convolutions for skeleton-based action recognition ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 140-149.
[24]	DUAN H， WANG J， CHEN K， et al. PYSKL： towards good practices for skeleton action recognition ［C］// Proceedings of the 30th ACM International Conference on Multimedia. New York： ACM， 2022： 7351-7354.
[25]	LI W， LIU M， LIU H， et al. GraphMLP： a graph MLP-like architecture for 3D human pose estimation ［J］. Pattern Recognition， 2025， 158： No.110925.
[26]	DUAN H， ZHAO Y， CHEN K， et al. Revisiting skeleton-based action recognition ［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2022： 2959-2968.
[27]	YAN S， XIONG Y， LIN D. Spatial temporal graph convolutional networks for skeleton-based action recognition ［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto： AAAI Press， 2018： 7444-7452.
[28]	SHI L， ZHANG Y， CHENG J， et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 12018-12027.

Action recognition algorithm for ADHD patients using skeleton and 3D heatmap

基于骨架和3D热图的注意缺陷多动障碍患者动作识别算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 28

Related Articles 15

Recommended Articles

Metrics

[1]	Ziliang LI, Guangli ZHU, Yulei ZHANG, Jiajia LIU, Yixuan JIAO, Shunxiang ZHANG. Aspect-based sentiment analysis model integrating syntax and sentiment knowledge [J]. Journal of Computer Applications, 2025, 45(6): 1724-1731.
[2]	Quan WANG, Qixiang LU, Pei SHI. Multi-graph diffusion attention network for traffic flow prediction [J]. Journal of Computer Applications, 2025, 45(5): 1472-1479.
[3]	Man CHEN, Xiaojun YANG, Huimin YANG. Pedestrian trajectory prediction based on graph convolutional network and endpoint induction [J]. Journal of Computer Applications, 2025, 45(5): 1480-1487.
[4]	Yufei LONG, Yuchen MOU, Ye LIU. Multi-source data representation learning model based on tensorized graph convolutional network and contrastive learning [J]. Journal of Computer Applications, 2025, 45(5): 1372-1378.
[5]	Weichao DANG, Chujun SONG, Gaimei GAO, Chunxia LIU. Multi-behavior recommendation based on cascading residual graph convolutional network [J]. Journal of Computer Applications, 2025, 45(4): 1223-1231.
[6]	Kun FU, Shicong YING, Tingting ZHENG, Jiajie QU, Jingyuan CUI, Jianwei LI. Graph data augmentation method for few-shot node classification [J]. Journal of Computer Applications, 2025, 45(2): 392-402.
[7]	Lifang WANG, Jingshuang WU, Pengliang YIN, Lihua HU. Action recognition algorithm based on attention mechanism and energy function [J]. Journal of Computer Applications, 2025, 45(1): 234-239.
[8]	Chuanlin PANG, Rui TANG, Ruizhi ZHANG, Chuan LIU, Jia LIU, Shibo YUE. Distributed power allocation algorithm based on graph convolutional network for D2D communication systems [J]. Journal of Computer Applications, 2024, 44(9): 2855-2862.
[9]	Guixiang XUE, Hui WANG, Weifeng ZHOU, Yu LIU, Yan LI. Port traffic flow prediction based on knowledge graph and spatio-temporal diffusion graph convolutional network [J]. Journal of Computer Applications, 2024, 44(9): 2952-2957.
[10]	Huanhuan LI, Tianqiang HUANG, Xuemei DING, Haifeng LUO, Liqing HUANG. Public traffic demand prediction based on multi-scale spatial-temporal graph convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2065-2072.
[11]	Shibin LI, Jun GONG, Shengjun TANG. Semi-supervised heterophilic graph representation learning model based on Graph Transformer [J]. Journal of Computer Applications, 2024, 44(6): 1816-1823.
[12]	Longtao GAO, Nana LI. Aspect sentiment triplet extraction based on aspect-aware attention enhancement [J]. Journal of Computer Applications, 2024, 44(4): 1049-1057.
[13]	Xianfeng YANG, Yilei TANG, Ziqiang LI. Aspect-level sentiment analysis model based on alternating‑attention mechanism and graph convolutional network [J]. Journal of Computer Applications, 2024, 44(4): 1058-1064.
[14]	Kaitian WANG, Qing YE, Chunlei CHENG. Classification method for traditional Chinese medicine electronic medical records based on heterogeneous graph representation [J]. Journal of Computer Applications, 2024, 44(2): 411-417.
[15]	Zucheng WU, Xiaojun WU, Tianyang XU. Image-text retrieval model based on intra-modal fine-grained feature relationship extraction [J]. Journal of Computer Applications, 2024, 44(12): 3776-3783.