Body movement emotion recognition method based on emotional latent space learning and CLIP model

doi:10.11772/j.issn.1001-9081.2024040529

Journal of Computer Applications ›› 0, Vol. ›› Issue (): 44-49.DOI: 10.11772/j.issn.1001-9081.2024040529

• Artificial intelligence • Previous Articles Next Articles

Body movement emotion recognition method based on emotional latent space learning and CLIP model

Hong LUO¹, Yujie SHEN², Juanjuan CHEN³, Dan WANG³()

^1.China Mobile （Hangzhou） Information Technology Company Limited，Hangzhou Zhejiang 310023，China
^2.Hangzhou Institute of Technology，Xidian University，Hangzhou Zhejiang 311231，China
^3.State Key Laboratory of Integrated Services Networks （Xidian University），Xi’an Shaanxi 710071，China

Received:2024-04-28 Revised:2024-06-26 Accepted:2024-06-27 Online:2025-01-24 Published:2024-12-31
Contact: Dan WANG

基于情感隐空间学习与CLIP模型的身体动作情感识别方法

罗红¹, 慎煜杰², 陈娟娟³, 王丹³()

^1.中移（杭州）信息技术有限公司，杭州 310023
^2.西安电子科技大学杭州研究院，杭州 311231
^3.空天地一体化综合业务网全国重点实验室（西安电子科技大学），西安 710071

通讯作者: 王丹
作者简介:罗红（1977—），女，广西田林人，高级工程师，主要研究方向：AI交互、智能家居
慎煜杰（1999—），男，浙江湖州人，硕士研究生，主要研究方向：情感计算、身体动作情感识别
陈娟娟（1998—），女，甘肃定西人，硕士研究生，主要研究方向：多模态情感计算、情感驱动的多轮对话
王丹（1992—），女，甘肃陇南人，副教授，博士，主要研究方向：机器学习、多智能体强化学习、知识建模与计算。

Abstract

Abstract:

The key to body movement emotion recognition lies in extracting emotional features existed in human body movements. To solve the problems of poor emotional feature learning capability and difficulty in improving emotion recognition accuracy in existing models， a body movement emotion recognition method based on Emotional Latent Space Learning （ELSL） and Contrastive Language-Image Pre-training （CLIP） model was proposed. Firstly， CLIP model was introduced to improve the emotional feature learning capability of the model. Secondly， for the fine-grained multi-label emotion classification task， ELSL method was proposed. By learning discriminative mappings from emotional latent space to various subspaces， the subtle differences between emotion categories and the feature information beneficial to the classification of each emotion category in various emotional subspaces. Experiments were carried out on real-world open scenarios-oriented Body Language Dataset （BoLD） The results demonstrate that the proposed method makes use of the advantages of CLIP model and latent space learning in feature learning effectively， leading to significant performance improvement. In specific， compared to Movement Analysis Network （MANet）， the proposed method has a 1.08 percentage points increase in mean Average Precision （mAP） and a 1.32 percentage points improvement in mean Area Under Receiver Operating Characteristic Curve （mRA）.

Key words: body movement emotion recognition, Contrastive Language-Image Pre-training （CLIP） model, latent space learning, prompt learning, multi-label classification

摘要：

身体动作情感识别的关键是提取人物身体动作蕴含的情感特征。针对现有模型情感特征学习能力较差且情感识别精度难以提升的问题，提出一种基于情感隐空间学习与对比语言-图像预训练（CLIP）模型的身体动作情感识别方法。首先，引入CLIP模型，从而增强模型的情感特征学习能力。其次，针对细粒度多标签情感分类任务，提出情感隐空间学习（ELSL）方法。该方法能通过学习情感隐空间向各个子空间的判别映射，在各个情感子空间上捕获情感类别之间的细微差异和对各情感类别的分类有益的特征信息。在面向真实世界开放场景的肢体语言数据集（BoLD）上的实验结果表明，所提方法充分利用了CLIP模型与隐空间学习在特征学习上的优势，取得了显著的性能提升。具体地，相较于运动分析网络（MANet），所提方法的平均精度均值（mAP）提高了1.08个百分点，平均受试者工作特征曲线下方面积（mRA）提高了1.32个百分点。

关键词: 身体动作情感识别, 对比语言-图像预训练模型, 隐空间学习, 提示学习, 多标签分类

CLC Number:

TP391.4

Hong LUO, Yujie SHEN, Juanjuan CHEN, Dan WANG. Body movement emotion recognition method based on emotional latent space learning and CLIP model[J]. Journal of Computer Applications, 0, (): 44-49.

罗红, 慎煜杰, 陈娟娟, 王丹. 基于情感隐空间学习与CLIP模型的身体动作情感识别方法[J]. 《计算机应用》唯一官方网站, 0, (): 44-49.

Figures/Tables 10

References 18

1	LUO Y， YE J， ADAMS R B， Jr， et al. ARBEE： towards automated recognition of bodily expression of emotion in the wild［J］. International Journal of Computer Vision， 2020， 128（1）： 1-25.
2	沈志娟. 基于身体动作的情感识别研究［D］. 深圳：中国科学院大学（中国科学院深圳先进技术研究院）， 2020：14-15.
3	FILNTISIS P P， EFTHYMIOU N， KOUTRAS P， et al. Fusing body posture with facial expressions for joint recognition of affect in child-robot interaction［J］. IEEE Robotics and Automation Letters， 2019， 4（4）： 4011-4018.
4	MANNOCCHI I， LAMICHHANE K， CARLI M， et al. HEROES： a video-based human emotion recognition database［C］// Proceedings of the 10th European Workshop on Visual Information. Piscataway： IEEE， 2022：1-6.
5	RADFORD A， KIM J W， HALLACY C， et al. Learning transferable visual models from natural language supervision［C］// Proceedings of the 38th International Conference on Machine Learning. New York： JMLR.org， 2021： 8748-8763.
6	ZHOU K， YANG J， LOY C C， et al. Learning to prompt for vision-language models［J］. International Journal of Computer Vision， 2022， 130（9）： 2337-2348.
7	XU H， GHOSH G， HUANG P Y， et al. VideoCLIP： contrastive pre-training for zero-shot video-text understanding［C］// Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Stroudsburg： ACL， 2021： 6787-6800.
8	RASHEED H， KHATTAK M U， MAAZ M， et al. Fine-tuned clip models are efficient video learners［C］// Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2023： 6545-6554.
9	NI B， PENG H， CHEN M， et al. Expanding language-image pretrained models for general video recognition［C］// Proceedings of the 2022 European Conference on Computer Vision， LNCS 13664. Cham： Springer， 2022： 1-18.
10	TAHGHIGHI P， KOOCHARI A， JALALI M. Deformable convolutional LSTM for human body emotion recognition［C］// Proceedings of the 2021 International Conference on Pattern Recognition， LNCS 12663. Cham： Springer， 2021： 741-747.
11	SHEN Z， CHENG J， HU X， et al. Emotion recognition based on multi-view body gestures［C］// Proceedings of the 2019 IEEE International Conference on Image Processing. Piscataway： IEEE， 2019： 3317-3321.
12	WANG L， XIONG Y， WANG Z， et al. Temporal segment networks： towards good practices for deep action recognition［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9912. Cham： Springer， 2016： 20-36.
13	GUNES H， PICCARDI M. A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior［C］// Proceedings of the 18th International Conference on Pattern Recognition. Piscataway： IEEE， 2006： 1148-1153.
14	FILNTISIS P P， EFTHYMIOU N， POTAMIANOS G， et al. Emotion understanding in videos through body， context， and visual-semantic embedding loss［C］// Proceedings of the 2020 European Conference on Computer Vision Workshops， LNCS 12535. Cham： Springer， 2020： 747-755.
15	PIKOULIS I， FILNTISIS P P， MARAGOS P. Leveraging semantic scene characteristics and multi-stream convolutional architectures in a contextual approach for video-based visual emotion recognition in the wild［C］// Proceedings of the 16th IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway： IEEE， 2021： 1-8.
16	ZHANG H， LEE F， CHEN Q. Multi-aspect feature fusion method for video-based multimodal emotion recognition ［EB/OL］. ［2023-11-09］. .
17	WU C， DAVAASUREN D， SHAFIR T， et al. Bodily expressed emotion understanding through integrating Laban movement analysis［J］. Patterns， 2023， 4（10）： No.100816.
18	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2017： 6000-6010.

方法	mAP	mRA
random	10.55	50.00
ST-GCN^［1］	12.63	55.96
TSN^［1］	17.02	62.70
文献［14］模型	16.56	62.66
文献［15］模型	19.30	66.94
文献［16］模型	20.01	68.21
MANet^［17］	21.25	68.32
本文方法	22.33	69.64

方法	mAP	mRA
random	10.55	50.00
ST-GCN^［1］	12.63	55.96
TSN^［1］	17.02	62.70
文献［14］模型	16.56	62.66
文献［15］模型	19.30	66.94
文献［16］模型	20.01	68.21
MANet^［17］	21.25	68.32
本文方法	22.33	69.64

可学习提示向量数M	ELSL层	mAP/%	mRA/%
2	无	21.88	69.20
4	无	21.98	69.62
8	无	22.07	69.66
16	无	22.09	69.40
2	有	22.16	69.14
4	有	22.33	69.64
8	有	22.28	69.71
16	有	22.08	69.56

可学习提示向量数M	ELSL层	mAP/%	mRA/%
2	无	21.88	69.20
4	无	21.98	69.62
8	无	22.07	69.66
16	无	22.09	69.40
2	有	22.16	69.14
4	有	22.33	69.64
8	有	22.28	69.71
16	有	22.08	69.56

可学习提示向量数M	时序层层数	mAP/%	mRA/%
4	0	19.75	65.47
4	1	22.33	69.64
4	2	22.24	69.22
4	3	22.19	68.92

Body movement emotion recognition method based on emotional latent space learning and CLIP model

基于情感隐空间学习与CLIP模型的身体动作情感识别方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 18

Related Articles 15

Recommended Articles

Metrics

文本输入类别	案例	mAP/%	mRA/%
1	［class］	22.33	69.64
2	a body language of ［class］	22.13	69.51
3	［body expression descriptors］	22.23	69.37

[1]	Yiheng SUN, Maofu LIU. Tender information extraction method based on prompt tuning of knowledge [J]. Journal of Computer Applications, 2025, 45(4): 1169-1176.
[2]	Jie HU, Qiyang ZHENG, Jun SUN, Yan ZHANG. Multi-label classification model based on multi-label relational graph and local dynamic reconstruction learning [J]. Journal of Computer Applications, 2025, 45(4): 1104-1112.
[3]	Can MA, Ruizhang HUANG, Lina REN, Ruina BAI, Yaoyao WU. Chinese spelling correction method based on LLM with multiple inputs [J]. Journal of Computer Applications, 2025, 45(3): 849-855.
[4]	Yan YANG, Feng YE, Dong XU, Xuejie ZHANG, Jin XU. Construction of digital twin water conservancy knowledge graph integrating large language model and prompt learning [J]. Journal of Computer Applications, 2025, 45(3): 785-793.
[5]	Xin LIU, Dawei YANG, Changheng SHAO, Haiwen WANG, Mingjiang PANG, Yanru LI. Hierarchical multi-label classification model for public complaints with long-tailed distribution [J]. Journal of Computer Applications, 2025, 45(1): 82-89.
[6]	Bin LI, Min LIN, Siriguleng, Yingjie GAO, Yurong WANG, Shujun ZHANG. Joint entity-relation extraction method for ancient Chinese books based on prompt learning and global pointer network [J]. Journal of Computer Applications, 2025, 45(1): 75-81.
[7]	Xindong YOU, Yingzi WEN, Xinpeng SHE, Xueqiang LYU. Triplet extraction method for mine electromechanical equipment field [J]. Journal of Computer Applications, 2024, 44(7): 2026-2033.
[8]	Chao WEI, Yanping CHEN, Kai WANG, Yongbin QIN, Ruizhang HUANG. Relation extraction method based on mask prompt and gated memory network calibration [J]. Journal of Computer Applications, 2024, 44(6): 1713-1719.
[9]	Xinyan YU, Cheng ZENG, Qian WANG, Peng HE, Xiaoyu DING. Few-shot news topic classification method based on knowledge enhancement and prompt learning [J]. Journal of Computer Applications, 2024, 44(6): 1767-1774.
[10]	Junfeng SHEN, Xingchen ZHOU, Can TANG. Dual-channel sentiment analysis model based on improved prompt learning method [J]. Journal of Computer Applications, 2024, 44(6): 1796-1806.
[11]	Yingjie GAO, Min LIN, Siriguleng, Bin LI, Shujun ZHANG. Prompt learning method for ancient text sentence segmentation and punctuation based on span-extracted prototypical network [J]. Journal of Computer Applications, 2024, 44(12): 3815-3822.
[12]	Bihui YU, Xingye CAI, Jingxuan WEI. Few-shot text classification method based on prompt learning [J]. Journal of Computer Applications, 2023, 43(9): 2735-2740.
[13]	Jinye LI, Ruizhang HUANG, Yongbin QIN, Yanping CHEN, Xiaoyu TIAN. Recognition of sentencing circumstances in adjudication documents based on abductive learning [J]. Journal of Computer Applications, 2022, 42(6): 1802-1807.
[14]	Wei REN, Hexiang BAI. Multi-label image classification method based on global and local label relationship [J]. Journal of Computer Applications, 2022, 42(5): 1383-1390.
[15]	Yongchun BAO, Jianchen ZHANG, Shouxin DU, Junjun ZHANG. Multi-label classification algorithm based on non-negative matrix factorization and sparse representation [J]. Journal of Computer Applications, 2022, 42(5): 1375-1382.