《计算机应用》唯一官方网站 ›› 2022, Vol. 42 ›› Issue (3): 743-749.DOI: 10.11772/j.issn.1001-9081.2021040846

• 2021年中国计算机学会人工智能会议(CCFAI 2021) • 上一篇    

基于深度注意力网络的课堂教学视频中学生表情识别与智能教学评估

于婉莹, 梁美玉(), 王笑笑, 陈徵, 曹晓雯   

  1. 北京邮电大学 计算机学院,北京 100876
  • 收稿日期:2021-05-24 修回日期:2021-07-26 接受日期:2021-08-05 发布日期:2021-11-09 出版日期:2022-03-10
  • 通讯作者: 梁美玉
  • 作者简介:于婉莹(1996—),女,吉林辽源人,硕士研究生,主要研究方向:深度学习、计算机视觉
    王笑笑(1996—),女,山西长治人,硕士研究生,主要研究方向:深度学习、跨模态搜索
    陈徵(1996—),男,甘肃金昌人,硕士研究生,CCF会员,主要研究方向:深度学习、计算机视觉
    曹晓雯(1998—),女,山西吕梁人,硕士研究生,主要研究方向:深度学习、跨模态搜索。
  • 基金资助:
    国家自然科学基金资助项目(61877006)

Student expression recognition and intelligent teaching evaluation in classroom teaching videos based on deep attention network

Wanying YU, Meiyu LIANG(), Xiaoxiao WANG, Zheng CHEN, Xiaowen CAO   

  1. School of Computer Science,Beijing University of Posts and Telecommunications,Beijing 100876,China
  • Received:2021-05-24 Revised:2021-07-26 Accepted:2021-08-05 Online:2021-11-09 Published:2022-03-10
  • Contact: Meiyu LIANG
  • About author:YU Wanying, born in 1996, M. S. candidate. Her research interests include deep learning, computer vision.
    WANG Xiaoxiao, born in 1996, M. S. candidate. Her research interests include deep learning, cross-modal retrieval.
    CHEN Zheng, born in 1996, M. S. candidate. His research interests include deep learning, computer vision.
    CAO Xiaowen, born in 1998, M. S. candidate. Her research interests include deep learning, cross-modal retrieval.
  • Supported by:
    National Natural Science Foundation of China(61877006)

摘要:

为了解决复杂课堂场景下学生表情识别的遮挡的问题,同时发挥深度学习在智能教学评估应用上的优势,提出了一种基于深度注意力网络的课堂教学视频中学生表情识别模型与智能教学评估算法。构建了课堂教学视频库、表情库和行为库,利用裁剪和遮挡策略生成多路人脸图像,在此基础上构建了多路深度注意力网络,并通过自注意力机制为多路网络分配不同权重。通过约束损失函数限制各路权重的分配,将人脸图像的全局特征表示为每个支路的特征乘上注意力权重的和除以所有支路的注意力权重之和,并基于学习到的人脸全局特征进行学生课堂表情分类,实现遮挡情况下学生人脸表情识别。提出了融合课堂学生表情和行为状态的智能教学评估算法,实现了课堂教学视频中学生表情识别与智能教学评估。在公开数据集FERplus与自建课堂教学视频数据集上进行实验对比与分析,验证了提出的课堂教学视频中学生表情识别模型能够达到87.34%的准确率,且提出的融合课堂学生表情和行为状态的智能教学评估算法在课堂教学视频数据集上也取得优秀的性能。

关键词: 深度学习, 深度注意力网络, 表情识别, 智能教学评估, 课堂教学视频

Abstract:

In order to solve the occlusion problem of student expression recognition in complex classroom scenes, and give full play to the advantages of deep learning in the application of intelligent teaching evaluation,a student expression recognition model and an intelligent teaching evaluation algorithm based on deep attention network in classroom teaching videos were proposed. A video library, an expression library and a behavior library for classroom teaching were constructed, then, multi-channel facial images were generated by cropping and occlusion strategies. A multi-channel deep attention network was built and self-attention mechanism was used to assign different weights to multiple channel networks. The weight distribution of each channel was restricted by a constrained loss function, then the global feature of the facial image was expressed as the quotient of the sum of the product of the feature times its attention weight of each channel divided by the sum of the attention weights of all channels. Based on the learned global facial feature, the student expressions in classroom were classified, and the student facial expression recognition under occlusion was realized. An intelligent teaching evaluation algorithm that integrates the student facial expressions and behavior states in classroom was proposed, which realized the recognition of student facial expressions and intelligent teaching evaluation in classroom teaching videos. By making experimental comparison and analysis on the public dataset FERplus and self-built classroom teaching video datasets, it is verified that the student facial expressions recognition model in classroom teaching videos achieves high accuracy of 87.34%, and the intelligent teaching evaluation algorithm that integrates the student facial expressions and behavior states in classroom achieves excellent performance on the classroom teaching video dataset.

Key words: deep learning, deep attention network, facial expression recognition, intelligent teaching evaluation, classroom teaching video

中图分类号: