连续手语识别中的文本纠正和补全方法

doi:10.11772/j.issn.1001-9081.2020060798

计算机应用 ›› 2021, Vol. 41 ›› Issue (3): 694-698.DOI: 10.11772/j.issn.1001-9081.2020060798

所属专题：人工智能

连续手语识别中的文本纠正和补全方法

龙广玉¹, 陈益强^1,2, 邢云冰²

1. 湘潭大学计算机学院·网络空间安全学院, 湖南湘潭 411105;
2. 中国科学院计算技术研究所, 北京 100190

收稿日期:2020-06-11 修回日期:2020-10-20 发布日期:2020-12-22 出版日期:2021-03-10
通讯作者: 陈益强
作者简介:龙广玉(1995-),女,广西宜州人,硕士研究生,CCF会员,主要研究方向:自然语言处理、数据挖掘;陈益强(1973-),男,湖南湘潭人,研究员,博士,CCF杰出会员,主要研究方向:泛在计算、可穿戴计算、智能人机交互;邢云冰(1982-),男,河北张家口人,高级工程师,硕士,主要研究方向:手语交互、感知计算、健康监护。
基金资助:
国家重点研发计划项目（2018YFC2002603）。

Text correction and completion method in continuous sign language recognition

LONG Guangyu¹, CHEN Yiqiang^1,2, XING Yunbing²

1. School of Computer Science&School of Cyberspace Science, Xiangtan University, Xiangtan Hunan 411105, China;
2. Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China

Received:2020-06-11 Revised:2020-10-20 Online:2020-12-22 Published:2021-03-10
Supported by:
This work is partially supported by the National Key Research and Development Program of China (2018YFC2002603).

摘要/Abstract

摘要： 针对基于视频的连续手语识别的文本结果存在语义模糊、语序混乱的问题，提出一种两步法将连续手语识别结果的手语文本转化为通顺、可懂的汉语文本。第一步，基于自然手语规则以及N元语言模型（N-gram）对连续手语识别的结果进行文本调序；第二步，利用汉语通用量词数据集训练双向长短期记忆（Bi-LSTM）网络模型，以解决手语语法无量词的问题，从而提升语句通顺度。使用绝对准确率和最长正确子序列占比作为文本调序的评价指标，实验结果显示，所提方法的文本调序结果绝对准确率为77.06%，最长正确子序列占比为86.55%，量词补全准确率为97.23%。所提的方法能够有效提升连续手语识别的文本结果的通畅度和可懂度，已成功应用于基于视频的连续手语识别，提升了听障人和健听人的无障碍交流体验。

关键词: 连续手语识别, N元语言模型, 文本调序, 双向长短记忆网络, 量词补全

Abstract: Aiming at the problem that the text results of continuous sign language recognition based on video have problems of semantic ambiguity and chaotic word order, a two-step method was proposed to convert the sign language text of the continuous sign language recognition result into a fluent and understandable Chinese text. In the first step, the natural sign language rules and N-gram language model (N-gram) were used to perform the text ordering of the continuous sign language recognition results. In the second step, a Bidirectional Long-Term Short-Term Memory (Bi-LSTM) network model was trained by using the Chinese universal quantifier dataset to solve the quantifier-free problem of the sign language grammar, so as to improve the fluency of texts. The absolute accuracy and the proportion of the longest correct subsequences were adopted as the evaluation indexes of text ordering. Experimental results showed that the text ordering results of the proposed method had the absolute accuracy of 77.06%, the proportion of the longest correct subsequences of 86.55%, and the accuracy of quantifier completion of 97.23%. The proposed method can effectively improve the smoothness and intelligibility of text results of continuous sign language recognition. It has been successfully applied to the video-based continuous sign language recognition, which improves the barrier-free communication experience between the hearing-impaired and the normal-hearing people.

Key words: continuous sign language recognition, N-gram language model, text ordering, Bidirectional Long-Term Short-Term Memory (Bi-LSTM) network, quantifier completion

中图分类号:

TP391.1

龙广玉, 陈益强, 邢云冰. 连续手语识别中的文本纠正和补全方法[J]. 计算机应用, 2021, 41(3): 694-698.

LONG Guangyu, CHEN Yiqiang, XING Yunbing. Text correction and completion method in continuous sign language recognition[J]. Journal of Computer Applications, 2021, 41(3): 694-698.

参考文献

[1] 刘润楠. 中国大陆手语语言学研究现状[J]. 中国特殊教育, 2005(5):26-29. (LIU R N. Current linguistic studies on sign language in mainland China[J]. Chinese Journal of Special Education,2005(5):26-29.)
[2] KOLLER O,FORSTER J,NEY H. Continuous sign language recognition:towards large vocabulary statistical recognition systems handling multiple signers[J]. Computer Vision and Image Understanding,2015,141:108-125.
[3] HUANG J, ZHOU W, ZHANG Q, et al. Video-based sign language recognition without temporal segmentation[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto,CA:AAAI Press,2018:2257-2264.
[4] CAMGOZ N C,HADFIELD S,KOLLER O,et al. Neural sign language translation[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:7784-7793.
[5] YUAN Z,BRISCOE T. Grammatical error correction using neural machine translation[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Stroudsburg, PA:Association for Computational Linguistics,2016:380-386.
[6] YANG Y,XIE P,TAO J,et al. Alibaba at IJCNLP-2017 task 1:Embedding grammatical features into LSTMs for Chinese grammatical error diagnosis task[C]//Proceedings of the 8th International Joint Conference on Natural Language Processing.[S. l.]:Asian Federation of Natural Language Processing,2017:41-46.
[7] FU K,HUANG J,DUAN Y,et al. Youdao's winning solution to the NLPCC-2018 task 2 challenge:a neural machine translation approach to Chinese grammatical error correction[C]//Proceedings of the 2018 International Conference Natural Language Processing, LNCS 11108. Cham:Springer,2018:341-350.
[8] GUBBINS J, VLACHOS A. Dependency language models for sentence completion[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA:Association for Computational Linguistics,2013:1405-1410.
[9] PARK H,CHO S,PARK J. Word RNN as a baseline for sentence completion[C]//Proceedings of the IEEE 5th International Congress on Information Science and Technology. Piscataway:IEEE,2018:183-187.
[10] ISLAM S,SARKAR M F,HUSSAIN T,et al. Bangla sentence correction using deep neural network based sequence to sequence learning[C]//Proceedings of the 21st International Conference of Computer and Information Technology. Piscataway:IEEE,2018:1-6.
[11] CAVNAR W B,TRENKLE J M. N-gram-based text categorization[C]//Proceedings of the 3rd Annual Symposium on Document Analysis and Information Retrieval. Las Vegas:ISRI,1994:161-175.
[12] 吕会华, 王红英, 巩卓. 国内外手语语序研究综述[J]. 中州大学学报,2014,31(3):73-79.(LYU H H,WANG H Y,GONG Z. Domestic and overseas researches on word orders in sign languages[J]. Journal of Zhongzhou University,2014,31(3):73-79.)
[13] SUTSKEVER I,VINYALS O,LE Q V. Sequence to sequence learning with neural networks[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2014:3104-3112.
[14] HUANG Z,XU W,YU K. Bidirectional LSTM-CRF models for sequence tagging[EB/OL].[2020-04-06]. https://arxiv.org/pdf/1508.01991.pdf.
[15] SCHUSTER M,PALIWAL K K. Bidirectional recurrent neural networks[J]. IEEE Transactions on Signal Processing,1997,45(11):2673-2681.
[16] BRIGHTMART. Brightmart/nlp_chinese_corpus:release version 1.0[EB/OL].[2020-04-04]. https://doi.org/10.5281/zenodo.3402023.
[17] 中国残疾人联合会教育就业部, 中国聋人协会. 中国手语日常会话[M]. 北京:华夏出版社,2006:1-140.(The Education and Employment Department of the China Disabled Persons' Federation,China Association of the Deaf and Hard of Hearing. Chinese Sign Language Daily Conversation[M]. Beijing:Huaxia Publishing House,2006:1-140.)
[18] REITER E. A structured review of the validity of BLEU[J]. Computational Linguistics,2018,44(3):393-401.
[19] 王敏, 郑家恒. 基于改进的隐马尔科夫模型的汉语词性标注[J]. 计算机应用,2006,26(S2):197-198,207.(WANG M, ZHENG J H. Chinese part-of-speech tagging based on improved hidden Markov model[J]. Journal of Computer Applications, 2006,26(S2):197-198,207.)

连续手语识别中的文本纠正和补全方法

Text correction and completion method in continuous sign language recognition

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	张庆杨凡方宇涵. 基于多模态信息融合的中文拼写纠错算法[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[2]	高颖杰, 林民, 斯日古楞null, 李斌, 张树钧. 基于片段抽取原型网络的古籍文本断句标点提示学习方法[J]. 《计算机应用》唯一官方网站, 2024, 44(12): 3815-3822.
[3]	王猛张大千周冰艳马倩影吕继东. 基于时序知识图谱补全的CTCS-3级列控车载接口设备故障诊断方法[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[4]	杨青朱焱. 改进语言规则中的表示的隐喻识别技术[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[5]	余婧陈艳平扈应黄瑞章秦永彬. 结合实体边界偏移的序列标注优化方法[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[6]	张伟牛家祥马继超沈琼霞. 深层语义特征增强的ReLM中文拼写纠错模型[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[7]	徐章杰陈艳平扈应黄瑞章秦永彬. 联合边界生成的多目标学习嵌套命名实体识别[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[8]	代震龙韩萌杨文艳朱诗能杨书蓉. 序列模式挖掘综述[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[9]	徐乐黄瑞章白瑞娜秦永彬. 基于意图正则化的深度半监督文本聚类[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[10]	彭一峰朱焱. 结合预处理方法和对抗学习的公平链接预测[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[11]	赵彪秦玉华田荣坤胡月航陈芳锐. 依赖类型及距离增强的方面级情感分析模型[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[12]	任登燃王淑营. 基于差分边界增强的风电装备嵌套实体识别模型[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[13]	田海燕黄赛豪张栋李寿山. 视觉指导的分词和词性标注[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[14]	帅健王中卿陈嘉沥. 基于代码生成的细粒度情感分析方法[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[15]	姜雨杉, 张仰森. 大语言模型驱动的立场感知事实核查[J]. 《计算机应用》唯一官方网站, 2024, 44(10): 3067-3073.