Abstract: Aiming at the problem that the text results of continuous sign language recognition based on video have problems of semantic ambiguity and chaotic word order, a two-step method was proposed to convert the sign language text of the continuous sign language recognition result into a fluent and understandable Chinese text. In the first step, the natural sign language rules and N-gram language model (N-gram) were used to perform the text ordering of the continuous sign language recognition results. In the second step, a Bidirectional Long-Term Short-Term Memory (Bi-LSTM) network model was trained by using the Chinese universal quantifier dataset to solve the quantifier-free problem of the sign language grammar, so as to improve the fluency of texts. The absolute accuracy and the proportion of the longest correct subsequences were adopted as the evaluation indexes of text ordering. Experimental results showed that the text ordering results of the proposed method had the absolute accuracy of 77.06%, the proportion of the longest correct subsequences of 86.55%, and the accuracy of quantifier completion of 97.23%. The proposed method can effectively improve the smoothness and intelligibility of text results of continuous sign language recognition. It has been successfully applied to the video-based continuous sign language recognition, which improves the barrier-free communication experience between the hearing-impaired and the normal-hearing people.

Key words: continuous sign language recognition, N-gram language model, text ordering, Bidirectional Long-Term Short-Term Memory (Bi-LSTM) network, quantifier completion

摘要： 针对基于视频的连续手语识别的文本结果存在语义模糊、语序混乱的问题，提出一种两步法将连续手语识别结果的手语文本转化为通顺、可懂的汉语文本。第一步，基于自然手语规则以及N元语言模型（N-gram）对连续手语识别的结果进行文本调序；第二步，利用汉语通用量词数据集训练双向长短期记忆（Bi-LSTM）网络模型，以解决手语语法无量词的问题，从而提升语句通顺度。使用绝对准确率和最长正确子序列占比作为文本调序的评价指标，实验结果显示，所提方法的文本调序结果绝对准确率为77.06%，最长正确子序列占比为86.55%，量词补全准确率为97.23%。所提的方法能够有效提升连续手语识别的文本结果的通畅度和可懂度，已成功应用于基于视频的连续手语识别，提升了听障人和健听人的无障碍交流体验。

关键词: 连续手语识别, N元语言模型, 文本调序, 双向长短记忆网络, 量词补全