Lip language recognition algorithm based on single-tag radio frequency identification

doi:10.11772/j.issn.1001-9081.2021061390

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (6): 1762-1769.DOI: 10.11772/j.issn.1001-9081.2021061390

Special Issue: 2021年全国开放式分布与并行计算学术年会(DPCS 2021)论文

• National Open Distributed and Parallel Computing Conference 2021 (DPCS 2021） • Previous Articles Next Articles

Lip language recognition algorithm based on single-tag radio frequency identification

Yingqi ZHANG, Dawei PENG, Sen LI, Ying SUN, Qiang NIU()

School of Computer Science and Technology，China University of Mining and Technology，Xuzhou Jiangsu 221116，China

Received:2021-08-03 Revised:2021-08-31 Accepted:2021-10-15 Online:2022-01-10 Published:2022-06-10
Contact: Qiang NIU
About author:ZHANG Yingq，born in 1996，M. S. candidate. Her research interests include Internet of Things，wireless sensing
PENG Dawei，born in 2001. His research interests include wireless sensing.
LI Sen，born in 1998. His research interests include wireless sensing.
SUN Ying，born in 1974，Ph. D.，professor. His research interests include intelligent information processing，artificial intelligence，pattern recognition，machine learning，data mining.
Supported by:
National Natural Science Foundation of China(51674255)

基于单标签射频识别的唇语识别算法

张瑛琪, 彭大卫, 李森, 孙莹, 牛强()

中国矿业大学计算机科学与技术学院，江苏徐州 221116

通讯作者: 牛强
作者简介:张瑛琪（1996—），女，辽宁营口人，硕士研究生，CCF会员，主要研究方向：物联网、无线感知
彭大卫（2001—），男，江苏邳州人，主要研究方向：无线感知
李森（1998—），男，河南郑州人，主要研究方向：无线感知
孙莹（2000—），女，江苏泰兴人，主要研究方向：无线感知
基金资助:
国家自然科学基金资助项目(51674255)

Abstract

Abstract:

In recent years， a wireless platform for speech recognition using multiple customized and stretchable Radio Frequency Identification （RFID） tags has been proposed， however， it is difficult for the tags to accurately capture large frequency shifts caused by stretching， and multiple tags need to be detected and recalibrated when the tags fall off or wear out naturally. In response to the above problems， a lip language recognition algorithm based on single-tag RFID was proposed， in which a flexible， easily concealable and non-invasive single universal RFID tag was attached to the face， allowing lip language recognition even if the user does not make a sound and relies only on facial micro-actions. Firstly， a model was established to process the Received Signal Strength （RSS） and phase changes of individual tags received by an RFID reader responding over time and frequency. Then the Gaussian function was used to preprocess the noise of the original data by smoothing and denoising， and the Dynamic Time Warping （DTW） algorithm was used to evaluate and analyze the collected signal characteristics to solve the problem of pronunciation length mismatch. Finally， a wireless speech recognition system was created to recognize and distinguish the facial expressions corresponding to the voice， thus achieving the purpose of lip language recognition. Experimental results show that the accuracy of RSS can reach more than 86.5% by the proposed algorithm for identifying 200 groups of digital signal characteristics of different users.

Key words: Radio Frequency IDentification (RFID), lip language recognition, single tag, Received Signal Strength (RSS), Dynamic Time Warping (DTW)

摘要：

近年来，有研究提出了使用多个定制且可拉伸的射频识别（RFID）标签进行语音识别的无线平台，但该标签难以精准捕捉拉伸引起的大频率偏移，而且需要探测多个标签，标签脱落或自然磨损时还须重新校准。针对以上问题，提出基于单标签RFID的唇语识别算法，将灵活、易于隐藏且没有侵入性的单个通用RFID标签贴在脸上，即使用户不发出声音，仅依靠面部的微动作也可进行唇语识别。首先建立模型处理RFID阅读器接收的单个标签随时间和频率响应的接收信号强度（RSS）和相位变化，然后采用高斯函数对原始数据的噪点进行平滑去噪预处理，再采用动态时间规整（DTW）算法对收集到的信号特征进行评估分析，以解决发音长短不匹配的问题；最后创建无线语音识别系统来识别区分与声音相对应的面部表情，从而达到识别唇语的目的。实验结果表明，对于识别不同用户的200组数字信号特征，该方法的RSS准确率可以达到86.5%以上。

关键词: 射频识别, 唇语识别, 单标签, 接收信号强度, 动态时间规整

CLC Number:

TP391

Yingqi ZHANG, Dawei PENG, Sen LI, Ying SUN, Qiang NIU. Lip language recognition algorithm based on single-tag radio frequency identification[J]. Journal of Computer Applications, 2022, 42(6): 1762-1769.

张瑛琪, 彭大卫, 李森, 孙莹, 牛强. 基于单标签射频识别的唇语识别算法[J]. 《计算机应用》唯一官方网站, 2022, 42(6): 1762-1769.

Figures/Tables 16

Fig. 1 Specific locations of RFID tags on face

Fig. 2 Phases and RSSs of multiple tags

Fig. 3 Phase and RSS of single tag

Fig. 4 Flowchart of system

Fig. 5 Phase and RSS information of raw data

Fig. 6 Comparison of phase and RSS before and after Gaussian preprocessing

Fig. 7 DTW cumulative distance matrix

Fig. 8 Comparison of optimal path before and after DTW

Fig. 9 Hardware components

Fig. 10 Experimental scenario deployment

Tab. 1 Accuracy of each user with Phase and RSS

数据集	性别	语速	准确率/%
数据集	性别	语速	相位	RSS
混合数据			75.1	82.8
用户A	女	较快	86.0	93.5
用户B	女	较慢	79.5	95.0
用户C	男	中	79.0	93.5
用户D	男	较快	78.5	86.5
用户E	男	较慢	67.0	87.0

Fig. 11 Confusion matrixes of phase and RSS for user A

Fig. 12 Confusion matrixes of phase and RSS for user B

Fig. 13 Accuracies of different sample sizes

Fig. 14 Schematic diagram of tag angle deployment

Fig. 15 RSS comparison of tag with different angles

References 24

1	PETAJAN E D. Automatic lipreading to enhance speech recognition （speech reading）［D］. Champaign， IL： University of Illinois at Urbana-Champaign， 1984： 4-23.
2	PETAJAN E， BISCHOFF B， BODOFF D， et al. An improved automatic lipreading system to enhance speech recognition［C］// Proceedings of the 1988 SIGCHI Conference on Human Factors in Computing Systems. New York： ACM， 1988： 19-25. 10.1145/57167.57170
3	BAI Z X， ZHANG X L. Speaker recognition based on deep learning： an overview［J］. Neural Networks， 2021， 140： 65-99. 10.1016/j.neunet.2021.03.004
4	HONMA N， TODA K， TSUNEKAWA Y. DoA estimation technique of back-scattering signal from RFID for gesture recognition［C］// Proceedings of the 2015 9th European Conference on Antennas and Propagation. Piscataway：IEEE， 2015： 1-2.
5	LIU J， CHEN X Y， CHEN S G， et al. TagSheet： sleeping posture recognition with an unobtrusive passive tag matrix［C］// Proceedings of the 2019 IEEE Conference on Computer Communications. Piscataway： IEEE， 2019： 874-882. 10.1109/infocom.2019.8737599
6	XIE L， YIN Y F， VASILAKOS A V， et al. Managing RFID data： challenges， opportunities and solutions［J］. IEEE Communications Surveys and Tutorials， 2014， 16（3）： 1294-1311. 10.1109/surv.2014.022614.00143
7	WANG J X， PAN C F， JIN H J， et al. RFID tattoo： a wireless platform for speech recognition［J］. Proceedings of the ACM on Interactive， Mobile， Wearable and Ubiquitous Technologies， 2019， 3（4）： No.155. 10.1145/3369812
8	WANG J， ABARI O， KESHAV S. Challenge： RFID hacking for fun and profit［C］// Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. New York： ACM， 2018： 461-470. 10.1145/3241539.3241561
9	YEAGER D J， HOLLEMAN J， PRASAD R， et al. NeuralWISP： a wirelessly powered neural interface with 1-m range［J］. IEEE Transactions on Biomedical Circuits and Systems， 2009， 3（6）： 379-387. 10.1109/tbcas.2009.2031628
10	YANG L， CHEN Y K， LI X Y， et al. Tagoram： real-time tracking of mobile RFID tags to high precision using COTS devices［C］// Proceedings of the 20th Annual International Conference on Mobile Computing and Networking. New York： ACM， 2014： 237-248. 10.1145/2639108.2639111
11	PRADHAN S， CHAI E， SUNDARESAB K， et al. RIO： a pervasive RFID-based touch gesture interface［C］// Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking. New York： ACM， 2017： 261-274. 10.1145/3117811.3117818
12	HA U， MA Y F， ZHONG Z X， et al. Learning food quality and safety from wireless stickers［C］// Proceedings of the 17th ACM Workshop on Hot Topics in Networks. New York： ACM， 2018： 106-112. 10.1145/3286062.3286078
13	STERPU G， SAAM C， HARTE N. Attention-based audio-visual fusion for robust automatic speech recognition［C］// Proceedings of the 20th ACM International Conference on Multimodal Interaction. New York： ACM， 2018： 111-115. 10.1145/3242969.3243014
14	AGRAWAL S， OMPRAKASH V R， RANVIJAY. Lip reading techniques： a survey［C］// Proceedings of the 2nd International Conference on Applied and Theoretical Computing and Communication Technology. Piscataway： IEEE， 2016： 753-757. 10.1109/icatcct.2016.7912100
15	ASSAEL Y M， SHILLINGFORD B， WHITESON S， et al. LipNet： sentence-level lipreading［EB/OL］. （2016-11-05）［2021-05-19］. .
16	BEDRI A， SAHNI H， THUKRAL P， et al. Toward silent-speech control of consumer wearables［J］. Computer， 2015， 48（10）：54-62. 10.1109/mc.2015.310
17	BRIGHAM K， KUMAR B V K V. Imagined speech classification with EEG signals for silent communication： a preliminary investigation into synthetic telepathy［C］// Proceedings of the 4th International Conference on Bioinformatics and Biomedical Engineering. Piscataway： IEEE， 2010： 1-4. 10.1109/icbbe.2010.5515807
18	CHUNG J S， ZISSERMAN A. Lip reading in the wild［C］// Proceedings of the 2016 Asian Conference on Computer Vision， LNIP 10112. Cham： Springer， 2017： 87-103.
19	CHUNG J S， ZISSERMAN A. Out of time： automated lip sync in the wild［C］// Proceedings of the 2016 Asian Conference on Computer Vision， LNIP 10117. Cham： Springer， 2017： 251-263.
20	WAND M， KOUTNÍK J， SCHMIDHUBER J. Lipreading with long short-term memory［C］// Proceedings of the 2016 IEEE International Conference on Acoustics， Speech and Signal Processing. Piscataway： IEEE， 2016： 6115-6119. 10.1109/icassp.2016.7472852
21	DOBKIN D M. The RF in RFID： UHF RFID in Practice［M］. 2nd ed. Waltham， MA： Newnes， 2013： 306-314. 10.1016/b978-0-12-394583-9.00005-3
22	COHEN K， LESHEM A. Distributed game-theoretic optimization and management of multichannel ALOHA networks［J］. IEEE/ACM Transactions on Networking， 2016， 24（3）： 1718-1731. 10.1109/tnet.2015.2431121
23	XIE B B， XIONG J， CHEN X J， et al. Tagtag： material sensing with commodity RFID［C］// Proceedings of the 17th Conference on Embedded Networked Sensor Systems. New York： ACM， 2019： 338-350. 10.1145/3356250.3360027
24	YANG L， LI Y， LIN Q Z， et al. Making sense of mechanical vibration period with sub-millisecond accuracy using backscatter signals［C］// Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. New York： ACM， 2016： 16-28. 10.1145/2973750.2973759

[1]	Lianpeng QIU, Chengyun SONG. Noise robust dynamic time warping algorithm [J]. Journal of Computer Applications, 2023, 43(6): 1855-1860.
[2]	Changgeng LIU, Yali LIU, Qipeng LU, Tao LI, Changlu LIN, Yi ZHU. Vehicle RKE two-factor authentication protocol resistant to physical cloning attack [J]. Journal of Computer Applications, 2023, 43(11): 3375-3384.
[3]	WU Kaifan, YIN Xinchun. Lightweight anonymous mutual authentication protocol based on random operators for radio frequency identification system [J]. Journal of Computer Applications, 2021, 41(6): 1621-1630.
[4]	SHEN Chen, TAI Lingxiang, PENG Yuwei. Dynamic workload matching method for automatic parameter tuning [J]. Journal of Computer Applications, 2021, 41(3): 657-661.
[5]	XIA Lunteng, ZHANG Li. Networked cane system for blind people based on K-nearest neighbor and dynamic time warping algorithms [J]. Journal of Computer Applications, 2020, 40(8): 2441-2448.
[6]	LIU Yan, ZHANG Yu. Radio frequency identification anti-collision algorithm based on Logistic mapping [J]. Journal of Computer Applications, 2020, 40(8): 2334-2339.
[7]	XU Xiaoxiang, CHANG Xiangmao, CHEN Fangjin. Continuous respiratory volume monitoring system during sleep based on radio frequency identification tag array [J]. Journal of Computer Applications, 2020, 40(5): 1534-1538.
[8]	LIU Shuai, LIU Changliang, ZHEN Chenggang. Multivariate time series fault warning for wind turbine gearbox [J]. Journal of Computer Applications, 2019, 39(4): 1229-1233.
[9]	SUN Shilei, WANG Chao, ZHAO Yuandi. Parameter independent clustering of air traffic trajectory based on silhouette coefficient [J]. Journal of Computer Applications, 2019, 39(11): 3293-3297.
[10]	DING Jianli, MU Tao, WANG Huaichao. Fast indoor positioning algorithm of airport terminal based on spectral regression kernel discriminant analysis [J]. Journal of Computer Applications, 2019, 39(1): 256-261.
[11]	HUANG Keke, LIU Yali, YIN Xinchun. Ultra-lightweight RFID mutual authentication protocol based on regeneration transformation [J]. Journal of Computer Applications, 2019, 39(1): 118-125.
[12]	LI Qiang, YU Fengqin. Improved pitch contour creation and selection algorithm for melody extraction [J]. Journal of Computer Applications, 2018, 38(8): 2411-2415.
[13]	CHANG Bingguo, ZANG Hongying. Time series classifier design based on piecewise dimensionality reduction and updated dynamic time warping [J]. Journal of Computer Applications, 2018, 38(7): 1910-1915.
[14]	XIE Benming, HAN Mingming, ZHANG Pan, ZHANG Wei. Optimization algorithm of dynamic time warping for speech recognition of aircraft towing vehicle [J]. Journal of Computer Applications, 2018, 38(6): 1771-1776.
[15]	WANG Shuai, YANG Xiaodong. RFID tag number estimation algorithm based on sequential linear Bayes method [J]. Journal of Computer Applications, 2018, 38(11): 3287-3292.

Lip language recognition algorithm based on single-tag radio frequency identification

基于单标签射频识别的唇语识别算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 24

Related Articles 15

Recommended Articles

Metrics