计算机应用 ›› 2016, Vol. 36 ›› Issue (7): 1887-1892.DOI: 10.11772/j.issn.1001-9081.2016.07.1887

• 人工智能 • 上一篇    下一篇

基于无监督学习算法的推特文本规范化

邓加原, 姬东鸿, 费超群, 任亚峰   

  1. 武汉大学 计算机学院, 武汉 430072
  • 收稿日期:2016-01-25 修回日期:2016-03-14 出版日期:2016-07-10 发布日期:2016-07-14
  • 通讯作者: 邓加原
  • 作者简介:邓加原(1991-),男,福建宁德人,硕士研究生,主要研究方向:自然语言处理;姬东鸿(1966-),男,河南驻马店人,教授,博士,CCF会员,主要研究方向:自然语言处理、数据挖掘、机器学习;费超群(1992-),男,河南驻马店人,硕士研究生,主要研究方向:自然语言处理;任亚峰(1986-),男,河南焦作人,博士,CCF会员,主要研究方向:自然语言处理、机器学习。
  • 基金资助:
    国家自然科学基金重点项目(61133012);国家哲学社会科学重大计划项目(11&ZD189);国家自然科学基金资助项目(61173062)。

Twitter text normalization based on unsupervised learning algorithm

DENG Jiayuan, JI Donghong, FEI Chaoqun, REN Yafeng   

  1. School of Computer, Wuhan University, Wuhan Hubei 430072, China
  • Received:2016-01-25 Revised:2016-03-14 Online:2016-07-10 Published:2016-07-14
  • Supported by:
    This work is partially supported by the State Key Program of National Natural Science Foundation of China (61133012), the National Philosophy Social Science Major Bidding Project of China (11&ZD189), the National Natural Science Foundation of China (61173062).

摘要: 推特文本中包含着大量的非标准词,这些非标准词是由人们有意或无意而创造的。对很多自然语言处理的任务而言,预先对推特文本进行规范化处理是很有必要的。针对已有的规范化系统性能较差的问题,提出一种创新的无监督文本规范化系统。首先,使用构造的标准词典来判断当前的推特是否需要标准化。然后,对推特中的非标准词会根据其特征来考虑进行一对一还是一对多规范化;对于需要一对多的非标准词,通过前向和后向搜索算法,计算出所有可能的多词组合。其次,对于多词组合中的非规范化词,基于二部图随机游走和误拼检查,来产生合适的候选。最后,使用基于上下文的语言模型来得到最合适的标准词。所提算法在数据集上获得86.4%的F值,超过当前最好的基于图的随机游走算法10个百分点。

关键词: 规范化, 无监督学习, 二部图, 随机游走, 拼写检查

Abstract: Twitter messages contain a large number of nonstandard tokens, created unintentionally or intentionally by people. It is crucial to normalize the nonstandard tokens for various natural language processing applications. In terms of the existing normalization systems which perform poorly, a novel unsupervised normalization system was proposed. First, a standard dictionary was used to determine whether a tweet needs to be normalized or not. Second, a nonstandard token was considered to take 1-to-1 or 1-to-N recovering based on its characteristics. For 1-to-N recovering, the nonstandard token would be divided into multiple possible words using forward and backward search. Third, some normalization candidates were generated for nonstandard tokens among multiple possible words by integrating random walk and spelling checker. Finally, the best normalized twitter could be obtained by taking all the candidates into consideration of n-gram language model. The experimental results on the manual dataset show that the proposed approach obtains F-score of 86.4%, which is 10 percentage points higher than that of current best graph-based random walk algorithm.

Key words: normalization, unsupervised learning, bipartite graph, random walk, spelling checker

中图分类号: