Journal of Computer Applications ›› 2016, Vol. 36 ›› Issue (10): 2772-2776.DOI: 10.11772/j.issn.1001-9081.2016.10.2772

Previous Articles     Next Articles

Micro-blog new word discovery method based on improved mutual information and branch entropy

YAO Rongpeng, XU Guoyan, SONG Jian   

  1. College of Computer and Information, Hohai University, Nanjing Jiangsu 211100, China
  • Received:2016-04-20 Revised:2016-05-26 Online:2016-10-10 Published:2016-10-10
  • Supported by:
    BackgroundThis work is partially supported by the National Science and Technology Support Program of China (2013BAB06B04), the Natural Science Foundation of Jiangsu Province (BK20130852), the Science and Technology Project of Jiangsu Water Resources Department (2013025), the Huaneng Group Company Headquarters Technology Project of China (HNKJ13-H17-04).

基于改进互信息和邻接熵的微博新词发现方法

夭荣朋, 许国艳, 宋健   

  1. 河海大学 计算机与信息学院, 南京 211100
  • 通讯作者: 夭荣朋,E-mail:1206573971@qq.com
  • 作者简介:夭荣朋(1989—),男,江苏徐州人,硕士研究生,主要研究方向:大数据、数据管理;许国艳(1971—),女,内蒙古赤峰人,副教授,博士,CCF会员,主要研究方向:大数据、数据管理;宋健(1991—),男,江苏盐城人,硕士研究生,主要研究方向:大数据、数据管理。
  • 基金资助:
    国家科技支撑计划项目(2013BAB06B04);江苏省自然科学基金资助项目(BK20130852);江苏水利科技项目(2013025);中国华能集团公司总部科技项目(HNKJ13-H17-04)。

Abstract: Aiming at the problem of data sparsity, poor portability and lack of recognition of multiple words (more than three words) in micro-blog new word discovery algorithm, a new word discovery algorithm based on improved Mutual Information (MI) and Branch Entropy (BE), named MBN-Gram, was proposed. Firstly, the N-Gram was used to extract the candidate terms of new words, and the rules of using frequency and stop words were used to filter the candidates. Then the improved MI and BE were used to expand and filter the candidates again. Finally, the corresponding dictionary was used to screen, so as to get new words. Theoretical and experimental analysis show that the accuracy rate, recall rate and F value of MBN-Gram algorithm were improved. Experimental results shows that the MBN-Gram algorithm is effective and feasible.

Key words: new word discovery, multi-word, N-gram, Mutual Information (MI), Branch Entropy (BE)

摘要: 针对目前微博新词发现算法中的数据稀疏、可移植性较差以及缺乏对多字词(大于三字)识别的问题,提出了基于改进互信息(MI)和邻接熵(BE)的微博新词发现算法——MBN-Gram。首先,利用N元递增算法(N-Gram)提取新词的候选项,对提取出来的候选新词使用频率和停用字等规则进行过滤;接着再利用改进MI和BE对候选项进行扩展及再过滤;最后,结合相应词典进行筛选,从而得到新词。通过理论及实验分析,MBN-Gram算法在准确率、召回率及F值上均有一定提高。实验结果表明,MBN-Gram算法是有效可行的。

关键词: 新词发现, 多字词, N-Gram, 互信息, 邻接熵

CLC Number: