基于无监督学习的专业领域分词歧义消解方法

doi:10.3724/SP.J.1087.2013.00780

计算机应用 ›› 2013, Vol. 33 ›› Issue (03): 780-783.DOI: 10.3724/SP.J.1087.2013.00780

基于无监督学习的专业领域分词歧义消解方法

修驰^1*,宋柔^1,2

1.北京工业大学计算机学院,北京 100022;
2.北京语言大学信息科学学院,北京 100083

收稿日期:2012-09-26 修回日期:2012-10-31 出版日期:2013-03-01 发布日期:2013-03-01
通讯作者: 修驰
作者简介:修驰(1984-),女,辽宁营口人,博士研究生,主要研究方向:中文分词、统计机器翻译; 宋柔(1947-),男,江苏苏州人,教授,博士生导师,硕士,主要研究方向:智能软件工具、中文分词、计算机辅助汉语校对、广义话题理论。
基金资助:
国家自然科学基金资助项目(60872121)。

Disambiguation of domain word segmentation based on unsupervised learning

XIU Chi^1*,SONG Rou^1,2

1.College of Computer Science, Beijing University of Technology, Beijing 100022, China;
2.College of Information Science, Beijing Language and Culture University, Beijing 100083,China

Received:2012-09-26 Revised:2012-10-31 Online:2013-03-01 Published:2013-03-01
Contact: Chi XIU

摘要/Abstract

摘要： 中文自然语言处理中专业领域分词的难度远远高于通用领域。特别是在专业领域的分词歧义方面,一直没有找到有效的解决方法。针对该问题提出基于无监督学习的专业领域分词歧义消解方法。以测试语料自身的字符串频次信息、互信息、边界熵信息为分词歧义的评价标准,独立、组合地使用这三种信息解决分词歧义问题。实验结果显示该方法可以有效消解专业领域的分词歧义,并明显提高分词效果。

关键词: 专业领域分词, 分词歧义, 字符串频次, 互信息, 边界熵

Abstract: Domain word segmentation is much more difficult than general word segmentation in Chinese natural language processing. The segmentation ambiguity has been lack of effective solution especially. Concerning this problem, an unsupervised learning method for domain segmentation ambiguity was proposed. String frequency, mutual information and boundary entropy were selected as evaluation standard for segmentation ambiguity. Individual and combination of these three kinds of information were used to solve the problem. The experimental results suggest that the proposed can solve the domain segmentation ambiguity efficiently and effectively.

Key words: domain word segmentation, segmentation ambiguity, string frequency, mutual information, boundary entropy

中图分类号:

TP391.1

修驰宋柔. 基于无监督学习的专业领域分词歧义消解方法[J]. 计算机应用, 2013, 33(03): 780-783.

XIU Chi SONG Rou. Disambiguation of domain word segmentation based on unsupervised learning[J]. Journal of Computer Applications, 2013, 33(03): 780-783.

参考文献

[1]CHEN K J, LIU S H. Word identification for Mandarin Chinese sentences[C]// COLING '92: Proceedings of the 14th Conference on Computational Linguistics. New York: ACM Press,1992:101-107.
[2]CHENG K S, YONG G H, WONG K F. A study on word-based and integral-bit Chinese text compression algorithms [J]. Journal of the American Society for Information Science, 1999, 50(3):218-228.
[3]XUE N W. Chinese word segmentation as character tagging [J]. International Journal of Computational Linguistics and Chinese Language Processing, 2003,8(1): 29-48.
[4]TSENG H, HANG P C, ANDREW G, et al. A conditional random field word segmenter for sighan bakeoff 2005[C]// Proceedings of the Fourth SIGHAN Workshop. Jeju Island, Korea: Association of Computational Linguistics, 2005:168-171.
[5]ZHANG Y, CLARK S. Chinese segmentation with a word-based perceptron algorithm[C]// Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Prague, Czech Republic: Association of Computational Linguistics, 2007:840-847.
[6]孙茂松,肖明,邹嘉彦. 基于无指导学习策略的无词表条件下的汉语自动分词[J]. 计算机学报,2004,27(6):736-742.
[7]JIN Z H, TANAKA-ISHII K. Unsupervised segmentation of Chinese text by use of branching entropy[C]// Proceedings of COLING/ACL 2006. Sidney, Australia: Association for Computational Linguistics, 2006:428-435.
[8]KITYZ C Y, WIKLS Y. Unsupervised learning of word boundary with description length gain[C]// Proceedings of the CoNLL99 ACL Workshop. Bergen, Norway: Association for Computational Linguistics, 1999:1-6.
[9]SUN W W, XU J. Enhancing Chinese word segmentation using unlabeled data[C]// Proceedings of 2011 Conference on Empirical Methods in Natural Language Processing. Edinburgh, Scotland, UK: Association for Computational Linguistics, 2011:970-979.

[10]ZHAO H, KITYZ C Y. An empirical comparison of goodness measures for unsupervised Chinese word segmentation with a unified framework[C]// Proceedings of the Third International Joint Conference on Natural Language Processing. Hyderabad, India: Asian Federation of Natural Language Processing, 2008:9-16.
[11]罗志勇,宋柔. 现代汉语通用分词系统中歧义切分的实用技术[J]. 计算机研究与发展, 2006,43(6):1122-1128.
[12]罗志勇,宋柔. 基于多特征的自适应新词识别[J]. 北京工业大学学报,2007,33(7):718-725.
[13]SUN X, WANG H F, LI WJ. Fast online training with frequency-adaptive learning rates for Chinese word segmentation and new word detection[C]// Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. Jeju Island, Korea: Association for Computational Linguistics, 2012:253-262.
[14]WANG H S, ZHU J, TANG S P, et al. A New unsupervised approach to word segmentation [J]. Computational Linguistics, 2011, 37(3):421-454.
[15]MAGISTRY P, SAGOT B. Unsupervised word segmentation: the case for Mandarin Chinese[C]// Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. Jeju Island, Korea: Association for Computational Linguistics,2012:383-387.
[16]修驰,宋柔. 基于“固结词串”实例的中文分词研究[J].中文信息学报,2012,26(3):59-64.
[17]刘群, 张华平, 俞鸿魁. 基于层叠隐马模型的汉语词法分析[J].计算机研究与发展,2004,41(8):1421-1429.
[18]何克抗, 徐辉.书面汉语自动分词专家系统设计原理[J].中文信息学报,1991,5(2) :1-14.

基于无监督学习的专业领域分词歧义消解方法

Disambiguation of domain word segmentation based on unsupervised learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	程玉胜, 宋帆, 王一宾, 钱坤. 基于专家特征的条件互信息多标记特征选择算法[J]. 计算机应用, 2020, 40(2): 503-509.
[2]	王煜, 徐建民. 用于网络新闻热点识别的热点新词发现[J]. 计算机应用, 2020, 40(12): 3513-3519.
[3]	雍菊亚, 周忠眉. 基于互信息的多级特征选择算法[J]. 计算机应用, 2020, 40(12): 3478-3484.
[4]	魏嘉旺, 王肖, 袁玉波. 人脸特征点定位的自适应窗回归方法[J]. 计算机应用, 2019, 39(5): 1459-1465.
[5]	毛莺池, 曹海, 平萍, 李晓芳. 基于最大联合条件互信息的特征选择[J]. 计算机应用, 2019, 39(3): 734-741.
[6]	胡健, 苏永东, 黄文载, 肖鹏, 刘玉婷, 杨本富. 基于互信息加权集成迁移学习的入侵检测方法[J]. 计算机应用, 2019, 39(11): 3310-3315.
[7]	徐洪峰, 孙振强. 多标签学习中基于互信息的快速特征选择方法[J]. 计算机应用, 2019, 39(10): 2815-2821.
[8]	梁志刚, 顾军华. 改进头脑风暴优化算法与Powell算法结合的医学图像配准[J]. 计算机应用, 2018, 38(9): 2683-2688.
[9]	唐小川, 邱曦伟, 罗亮. 基于交互作用的文本分类特征选择算法[J]. 计算机应用, 2018, 38(7): 1857-1861.
[10]	喻德旷, 杨谊. 肿瘤特征基因选择的互信息最值过滤原则与粒子群优化算法[J]. 计算机应用, 2018, 38(2): 421-426.
[11]	王则林, 郝水侠. 运用差分演化算法实现包匹配多层核心基的提取[J]. 计算机应用, 2017, 37(3): 777-781.
[12]	刘胜男, 宁纪锋. 基于超像素的点互信息边界检测算法[J]. 计算机应用, 2016, 36(8): 2296-2300.
[13]	夭荣朋, 许国艳, 宋健. 基于改进互信息和邻接熵的微博新词发现方法[J]. 计算机应用, 2016, 36(10): 2772-2776.
[14]	国强, 秦月. 改进的基于网络编码中继转发方案[J]. 计算机应用, 2016, 36(1): 61-65.
[15]	韩敏, 孙卓然. 基于小波变换和AdaBoost极限学习机的癫痫脑电信号分类[J]. 计算机应用, 2015, 35(9): 2701-2705.