计算机应用 ›› 2010, Vol. 30 ›› Issue (8): 2034-2037.

• 人工智能 • 上一篇    下一篇

基于字词分类的层次分词方法研究

张聪品1,赵理莉2,吴长茂2   

  1. 1. 河南师范大学计算机与信息技术学院
    2.
  • 收稿日期:2010-02-26 修回日期:2010-04-18 发布日期:2010-07-30 出版日期:2010-08-01
  • 通讯作者: 张聪品
  • 基金资助:
    河南省基础与前沿技术研究计划项目;河南省高等学校青年骨干教师资助计划项目;河南省科技攻关项目

Method of Chinese word segmentation based on character-word classification

  • Received:2010-02-26 Revised:2010-04-18 Online:2010-07-30 Published:2010-08-01
  • Contact: ZHANG Cong-Pin

摘要: 中文分词是自然语言处理的基础性问题。条件随机场模型分词过程中出现的切分粒度过小和多字粘连造成的错分问题,是影响分词结果的两个主要原因。提出了一个基于字词分类的层次分词模型,该模型采用多部有效词典进行处理,在外层分词系统中解决切分粒度过小问题;在内层核心层,条件随机场分词后再处理多字粘连问题。实验结果表明,采用加入多词典的字词结合层次分类模型F-测度值有较大的提高,有助于得到好的分词结果。

关键词: 中文分词, 字词分类, 多词典分词, 条件随机场

Abstract: Chinese Word Segmentation (CWS) is the basic problem of natural language processing. A joint character-word classification model for Chinese Word Segmentation was presented, which mainly dealt with the problem of Conditional Random Field (CRF). On the one hand, the majority of errors in CRF caused by fine granularity were figured out in outside-layer of the model. On the other hand, excessive linked words caused by improper segmentations that were settled in core inside-layer. The experimental results show that the value of F is greatly improved and the good results of word segmentation are easily gained by choosing the hierarchical word segmentation model based on character word classification.

Key words: Chinese Word Segmentation (CWS), character word classification, multi-dictionary cws, Conditional Random Field (CRF)