Analysis on Effect Range of Context in Chinese Word Segmentation based Word -position Tagging

Journal of Computer Applications ›› 2012, Vol. 32 ›› Issue (05): 1340-1342.

• Artificial intelligence • Previous Articles Next Articles

Analysis on Effect Range of Context in Chinese Word Segmentation based Word -position Tagging

WANG Xi-jie

School of Computer and Information Engineering, Anyang Normal University, Anyang Henan 455000, China

Received:2011-10-09 Revised:2011-12-16 Online:2012-05-01 Published:2012-05-01
Contact: WANG Xi-jie

词位标注汉语分词中上下文有效范围定量分析

王希杰

安阳师范学院计算机与信息工程学院，河南安阳 455000

通讯作者: 王希杰
作者简介:王希杰（1973-），男，河南卫辉人，讲师，硕士，CCF会员，主要研究方向：自然语言处理、信息抽取、文本数据挖掘。
基金资助:
国家自然科学基金资助项目(61040026);河南省教育厅高等学校青年骨干教师项目（2009GGJS-108）

Abstract

Abstract: In Chinese word segmentation with Conditional Random Field (CRF）, the size of feature window plays a crucial role in corpus training. To find the proper size of feature window, a group of feature templates were selected for the comparative tests performed on Bakeoff2005 with toolkit CRF++0.53 considering the effective range of context. The results are: (1) contribution of below-context is greater than above-context;(2) size of feature window influencing the segment performance is no larger than 5, the proper size is four or five.

Key words: Chinese word segmentation, Conditional Random Field (CRF), context, feature window, feature template

摘要： 在利用条件随机场进行基于词位标注的汉语分词时，特征窗口的宽度是决定条件随机场学习效果的重要参数。针对特征窗口最佳宽度的选择问题，设计了一组特征模板，并选取Bakeoff2005中的测试语料，使用CRF++0.53工具包进行了对比实验，定量分析了影响分词效果的有效上下文范文。通过实验得出以下结论：下文对分词性能贡献要大于上文；影响分词性能的特征窗口的宽度不超过五，以四字或五字窗口为宜。

关键词: 汉语分词, 条件随机场, 上下文, 特征窗口, 特征模板

CLC Number:

TP391.1

WANG Xi-jie. Analysis on Effect Range of Context in Chinese Word Segmentation based Word -position Tagging[J]. Journal of Computer Applications, 2012, 32(05): 1340-1342.

王希杰. 词位标注汉语分词中上下文有效范围定量分析[J]. 计算机应用, 2012, 32(05): 1340-1342.

References

［1］姜维，王晓龙，关毅，等. 基于多知识源的中文词法分析系统［J］. 计算机学报， 2007，30(1):137-145.

［2］刘群，张华平，俞鸿魁，等. 基于层叠隐马模型的汉语词法分析［J］. 计算机研究与发展, 2004, 41(8):1421-1429.

［3］罗彦彦，黄德根.基于CRFs边缘概率的中文分词［J］.中文信息学报，2009，23（5）:3-8.

［4］赵伟，侯红旭，从伟，等.基于条件随机场的蒙古词语切分研究［J］.中文信息学报，2010，24（5）：31-35.

［5］黄德根，焦世斗，周惠巍.基于子词的双层CRFs中文分词［J］.计算机研究与发展，2010,47（5）：962-968.

［6］于江德，睢丹，樊孝忠.基于字的词位标注汉语分词［J］.山东大学学报,2010,40(5):117-122.

［7］XUE NIANWEN. Chinese word segmentation as character tagging ［J］. Computational Linguistics and Chinese Language Processing, 2003, 8(1): 29-48.

［8］赵海，揭春雨. 基于有效子串标注的中文分词［J］.中文信息学报，2007,21(5):8-13.

［9］鲁松，白硕.自然语言处理中词语上下文有效范围的定量描述[J].计算机学报，2001，24（7）：742-747.

［10］LAFFERTY J, PEREIRA F, MCCALLUM A. Conditional random fields: probabilistic models for segmenting and labeling sequence data ［C］// In Proceedings of 18th International Conference on Machine Learning, Washington, DC: IEEE Computer Society, 2001:282-289.

［11］PEREIRA L J,NOCEDAL J,SCHNABEL R B. Representations of quasi-Newton matrices and their use in limited memory methods［J］. Mathematical Programming,1994,62(2):129-156.

［12］于江德，谷川，葛文英，等. 一种基于字和子串联合标注的汉语分词方法［J］. 山西大学学报，2011,34（3）：357-362.

［13］郑荣廷,吉久名,李楠.特征区间的有效划分对CRF识别效果的影响［J］. 图书情报工作，2011,55（4）：114-118.

［14］赵晓凡，胡顺义, 刘永革. CRF模型中参数f在字标注汉语分词中的适用性研究［J］. 郑州大学学报，2011,32（4）：103-106.

［15］黄昌宁，赵海. 中文分词十年回顾［J］. 中文信息学报，2007，21(3): 8-19.

[1]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[2]	Chunxue ZHANG, Liqing QIU, Cheng’ai SUN, Caixia JING. Purchase behavior prediction model based on two-stage dynamic interest recognition [J]. Journal of Computer Applications, 2024, 44(8): 2365-2371.
[3]	Caiqin WANG, Yuhao ZHOU, Shunxiang ZHANG, Yanhui WANG, Xiaolong WANG. Aspect-opinion pair extraction of new energy vehicle complaint text based on context enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2430-2436.
[4]	Hao CHAO, Shuqi FENG, Yongli LIU. Convolutional recurrent neural network optimized by multiple context vectors in EEG-based emotion recognition [J]. Journal of Computer Applications, 2024, 44(7): 2041-2046.
[5]	Yongfeng DONG, Jiaming BAI, Liqin WANG, Xu WANG. Chinese named entity recognition combining prior knowledge and glyph features [J]. Journal of Computer Applications, 2024, 44(3): 702-708.
[6]	Di ZHOU, Zili ZHANG, Jia CHEN, Xinrong HU, Ruhan HE, Jun ZHANG. Stomach cancer image segmentation method based on EfficientNetV2 and object-contextual representation [J]. Journal of Computer Applications, 2023, 43(9): 2955-2962.
[7]	Hao YANG, Yi ZHANG. Feature pyramid network algorithm based on context information and multi-scale fusion importance awareness [J]. Journal of Computer Applications, 2023, 43(9): 2727-2734.
[8]	Qing WANG, Xiuwei GAO, Yehai XIE, Guilong LIU. Inner product reduction in formal context [J]. Journal of Computer Applications, 2023, 43(4): 1079-1085.
[9]	Qingtang LIU, Xinqian MA, Jie ZHOU, Linjing WU, Pengxiao ZHOU. Understanding of math word problems integrating commonsense knowledge base and grammatical features [J]. Journal of Computer Applications, 2023, 43(2): 356-364.
[10]	Fei XIA, Shuaiqi CHEN, Min HUA, Bihong JIANG. Chinese word segmentation method in electric power domain based on improved BERT [J]. Journal of Computer Applications, 2023, 43(12): 3711-3718.
[11]	Zixing YU, Shaojun QU, Xin HE, Zhuo WANG. High-low dimensional feature guided real-time semantic segmentation network [J]. Journal of Computer Applications, 2023, 43(10): 3077-3085.
[12]	Kai WEN, Weiwei TANG, Junchen XIONG. Real-time segmentation algorithm based on attention mechanism and effective factorized convolution [J]. Journal of Computer Applications, 2022, 42(9): 2659-2666.
[13]	Xiaoyu CHEN, Wei WANG. Semantic extraction of domain-dependent mathematical text [J]. Journal of Computer Applications, 2022, 42(8): 2386-2393.
[14]	Cheng HUANG, Qianrui ZHAO. Sensitive information detection method based on attention mechanism-based ELMo [J]. Journal of Computer Applications, 2022, 42(7): 2009-2014.
[15]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.

Analysis on Effect Range of Context in Chinese Word Segmentation based Word -position Tagging

词位标注汉语分词中上下文有效范围定量分析

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics