Chinese word segment based on character representation learning

doi:10.11772/j.issn.1001-9081.2016.10.2794

Journal of Computer Applications ›› 2016, Vol. 36 ›› Issue (10): 2794-2798.DOI: 10.11772/j.issn.1001-9081.2016.10.2794

Previous Articles Next Articles

Chinese word segment based on character representation learning

LIU Chunli, LI Xiaoge, LIU Rui, FAN Xian, DU Liping

College of Computer Science and Technology, Xi'an University of Posts and Telecommunications, Xi'an Shaanxi 710121, China

Received:2016-03-24 Revised:2016-06-21 Online:2016-10-10 Published:2016-10-10
Supported by:
BackgroundThis work is partially supported by the National Natural Science Foundation of China (61373116), the Development Funds for the Key Subjects of the Universities in Shaanxi Province (112-1602), the Graduate Innovative Foundation of Xi'an University of Posts & Telecommunications (ZL2013-30).

基于表示学习的中文分词

刘春丽, 李晓戈, 刘睿, 范贤, 杜丽萍

西安邮电大学计算机学院, 西安 710121

通讯作者: 李晓戈,E-mail:lixg@xupt.edu.cn
作者简介:刘春丽(1990—),女,山西临汾人,硕士研究生,主要研究方向:自然语言处理、文本数据挖掘;李晓戈(1962—),男,安徽合肥人,教授,博士,主要研究方向:自然语言处理、机器学习、数据挖掘;刘睿(1992—),男,陕西咸阳人,硕士研究生,主要研究方向:自然语言处理、大数据;范贤(1991—),女,陕西咸阳人,硕士研究生,主要研究方向:情感分析、大数据;杜丽萍(1987—),女,陕西宝鸡人,硕士研究生,主要研究方向:自然语言处理、大数据。
基金资助:
国家自然科学基金资助项目（61373116）；陕西省普通高等学校重点学科专项资金资助项目（112-1602）；西安邮电大学研究生创新基金资助项目（ZL2013-30）。

Abstract

Abstract: In order to improve the accuracy and the Out Of Vocabulary (OOV) recognition rate of the Chinese word segmentation, a Chinese word segmentation system based on character representation learning method was proposed. Firstly, the word in the text was mapped to a vector in a high-dimentioanl vecter space using Skip-gram model; then the K-means clustering algorithm was used to acquire clusters of the word vector, and the clustering results were regarded as features of Conditional Random Fields (CRF) model for training. Finally the CRF model was used for word segmentation and OOV recognition. The influences of the word vector dimensions, the number of clusters and different cluster algorithm on word segmentation were analyzed. Experiments were conducted on the 4th CCF Conference on Natural Language Processing & Chinese Computing (NLPCC2015) corpus. Experimental results show that the proposed system can effectively improve Chinese short text segmentation performance without using external knowledge, the F-value and the OOV recognition rate achieve to 95.67% and 94.78% respectively.

Key words: representation learning, word vector, clustering, Conditional Random Field (CRF), Chinese word segmentation

摘要： 为提高中文分词的准确率和未登录词（OOV）识别率，提出了一种基于字表示学习方法的中文分词系统。首先使用Skip-gram模型将文本中的词映射为高维向量空间中的向量；其次用K-means聚类算法将词向量聚类，并将聚类结果作为条件随机场（CRF）模型的特征进行训练；最后基于该语言模型进行分词和未登录词识别。对词向量的维数、聚类数及不同聚类算法对分词的影响进行了分析。基于第四届自然语言处理与中文计算会议（NLPCC2015）提供的微博评测语料进行测试，实验结果表明，在未利用外部知识的条件下，分词的F值和OOV识别率分别达到95.67%和94.78%，证明了将字的聚类特征加入到条件随机场模型中能有效提高中文短文本的分词性能。

关键词: 表示学习, 词向量, 聚类, 条件随机场, 中文分词

CLC Number:

TP391.1

LIU Chunli, LI Xiaoge, LIU Rui, FAN Xian, DU Liping. Chinese word segment based on character representation learning[J]. Journal of Computer Applications, 2016, 36(10): 2794-2798.

刘春丽, 李晓戈, 刘睿, 范贤, 杜丽萍. 基于表示学习的中文分词[J]. 计算机应用, 2016, 36(10): 2794-2798.

References

[1] 魏晓宁.基于隐马尔可夫模型的中文分词研究[J]. 电脑知识与技术(学术交流), 2007, 4(11):885-886.(WEI X N. HMM-based of study on Chinese language classifying words [J]. Computer Knowledge and Technology (Academic Exchange), 2007, 4(11):885-886.)
[2] ANDREW M, DAYNE F, FEMANDO P. Maximum entropy Markov models for information extraction and segmentation [C]//Proceedings of the Seventeenth International Conference on Machine Learning. New York: ACM, 2000: 591-598.
[3] LAFFERTY J, MCCALLUM A, PEREIRA F. Conditional random fields: probabilistic models for segmenting and labeling sequence data [C]//Proceedings of the 18th International Conference on Machine Learning. New York: ACM, 2001:282-289.
[4] HINTON G E, SALAKHUTDINOV R R. Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313(5786): 504-507.
[5] TURIAN J, RATINOV L, BENGIO Y. Word representations: a simple and general method for semi-supervised learning[C]//Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2010: 384-394.
[6] KOO T, CARRERAS X, COLLINS M. Simple semi-supervised dependency parsing[C]//Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2008: 595-603.
[7] MANN G S, MCCALLUM A. Generalized expectation criteria for semi-supervised learning of conditional random fields[C]//Proceedings of the 2008 Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2010:1374-1377.
[8] YU D, WANG S, DENG L. Sequential labeling using deep-structured conditional random fields[J]. IEEE Journal of Selected Topics in Signal Processing, 2010, 4(6):965-973.
[9] ZHENG X Q, CHEN H Y, XU T Y. Deep learning for Chinese word segmentation and POS tagging[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle: [s. n.], 2013: 647-657.
[10] 来斯惟, 徐立恒, 陈玉博, 等.基于表示学习的中文分词算法探索[J]. 中文信息学报, 2013, 27(5): 8-14.(LAI S W, XU L H, CHEN Y B, et al. Chinese word segment based on character representation learning [J]. Journal of Chinese Information Processing, 2013, 27(5): 8-14.)
[11] QIU X, QIAN P, YIN L, et al. Overview of the NLPCC 2015 shared task: Chinese word segmentation and POS tagging for micro-blog texts (2015)[EB/OL]. [2015-03-10]. http://arxiv.org/abs/1505.0759.
[12] word2vec[EB/OL]. [2015-03-12]. https://code.google.com/p/word2vec/.
[13] MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality[EB/OL]. [2015-03-10]. https://arxiv.org/abs/1310.4546.
[14] WU X, ZHOU J, SUN Y et al. Generalization of words for Chinese dependency parsing[C]//Proceedings of the 4th CCF Conference on Natural Language Processing and Chinese Computing, LNCS 9362. Berlin: Springer, 2015: 36-46.
[15] MILLER S, GUINNESS J, ZAMANIAN A. Name tagging with word clusters and discriminative training[EB/OL]. [2015-03-10]. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.105.9395.
[16] CRF++[EB/OL]. [2015-03-20]. http://sourceforge.net/projects/crfpp/.
[17] GAO J F, LI M, WU A, et al. Chinese word segmentation and named entity recognition: a pragmatic approach[J]. Computational Linguistics, 2005, 31(4):531-574.
[18] SUN X, WANG H, LI W. Fast online training with frequency-adaptive learning rates for Chinese word segmentation and new word detection[C]//Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2012: 253-262.
[19] 杜丽萍, 李晓戈, 于根, 等.基于互信息改进算法的新词发现对中文分词系统改进[J]. 北京大学学报(自然科学版),2016, 52(1): 35-40.(DU L P, LI X G, YU G, et al. New word detection based on an improved PMI algorithm for enhancing segmentation system [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2016, 52(1): 35-40.)
[20] MIN K R, MA C G, ZHAO T M, et al. BonsonNLP: an ensemble approach for word segmentation and POS tagging[C]//Proceedings of the 4th CCF Conference on Natural Language Processing & Chinese Computing. Berlin: Springer, 2015: 520-526.

Chinese word segment based on character representation learning

基于表示学习的中文分词

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	CHEN Hengheng, NI Zhiwei, ZHU Xuhui, JIN Yuanyuan, CHEN Qian. Differential privacy high-dimensional data publishing method via clustering analysis [J]. Journal of Computer Applications, 2021, 41(9): 2578-2585.
[2]	ZHU Cheng, ZHAO Xiaoqi, ZHAO Liping, JIAO Yuhong, ZHU Yafei, CHENG Jianying, ZHOU Wei, TAN Ying. Classification of functional magnetic resonance imaging data based on semi-supervised feature selection by spectral clustering [J]. Journal of Computer Applications, 2021, 41(8): 2288-2293.
[3]	WANG Zisen, LIANG Ying, LIU Zhengjun, XIE Xiaojie, ZHANG Wei, SHI Hongzhou. Matching method for academic expertise of research project peer review experts [J]. Journal of Computer Applications, 2021, 41(8): 2418-2426.
[4]	ZENG Xiangyin, ZHENG Bochuan, LIU Dan. Detection of left and right railway tracks based on deep convolutional neural network and clustering [J]. Journal of Computer Applications, 2021, 41(8): 2324-2329.
[5]	WANG Wei, ZHAO Erping, CUI Zhiyuan, SUN Hao. Disambiguation method of multi-feature fusion based on HowNet sememe and Word2vec word embedding representation [J]. Journal of Computer Applications, 2021, 41(8): 2193-2198.
[6]	ZHANG Meng, LI Weihua. Influence maximization algorithm based on user interactive representation [J]. Journal of Computer Applications, 2021, 41(7): 1964-1969.
[7]	DAI Yanran, DAI Guoqing, YUAN Yubo. Multi-face foreground extraction method based on skin color learning [J]. Journal of Computer Applications, 2021, 41(6): 1659-1666.
[8]	WANG Jiarui, TAN Guoping, ZHOU Siyuan. Clustered wireless federated learning algorithm in high-speed internet of vehicles scenes [J]. Journal of Computer Applications, 2021, 41(6): 1546-1550.
[9]	ZHANG Rong, ZHANG Xianguo. Opinion spam detection based on hierarchical heterogeneous graph attention network [J]. Journal of Computer Applications, 2021, 41(5): 1275-1281.
[10]	LI Guorong, YE Jimin, ZHEN Yuanting. Time series clustering based on new robust similarity measure [J]. Journal of Computer Applications, 2021, 41(5): 1343-1347.
[11]	MA Jianhong, CAO Wenbin, LIU Yuangang, XIA Shuang. Patent clustering method based on functional effect [J]. Journal of Computer Applications, 2021, 41(5): 1361-1366.
[12]	WANG Zhihe, CHANG Xiaoqing, DU Hui. Adaptive affinity propagation clustering algorithm based on universal gravitation [J]. Journal of Computer Applications, 2021, 41(5): 1337-1342.
[13]	LONG Chaoqi, JIANG Yu, XIE Yu. Improved wavelet clustering algorithm based on peak grid [J]. Journal of Computer Applications, 2021, 41(4): 1122-1127.
[14]	LI Xingfeng, HUANG Yuqing, REN Zhenwen, LI Yihong. Robust multi-view clustering algorithm based on adaptive neighborhood [J]. Journal of Computer Applications, 2021, 41(4): 1093-1099.
[15]	FAN Wei, WANG Huimin, XING Yan. Auto-encoder based multi-view attributed network representation learning model [J]. Journal of Computer Applications, 2021, 41(4): 1064-1070.