基于条件随机场的顿号边界识别

doi:10.11772/j.issn.1001-9081.2015.10.2838

计算机应用 ›› 2015, Vol. 35 ›› Issue (10): 2838-2842.DOI: 10.11772/j.issn.1001-9081.2015.10.2838

• 第十五届中国机器学习会议(CCML2015)论文 • 上一篇下一篇

基于条件随机场的顿号边界识别

莫怡文¹, 姬东鸿², 黄江平²

1. 武汉大学文学院, 武汉 430072;
2. 武汉大学计算机学院, 武汉 430072

收稿日期:2015-05-29 修回日期:2015-08-08 出版日期:2015-10-10 发布日期:2015-10-14
通讯作者: 莫怡文(1978-),女,湖北武汉人,博士研究生,主要研究方向:中文信息处理,mofan@whu.edu.cn
作者简介:姬东鸿(1967-),男,湖北武汉人,教授,博士,CCF会员,主要研究方向:自然语言处理、机器学习;黄江平(1985-),男,湖北武汉人,博士研究生,主要研究方向:自然语言处理。
基金资助:
国家自然科学基金资助项目(61133012,61373108)。

Slight-pause marks boundary identification based on conditional random field

MO Yiwen¹, JI Donghong², HUANG Jiangping²

1. College of Chinese Language and Literature, Wuhan University, Wuhan Hubei 430072, China;
2. Computer School, Wuhan University, Wuhan Hubei 430072, China

Received:2015-05-29 Revised:2015-08-08 Online:2015-10-10 Published:2015-10-14

摘要/Abstract

摘要： 标点符号的边界识别是自然语言处理的重要研究内容,它是分词、语块识别等应用的基础。为了实现中文中用于表示并列成分分割的顿号的边界识别,采用了用于序列分割和标记的条件随机场(CRF)方法进行顿号边界识别。首先对顿号边界识别任务进行了两种类型的描述,然后对顿号语料的标注方法和过程以及特征选择进行了研究,通过采用语料推荐和十折交叉验证两种数据集分配方法分别进行了边界识别实验。实验结果表明,通过条件随机场方法结合选择的边界识别特征能够进行顿号边界识别,其顿号边界识别的F值在基准实验的基础上提高了10.57%,由顿号分隔的词语识别其F值可达85.24%。

关键词: 条件随机场, 顿号, 边界识别, 特征选择

Abstract: The boundary identification of punctuation marks is an important research field of natural language processing. It is the basis of the application of word segmentation and phrase chunking. In order to solve the problem that the boundary identification of Chinese slight-pause marks which split the coordinate words and phrases in Chinese, the Conditional Random Field (CRF) that used for sequence segmentation and labeling was adopted for slight-pause marks boundary identification. At first, the slight-puase marks boundary recognition task was described in two types, and then the slight-puase marks corpus tagging method and process and feature selection were studied. According to the methods of corpus recommendation and ten-fold cross validation, a series of experiments were carried out in slight-pause marks. The experimental result shows that the proposed method plays an effective role in slight-pause marks boundary identification with selected boundary identification features. And F-measure of boundary identification increased by 10.57% on baseline as well as the F-measure of words divided by slight-pause marks achieves 85.24%.

Key words: Conditional Random Field (CRF), slight-pause mark, boundary identification, feature selection

中图分类号:

TP301

莫怡文, 姬东鸿, 黄江平. 基于条件随机场的顿号边界识别[J]. 计算机应用, 2015, 35(10): 2838-2842.

MO Yiwen, JI Donghong, HUANG Jiangping. Slight-pause marks boundary identification based on conditional random field[J]. Journal of Computer Applications, 2015, 35(10): 2838-2842.

参考文献

[1] TIBOR K, JAN S. Unsupervised multilingual sentence boundary detection[J]. Computational Linguistics, 2006, 32(4):485-525.
[2] PENG F, FENG F, McCALLUM A. Chinese segmentation and new word detection using conditional random fields[C]//Proceedings of the 20th International Conference on Computational Linguisitcs. Stroundsburg: Association for Computational Linguistics, 2004:562-568.
[3] ZHOU J, QU W, ZHANG F. Exploting chunk-level features to improve phrase chunking[C]//Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Stroundsburg: Association for Computational Linguistics, 2012:557-567.
[4] XUE N, YANG Y. Chinese sentence segmentation as comma classification[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers. Stroundsburg: Association for Computational Linguistics, 2011,2:631-635.
[5] YANG Y, XUE N. Chinese comma disambiguation for discourse analysis[C]//Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers. Stroundsburg:Association for Computational Linguistics, 2012,1:786-794.
[6] KONG F, ZHOU G. Chinese comma disambiguation on k-best parse trees[M]//Natural Language Processing and Chinese Computing. Berlin: Springer-Verlag,2014: 13-22.
[7] LI X, ZONG C. A hierarchical parsing approach with punctuation processing for long Chinese sentences[J]. Journal of Chinese Information Processing, 2006, 20(4):8-15.
[8] LI Y, GU J, ZHOU G. Adding colon and semicolon label feature to Chinese comma classification[J]. Journal of Chinese Information Processing, 2014, 28(5):215-222.
[9] JIN M, KIM M, KIM D, et al. Segmentation of Chinese long sentences using commas[C]//Proceedings of the 3rd ACL SIGHAN Workshop. New York: ACM Press,2004:1-8.
[10] QIU L, ZHANG Y, JIN P, et al. Multi-view Chinese treebanking [C]//Proceedings of the 25th International Conference on Computational Linguistics. Stroundsburg: Association for Computational Linguistics, 2014:257-268.
[11] General Adminitration of Quality Supervision, Inspection and Quarantine of the People's Republic of China. GB/T15834-2011, General rules for punctuation[S]. Beijing: China Standards Press,2011.(中华人民共和国国家质量监督检验检疫总局.GB/T15834-2011,标点符号的一般规则[S].北京:中国标准出版社,2011.)
[12] XUE N, XIA F, CHIOU F, et al. The penn Chinese TreeBank: phrase structure annotation of a large corpus[J]. Natural Language Engineering, 2005,11(2):207-238.
[13] SUN X, MATSUZAKI T, LI W. Latent structured perceptrons for large-scale learning with hidden information[J]. IEEE Transactions on Knowledge and Data Engineering, 2013,25(9):2063-2075.
[14] SUN X. Structure regularization for structured prediction[EB/OL]. [2015-01-01]. http://papers.nips.cc/paper/5563-structure-regularization-for-structured-prediction.pdf.
[15] LAFFERTY J, McCALLUM A, PERNANDO F C N. Conditional random fields: probabilistic models for segmenting and labeling sequence data[EB/OL]. [2015-01-01]. http://repository.upenn.edu/cgi/viewcontent.cgi?article=1162&context=cis_papers.
[16] SHA F, FERNANDO P. Shallow parsing with conditional random fields[C]//Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology. Stroundsburg: Association for Computational Linguistics, 2003:134-141.
[17] McCALLUM A. Efficiently inducing features of conditional random fields[C]//Proceedings of the Nineteenth Conference on Uncertainty in Artificial Intelligence. San Francisco: Morgan Kaufmann Publishers, 2003:403-410.
[18] SUTTON C, ROHANIMANESH K, McCALLUM A. Dynamic conditional random fields: factorized probabilistic models for labeling and segmenting sequence data[C]//Proceedings of the 21st International Conference on Machine Learning. San Francisco: Morgan Kauffmann Publishers, 2004.
[19] SUTTON C, McCALLUM A, ROHANIMANESH K. Dynamic conditional random fields: factorized probabilistic models for labeling and segmentation sequence[J]. Journal of Machine Learning Research, 2007,8:693-723.
[20] FLEISS J L. Measuring nominal scale agreement among many raters[J]. Psychological Bulletin, 1971, 76(5):378-382.

基于条件随机场的顿号边界识别

Slight-pause marks boundary identification based on conditional random field

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	湛航, 何朗, 黄樟灿, 李华峰, 张蔷, 谈庆. 改进的基于层次距离的基因表达式编程特征选择分类算法[J]. 计算机应用, 2021, 41(9): 2658-2667.
[2]	祝承, 赵晓琦, 赵丽萍, 焦玉宏, 朱亚飞, 陈建英, 周伟, 谭颖. 基于谱聚类半监督特征选择的功能磁共振成像数据分类[J]. 计算机应用, 2021, 41(8): 2288-2293.
[3]	李蒙蒙, 秦伟, 刘艺, 刁兴春. 结合头脑风暴优化的混合蚁群优化算法[J]. 计算机应用, 2021, 41(8): 2412-2417.
[4]	林筠超, 万源. 基于图结构优化的自适应多度量非监督特征选择方法[J]. 计算机应用, 2021, 41(5): 1282-1289.
[5]	贾鹤鸣, 姜子超, 李瑶, 孙康健. 基于改进斑点鬣狗优化算法的同步优化特征选择[J]. 计算机应用, 2021, 41(5): 1290-1298.
[6]	张志浩, 林耀进, 卢舜, 郭晨, 王晨曦. 缺失标记下基于类属属性的多标记特征选择[J]. 计算机应用, 2021, 41(10): 2849-2857.
[7]	黄学雨, 徐浩特, 陶剑文. 具有特征选择的多源自适应分类框架[J]. 计算机应用, 2020, 40(9): 2499-2506.
[8]	顾桐, 许国良, 李万林, 李家浩, 王志愿, 雒江涛. 基于集成LightGBM和贝叶斯优化策略的房价智能评估模型[J]. 计算机应用, 2020, 40(9): 2762-2767.
[9]	肖跃雷, 张云娇. 基于特征选择和超参数优化的恐怖袭击组织预测方法[J]. 计算机应用, 2020, 40(8): 2262-2267.
[10]	刘丹, 姚立霜, 王云锋, 裴作飞. 面向类不平衡流量数据的分类模型[J]. 计算机应用, 2020, 40(8): 2327-2333.
[11]	汪志远, 降爱莲, 奥斯曼·穆罕默德. 基于正则互表示的无监督特征选择方法[J]. 计算机应用, 2020, 40(7): 1896-1900.
[12]	胡甜甜, 但雅波, 胡杰, 李想, 李少波. 基于注意力机制的Bi-LSTM结合CRF的新闻命名实体识别及其情感分类[J]. 计算机应用, 2020, 40(7): 1879-1883.
[13]	谢琪, 徐旭, 程耕国, 陈和平. 基于新的森林优化算法的特征选择算法[J]. 计算机应用, 2020, 40(5): 1266-1271.
[14]	曹堉, 王成, 王鑫, 高悦尔. 基于时空节点选择和深度学习的城市道路短时交通流预测[J]. 计算机应用, 2020, 40(5): 1488-1493.
[15]	曾元鹏, 王开军, 林崧. 面向二类区分能力的干扰熵特征选择方法[J]. 计算机应用, 2020, 40(3): 626-630.