融合本体和改进禁忌搜索策略的气象灾害主题爬虫方法

doi:10.11772/j.issn.1001-9081.2019122238

计算机应用 ›› 2020, Vol. 40 ›› Issue (8): 2255-2261.DOI: 10.11772/j.issn.1001-9081.2019122238

融合本体和改进禁忌搜索策略的气象灾害主题爬虫方法

刘景发^1,2, 顾瑶平¹, 刘文杰¹

1. 南京信息工程大学计算机与软件学院, 南京 210044;
2. 广东外语外贸大学信息科学与技术学院, 广州 510006

收稿日期:2020-01-07 修回日期:2020-03-10 出版日期:2020-08-10 发布日期:2020-05-14
通讯作者: 刘景发(1972-),男,湖南衡阳人,教授,博士生导师,博士,CCF高级会员,主要研究方向:网络爬虫、智能计算、本体;jfliu@gdufs.edu.cn
作者简介:顾瑶平(1995-),女,江苏盐城人,硕士研究生,主要研究方向:智能计算、网络爬虫;刘文杰(1979-),男,湖北荆州人,副教授,博士,主要研究方向:量子机器学习、量子算法与软件。
基金资助:
国家社会科学基金重大招标项目（16ZDA047）；江苏省自然科学基金资助项目（BK20181409）；广州市基础与应用基础研究项目。

Focused crawler method combining ontology and improved Tabu search for meteorological disaster

LIU Jingfa^1,2, GU Yaoping¹, LIU Wenjie¹

1. School of Computer and Software, Nanjing University of Information Science and Technology, Nanjing Jiangsu 210044, China;
2. School of Information Science and Technology, Guangdong University of Foreign Studies, Guangzhou Guangdong 510006, China

Received:2020-01-07 Revised:2020-03-10 Online:2020-08-10 Published:2020-05-14
Supported by:
This work is partially supported by the Major Program of the National Social Science Foundation of China (16ZDA047), the Natural Science Foundation of Jiangsu Province (BK20181409), the Project of Guangzhou Basic and Applied Fundamental Research.

摘要/Abstract

摘要： 针对传统主题爬虫方法容易陷入局部最优和主题描述不足的问题，提出一种融合本体和改进禁忌搜索策略（On-ITS）的主题爬虫方法。首先利用本体语义相似度计算主题语义向量，基于超级文本标记语言（HTML）网页文本特征位置加权构建网页文本特征向量，然后采用向量空间模型计算网页的主题相关度。在此基础上，计算锚文本主题相关度以及链接指向网页的PR值，综合分析链接优先度。另外，为了避免爬虫陷入局部最优，设计了基于ITS的主题爬虫，优化爬行队列。以暴雨灾害和台风灾害为主题，在相同的实验环境下，基于On-ITS的主题爬虫方法比对比算法的爬准率最多高58%，最少高8%，其他评价指标也很好。基于On-ITS的主题爬虫方法能有效提高获取领域信息的准确性，抓取更多与主题相关的网页。

关键词: 主题爬虫, 禁忌搜索, 本体, 主题相关度, 气象灾害

Abstract: Considering the problems that the traditional focused crawler is easy to fall into local optimum and has insufficient topic description, a focused crawler method combining Ontology and Improved Tabu Search (On-ITS) was proposed. First, the topic semantic vector was calculated by ontology semantic similarity, and the Web page text feature vector was constructed by Hyper Text Markup Language (HTML) Web page text feature position weighting. Then, the vector space model was used to calculate the topic relevance of Web pages. On this basis, in order to analyze the comprehensive priority of link, the topic relevance of the link anchor text and the PR (PageRank) value of Web page to the link were calculated. In addition, to avoid the crawler falling into local optimum, the focused crawler based on ITS was designed to optimize the crawling queue. Experimental results of the focused crawler on the topics of rainstorm disaster and typhoon disaster show that, under the same environment, the accuracy of the On-ITS method is higher than those of the contrast algorithms by maximum of 58% and minimum of 8%, and other evaluation indicators of the proposed algorithm are also very excellent. On-ITS focused crawler method can effectively improve the accuracy of obtaining domain information and catch more topic-related Web pages.

Key words: focused crawler, Tabu search, ontology, topic relevance, meteorological disaster

中图分类号:

TP391.1

刘景发, 顾瑶平, 刘文杰. 融合本体和改进禁忌搜索策略的气象灾害主题爬虫方法[J]. 计算机应用, 2020, 40(8): 2255-2261.

LIU Jingfa, GU Yaoping, LIU Wenjie. Focused crawler method combining ontology and improved Tabu search for meteorological disaster[J]. Journal of Computer Applications, 2020, 40(8): 2255-2261.

参考文献

[1] DU Y, HU Q, LI X, et al. Ranking Web page with path trust knowledge graph[C]//Proceeding of the 5th International Conference on Intelligent Science and Big Data Engineering, LNCS 9243. Cham:Springer, 2015:66-75.
[2] LIU W, DU Y. A novel focused crawler based on cell-like membrane computing optimization algorithm[J]. Neurocomputing, 2014, 123:266-280.
[3] DU Y, PEN Q, GAO Z. A topic-specific crawling strategy based on semantics similarity[J]. Data and Knowledge Engineering, 2013, 88:75-93.
[4] 费晨杰,刘柏嵩. 基于LDA扩展主题词库的主题爬虫研究[J]. 计算机应用与软件, 2018, 35(4):49-54. (FEI C J, LIU B S. Focused crawler based on LDA extended topic terms[J]. Computer Applications and Software, 2018, 35(4):49-54.)
[5] 程元堃,廖闻剑,程光. 词向量聚类加权Shark-search的主题爬虫策略研究[J]. 计算机与数字工程, 2018, 46(1):144-148. (CHENG Y K, LIAO W J, CHENG G. Strategy of focused crawler with word embedding clustering weighted in shark-search algorithm[J]. Computer and Digital Engineering, 2018, 46(1):144-148.)
[6] ZHANG W, CHEN Y. Bayes topic prediction model for focused crawling of vertical search engine[C]//Proceeding of the 2014 IEEE Computers, Communications and IT Applications Conference. Piscataway:IEEE, 2014:294-299.
[7] DU Y, LIU W, LV X, et al. An improved focused crawler based on semantic similarity vector space model[J]. Applied Soft Computing, 2015, 36:392-407.
[8] 王冲,纪仙慧. 基于用户兴趣与主题相关的PageRank算法改进研究[J]. 计算机科学, 2016, 43(3):275-278, 312. (WANG C, JI X H. Improved PageRank algorithm based on user interest and topic[J]. Computer Science, 2016, 43(3):275-278, 312.)
[9] 刘昊,洪宇,姚亮,等. 基于HITS算法的双语句对挖掘优化方法[J]. 中文信息学报, 2017, 31(2):25-35. (LIU H, HONG Y, YAO L, et al. HITS-based optimization method for bilingual corpus mining[J]. Journal of Chinese Information Processing, 2017, 31(2):25-35.)
[10] YUAN Z, ZHANG W, FU H, et al. A PageRank-improved ranking algorithm based on cheating similarity and cheating relevance[C]//Proceeding of the IEEE/ACIS 16th International Conference on Computer and Information Science. Piscataway:IEEE, 2017:257-263.
[11] ZHANG X, YU H, ZHANG C, et al. An improved weighted HITS algorithm based on similarity and popularity[C]//Proceeding of the 2nd International Multi-symposiums on Computer and Computational Sciences. Piscataway:IEEE, 2007:477-480.
[12] SEYFI A, PATEL A, JÚNIOR J C. Empirical evaluation of the link and content-based focused treasure-crawler[J]. Computer Standards and Interfaces, 2016, 44:54-62.
[13] 李璐,张国印,李正文. 基于SVM的主题爬虫技术研究[J]. 计算机科学, 2015, 42(2):118-122. (LI L, ZHANG G Y, LI Z W. Research on focused crawling technology based on SVM[J]. Computer Science, 2015, 42(2):118-122.)
[14] RAWAT S, PATIL D R. Efficient focused crawling based on best first search[C]//Proceeding of the 3rd IEEE International Advance Computing Conference. Piscataway:IEEE, 2013:908-911.
[15] 荆文鹏,王育坚,董伟伟. 自适应遗传算法在主题爬虫搜索策略中的应用研究[J]. 计算机科学, 2016, 43(8):254-257. (JING W P, WANG Y J, DONG W W. Research on adaptive genetic algorithm in application of focused crawler search strategy[J]. Computer Science, 2016, 43(8):254-257.)
[16] ZHENG S. Genetic and ant algorithms based focused crawler design[C]//Proceeding of the 2nd International Conference on Innovations in Bio-Inspired Computing and Applications. Piscataway:IEEE, 2011:374-378.
[17] 东熠,刘景发,刘文杰. 基于多目标蚁群算法的主题爬虫策略[J/OL]. 计算机工程.[2020-03-30].https://kns.cnki.net/KCMS/detail/31.1289.TP.20191108.1133.002.html. (DONG Y, LIU J F, LIU W J. Focused crawler strategy based on multi-objective ant colony algorithm[J/OL]. Computeer Engineering.[2020-03-30].https://kns.cnki.net/KCMS/detail/31.1289.TP.20191108.1133.002.html.)
[18] 刘景发,李帆,蒋盛益. 基于综合优先度和主机信息的暴雨灾害主题退火爬虫算法[J]. 计算机科学, 2019, 46(2):215-222. (LIU J F, LI F, JIANG S Y. Focused annealing crawler algorithm for rainstorm disasters based on comprehensive priority and host information[J]. Computer Science, 2019, 46(2):215-222.)
[19] KAND X, MIAO D. A study on information granularity in formal concept analysis based on concept-bases[J]. Knowledge-Based Systems, 2016, 105:147-159
[20] KUMAR M, VIG R. Term-frequency Inverse-document frequency Definition Semantic (TIDS) based focused Web crawler[C]//Proceeding of the 4th International Conference on Computing and Communication Systems, CCIS 270. Berlin:Springer, 2012:31-36.
[21] 马雷雷,李宏伟,连世伟,等. 一种基于本体语义的灾害主题爬虫策略[J]. 计算机工程, 2016, 42(11):50-56. (MA L L, LI H W, LIAN S W, et al. A strategy of disaster focused crawler based on ontology semantics[J]. Computer Engineering, 2016, 42(11):50-56.)
[22] SALEH A I, ABULWAFA A E, AL RAHMAWY M F. A Web page distillation strategy for efficient focused crawling based on Optimized Naive Bayes (ONB) classifier[J]. Applied Soft Computing, 2017, 53:181-204.
[23] 李刚,刘景发. 基于禁忌搜索的启发式算法求解带平衡约束的圆形装填问题[J]. 中国科学:信息科学, 2011, 41(9):1076-1088. (LI G, LIU J F. Heuristic algorithm based on tabu search for the circular packing problem with equilibrium constraints[J]. SCIENTIA SINICA Informationis, 2011, 41(9):1076-1088.)

融合本体和改进禁忌搜索策略的气象灾害主题爬虫方法

Focused crawler method combining ontology and improved Tabu search for meteorological disaster

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	周烁, 仇润鹤, 唐旻俊. 基于禁忌搜索和Q-learning的CR-NOMA系统的功率分配算法[J]. 计算机应用, 2021, 41(7): 2026-2032.
[2]	秦丽, 郝志刚, 李国亮. 国家食品安全标准图谱的构建及关联性分析[J]. 计算机应用, 2021, 41(4): 1005-1011.
[3]	代婷婷, 周乐, 余秦勇, 黄细凤, 谢军, 宋明慧, 刘峤. 基于需求评估响应的循环迭代本体构建方法[J]. 计算机应用, 2020, 40(9): 2712-2718.
[4]	马冬雪, 宋设, 谢振平, 刘渊. 领域本体驱动的招投标网页解析方法[J]. 计算机应用, 2020, 40(6): 1574-1579.
[5]	侯艳辉, 董慧芳, 郝敏, 崔雪莲. 基于本体特征的影评细粒度情感分类[J]. 计算机应用, 2020, 40(4): 1074-1078.
[6]	蔡芸, 刘朋青, 熊禾根. 基于量子遗传混合算法的泊位联合调度[J]. 计算机应用, 2020, 40(3): 897-901.
[7]	赵一, 段兴, 谢仕义, 梁春林. 面向特定目标自识别的交通图像语义检索方法[J]. 计算机应用, 2020, 40(2): 553-560.
[8]	马建刚, 张鹏, 马应龙. 基于知识块摘要和词转移距离的高效司法文档分类[J]. 计算机应用, 2019, 39(5): 1293-1298.
[9]	唐晨, 李勇华, 饶梦妮, 胡钢俊. 动态需求跟踪中多义关键词的语义判断方法[J]. 计算机应用, 2019, 39(5): 1299-1304.
[10]	周围, 向丹蕾, 郭梦雨. MIMO-GFDM系统中低复杂度动态禁忌搜索检测算法的改进[J]. 计算机应用, 2019, 39(4): 1133-1137.
[11]	李飞龙, 赵春艳, 范如梦. 基于禁忌搜索算法求解随机约束满足问题[J]. 计算机应用, 2019, 39(12): 3584-3589.
[12]	何东东, 李引珍. 多车型绿色车辆路径问题优化模型[J]. 计算机应用, 2018, 38(12): 3618-3624.
[13]	刘其源, 焦健, 曹宏盛. Android隐式信息流检测的本体模型[J]. 计算机应用, 2018, 38(1): 61-66.
[14]	桑成, 程健, 史一明. 闭环全生命周期管理系统中知识集成与语义标注[J]. 计算机应用, 2017, 37(6): 1728-1734.
[15]	杨小东, 康雁, 柳青, 孙金文. 求解作业车间调度问题的混合帝国主义竞争算法[J]. 计算机应用, 2017, 37(2): 517-522.