Focused crawling method based on improved C4.5 exploiting anchor text

• Typical applications • Previous Articles Next Articles

Focused crawling method based on improved C4.5 exploiting anchor text

Received:2006-06-05 Revised:2006-07-26 Online:2006-12-01 Published:2006-12-01

一种基于锚文本和改进C4.5决策树算法的主题爬行方法

刘金红陆余良

解放军电子工程学院网络系解放军电子工程学院网络系

通讯作者: 刘金红

Abstract

Abstract: A new focused crawling method based on anchor text and improved C4.5 decision tree algorithm was proposed. It exploited the anchor text of URL to train the decision tree, and then applied the decision tree model to decide whether a downloaded page was on topic and how to choose the next URL to visit. Finally, a prototype system named DTFC based on this method was implemented, and experiments in four university websites were carried out in allusion to "academic report". The experimental results show that DTFC outperforms two standard crawlers for focused crawling.

Key words: focused crawler, anchor text, decision tree

摘要： 提出了一种基于锚文本和改进C4.5决策树算法的主题爬行方法：基于锚文本词项集训练决策树，然后基于决策树模型来计算网页的主题相关性和待爬行URL的优先级顺序。最后，应用该方法在四所大学网站网页数据集上针对“学术报告”主题进行了主题爬行实验，并与两种标准的网络爬虫进行了性能对比，实验结果验证了该方法的有效性。

关键词: 主题网络爬虫, 锚文本, 决策树

. Focused crawling method based on improved C4.5 exploiting anchor text[J]. Journal of Computer Applications.

刘金红陆余良 . 一种基于锚文本和改进C4.5决策树算法的主题爬行方法[J]. 计算机应用.

[1]	Xin XIE, Xianyong ZHANG, Xuanye WANG, Pengfei TANG. Neighborhood decision tree construction algorithm based on variable-precision neighborhood equivalent granules [J]. Journal of Computer Applications, 2022, 42(2): 382-388.
[2]	WANG Yahui, QIAN Yuhua, LIU Guoqing. Ordinal decision tree algorithm based on fuzzy advantage complementary mutual information [J]. Journal of Computer Applications, 2021, 41(10): 2785-2792.
[3]	ZHOU Xiang, ZHAI Junhai, HUANG Yajie, SHEN Ruicai, HOU Yingzhen. Instance selection algorithm for big data based on random forest and voting mechanism [J]. Journal of Computer Applications, 2021, 41(1): 74-80.
[4]	LIU Jingfa, GU Yaoping, LIU Wenjie. Focused crawler method combining ontology and improved Tabu search for meteorological disaster [J]. Journal of Computer Applications, 2020, 40(8): 2255-2261.
[5]	Yi ZHAO, Xing DUAN, Shiyi XIE, Chunlin LIANG. Traffic image semantic retrieval method based on specific object self-recognition [J]. Journal of Computer Applications, 2020, 40(2): 553-560.
[6]	WANG Wei, XIE Yaobin, YIN Qing. Decision tree improvement method for imbalanced data [J]. Journal of Computer Applications, 2019, 39(3): 623-628.
[7]	MO Zan, GAI Yanrong, FAN Guanlong. Credit card fraud classification based on GAN-AdaBoost-DT imbalanced classification algorithm [J]. Journal of Computer Applications, 2019, 39(2): 618-622.
[8]	GUO Bingnan, WU Guangchao. Classification of online loan based on improved cost-sensitive decision tree [J]. Journal of Computer Applications, 2019, 39(10): 2888-2892.
[9]	YAN Hongwen, SHENG Chenggong. Short-term bus load forecasting based on hierarchical clustering method and extreme learning machine [J]. Journal of Computer Applications, 2018, 38(8): 2437-2441.
[10]	SHE Rongrong, ZHANG Liping, HOU Min, YAN Sheng. Recommending clone refactoring method based on decision tree [J]. Journal of Computer Applications, 2018, 38(7): 2037-2043.
[11]	XU Guangxian, ZHAO Yue, LAI Junning. Deterministic layered construction algorithm based on network coding [J]. Journal of Computer Applications, 2018, 38(3): 769-775.
[12]	DUAN Dagao, GAI Xinxin, HAN Zhongming, LIU Bingxin. Micro-blog misinformation detection based on gradient boost decision tree [J]. Journal of Computer Applications, 2018, 38(2): 410-414.
[13]	HUANG Yuyang, DONG Minggang, JING Chao. Genetic instance selection algorithm for K-nearest neighbor classifier [J]. Journal of Computer Applications, 2018, 38(11): 3112-3118.
[14]	WANG Ting, WANG Qi, HUANG Yueqi, YIN Yichao, GAO Ju. Automatic hyponymy extracting method based on symptom components [J]. Journal of Computer Applications, 2017, 37(10): 2999-3005.
[15]	CHEN Huanhuan, CHEN Xiaohong, RUAN Tong, GAO Daqi, WANG Haofen. Knowledge driven automatic annotating algorithm for game strategies [J]. Journal of Computer Applications, 2017, 37(1): 278-283.