Journal of Computer Applications ›› 2016, Vol. 36 ›› Issue (3): 731-734.DOI: 10.11772/j.issn.1001-9081.2016.03.731

Previous Articles     Next Articles

Web spam detection based on random forest and under-sampling ensemble

LU Xiaoyong1, CHEN Musheng2   

  1. 1. School of Software, Nanchang University, Nanchang Jiangxi 330047, China;
    2. Information Engineering School, Nanchang University, Nanchang Jiangxi 330031, China
  • Received:2015-08-10 Revised:2015-10-03 Online:2016-03-10 Published:2016-03-17
  • Supported by:
    This work is partially supported by the Sciences and Technology Support Program of Jiangxi Province (20131102040039).

基于随机森林和欠采样集成的垃圾网页检测

卢晓勇1, 陈木生2   

  1. 1. 南昌大学 软件学院, 南昌 330047;
    2. 南昌大学 信息工程学院, 南昌 330031
  • 通讯作者: 陈木生
  • 作者简介:卢晓勇(1957-),男,江西高安人,教授,博士,主要研究方向:信息管理与信息系统、工业工程;陈木生(1977-),男,江西于都人,博士研究生,主要研究方向:数据挖掘与知识发现、信息管理与信息系统。
  • 基金资助:
    江西省科技支撑计划项目(20131102040039)。

Abstract: In order to solve the problem of imbalance classification and "curse of dimensionality", a binary classifier algorithm based on Random Forest (RF) and under-sampling ensemble was proposed to detect Web spam. Firstly, majority samples in training dataset were sampled into several sub sample sets, each of them was combined with minority samples and several balanced training sample sub sets were generated; then several RF classifiers were trained by these training sample sub sets to classify the testing samples; finally, the testing samples' classifications were determined by voting. Experiments on the WEBSPAM UK-2006 dataset show that the ensemble classifier outperformed RF, Bagging with RF and Adaboost with RF etc., and its accuracy, F1-measure, AUC increased by at least 14%, 13% and 11%. Compared with the winners of Web spam challenge 2007, the ensemble classifier increased F1-measure by at least 1% and reached to the optimum result in AUC.

Key words: Web spam detection, Random Forest (RF), under-sampling, ensemble classifier, machine learning

摘要: 为解决垃圾网页检测过程中的不平衡分类和"维数灾难"问题,提出一种基于随机森林(RF)和欠采样集成的二元分类器算法。首先使用欠采样技术将训练样本集大类抽样成多个子样本集,再将其分别与小类样本集合并构成多个平衡的子训练样本集;然后基于各个子训练样本集训练出多个随机森林分类器;最后用多个随机森林分类器对测试样本集进行分类,采用投票法确定测试样本的最终所属类别。在WEBSPAM UK-2006数据集上的实验表明,该集成分类器算法应用于垃圾网页检测比随机森林算法及其Bagging和Adaboost集成分类器算法效果更好,准确率、F1测度、ROC曲线下面积(AUC)等指标提高至少14%,13%和11%。与Web spam challenge 2007 优胜团队的竞赛结果相比,该集成分类器算法在F1测度上提高至少1%,在AUC上达到最优结果。

关键词: 垃圾网页检测, 随机森林, 欠采样, 集成分类器, 机器学习

CLC Number: