Web robot detection algorithm based on episode analysis

Journal of Computer Applications ›› 2005, Vol. 25 ›› Issue (07): 1580-1583.

• Database Technology • Previous Articles Next Articles

Web robot detection algorithm based on episode analysis

GUO Wei-gang^1，2， JU Shi-guang²

1. Information and Educational Technology Center, Foshan University; 2. School of Computer Science and Telecommunicaiton Engineering, Jiangsu University

Received:2005-01-04 Online:2011-04-22 Published:2005-07-01

一个基于事务分析的Web Robot检测算法

郭伟刚^1,2，鞠时光²

1.佛山科学技术学院信息与教育技术中心，广东佛山 528000；
2.江苏大学计算机科学与通信工程学院，江苏镇江 212013

作者简介:郭伟刚（1966-），男，江苏江阴人，副教授，主要研究方向：数据挖掘、计算机图形学、计算机在教育中的应用；鞠时光（1955-），男，江苏南通人，教授，博士生导师，主要研究方向：数据库、可视化语言、信息安全

Abstract

Abstract:

After analyzing the navigational patterns of Web robots, the feature was found that the access sequence of robots usually didnt satisfy the hyperlink relations. The concept of episode was defined and a new algorithm based on episode analysis was proposed. The experiments show that the new algorithm can detect the unknown robots and unfriendly robots who do not obey the standard for robot exclusion.

Key words: search engine, Web robot, user episode, detection, Web log

摘要：

分析了网络机器人（Web Robot）的访问行为特点，发现Robot的访问序列一般不会形成具有链接关系的路径。在定义了用户事务的概念的基础上，提出了一个基于事务分析的检测算法。经实验验证，该算法可以有效地检测未知的和不遵守网络机器人排斥标准的Robot。

关键词: 搜索引擎；网络机器人；用户事务；检测；Web日志

GUO Wei-gang，JU Shi-guang. Web robot detection algorithm based on episode analysis[J]. Journal of Computer Applications, 2005, 25(07): 1580-1583.

郭伟刚，鞠时光. 一个基于事务分析的Web Robot检测算法[J]. 计算机应用, 2005, 25(07): 1580-1583.

[1]	NI Ping, CHEN Wei. Reflective cross-site scripting vulnerability detection based on fuzzing test [J]. Journal of Computer Applications, 2021, 41(9): 2594-2601.
[2]	ZHANG Yongbin, CHANG Wenxin, SUN Lianshan, ZHANG Hang. Detection method of domains generated by dictionary-based domain generation algorithm [J]. Journal of Computer Applications, 2021, 41(9): 2609-2614.
[3]	LIU Xiaoguang, JIN Shaokang, WEI Zihui, LIANG Tie, WANG Hongrui, LIU Xiuling. Real-time fall detection method based on threshold and extremely randomized tree [J]. Journal of Computer Applications, 2021, 41(9): 2761-2766.
[4]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[5]	MA Jialiang, CHEN Bin, SUN Xiaofei. General object detection framework based on improved Faster R-CNN [J]. Journal of Computer Applications, 2021, 41(9): 2712-2719.
[6]	ZHANG Shipeng, LI Yongzhong, DU Xiangtong. Intrusion detection model based on semi-supervised learning and three-way decision [J]. Journal of Computer Applications, 2021, 41(9): 2602-2608.
[7]	WANG Hebing, ZHANG Chunmei. Facial landmark detection based on ResNeXt with asymmetric convolution and squeeze excitation [J]. Journal of Computer Applications, 2021, 41(9): 2741-2747.
[8]	FAN Wei, LI Chenxuan, XING Yan, HUANG Rui, PENG Hongjian. Binary classification to multiple classification progressive detection network for aero-engine damage images [J]. Journal of Computer Applications, 2021, 41(8): 2352-2357.
[9]	HU Tianjie, HU Wenjun, WANG Shitong. Distribution entropy penalized support vector data description [J]. Journal of Computer Applications, 2021, 41(8): 2212-2218.
[10]	ZENG Xiangyin, ZHENG Bochuan, LIU Dan. Detection of left and right railway tracks based on deep convolutional neural network and clustering [J]. Journal of Computer Applications, 2021, 41(8): 2324-2329.
[11]	CHEN Jing, MAO Yingchi, CHEN Hao, WANG Longbao, WANG Zicheng. Dam defect object detection method based on improved single shot multibox detector [J]. Journal of Computer Applications, 2021, 41(8): 2366-2372.
[12]	REN Qiuru, YANG Wenzhong, WANG Chuanjian, WEI Wenyu, QIAN Yunyun. Review of remote sensing image change detection [J]. Journal of Computer Applications, 2021, 41(8): 2294-2305.
[13]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[14]	MENG Fan, CHEN Guang, WANG Yong, GAO Yang, GAO Dequn, JIA Wenlong. Multi-granularity temporal structure representation based outlier detection method for prediction of oil reservoir [J]. Journal of Computer Applications, 2021, 41(8): 2453-2459.
[15]	XIAO Zhenyuan, WANG Yihan, LUO Jianqiao, XIONG Ying, LI Bailin. RefineDet based on subsection weighted loss function [J]. Journal of Computer Applications, 2021, 41(7): 1928-1932.

Web robot detection algorithm based on episode analysis

一个基于事务分析的Web Robot检测算法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics