关键词: 陷阱技术, 覆盖率, 爬虫检测, 召回率

Abstract: Spider known as Web crawler, a program for capturing network resources, is widely used in the field of search engines. However, it also raises many problems such as privacy leakage and copyright dissension. Therefore, it is necessary to take measures to detect and restrict the Web crawler's behavior. This paper firstly briefly reviewed the current achievements in Web crawler's detection and the utilization of the trap technique for this purpose. And then, a structural trap technique was proposed to construct Website models and corresponding detection algorithms. Finally the authors measured the sensitivity of the model and summarized its performance. The results indicate that the precision stemming from the structural trap technique is generally consistent with the one from artificial analyses.

Key words: trap techniques, coverage rate, spider detection, recall rate