关键词: Web数据抓掘, 并行爬虫, 增量更新策略, 余弦向量法

Abstract: This paper discussed the architecture of parallel Web crawler system. Incremental crawling method was used to the system to improve the efficiency of massive information updating. Meanwhile, considering the difference of crawler in the system and with the aim of fully usage of crawler in cluster system, Cosine vector parallel crawling model was introduced to solve this problem. After giving the definitions of crawling task vector and crawler vector, relevant parallel crawling algorithms were designed. The results confirm that the system is effective in distribution adaptability and runs well in maintaining the "freshness" of the Web repository.

Key words: Web data mining, parallel crawler, increment update strategy, cosine vector