Abstract: To provide better Web search services, the key techniques of the full-text retrieval were studied and a middleware was designed and implemented. By using a multi-thread website crawler program, the Web pages of the given URLs were collected. Bloom-Filter algorithm was employed to get rid of large-scale duplicate URLs in the collected Web pages. A new content extraction approach based on the Web tags was presented to extract the full-text content of Web pages for indexing and searching. The experimental results verify the efficiency of the content extraction method. Furthermore, to improve the search experience of users, many personalized search assistances were provided by this middleware. Boso, a blog search engine, was developed to test and verify the presented middleware. The results show that the presented middleware can be applied to actual search engines.

Key words: Web full-text retrieval, middleware, URL filter, Web tag, content extraction

摘要： 为了更好地为Web站内检索提供服务，对Web全文检索的关键技术进行了研究，设计并实现了一个用于Web全文检索的中间件。该中间件通过多线程网页爬虫程序来抓取网页；对抓取结果采用基于Bloom-Filter的大规模URL去重算法来进行高效消重；应用所提出的新的基于标签的正文提取方法来进行网页正文提取，实验结果表明该正文提取方法是有效可行的；此外，为了改善用户的搜索体验，该中间件还提供了一些个性化搜索辅助功能。最后，实现了一个完整的博客搜索系统Boso(博搜)来验证中间件的性能，结果说明该中间件是可以投入实际应用的。

关键词: Web全文检索, 中间件, URL去重, 网页标签, 正文提取