Automatic detection of online duplication documents and its application in enterprise data warehouse

doi:10.3724/SP.J.1087.2005.01713

Journal of Computer Applications ›› 2005, Vol. 25 ›› Issue (07): 1713-1715.DOI: 10.3724/SP.J.1087.2005.01713

• Typical applications • Previous Articles Next Articles

Automatic detection of online duplication documents and its application in enterprise data warehouse

BAI Guang-hui^1,2，LIAN Hao²，LIU Yue³，CHENG Xue-qi³

1. China Netcom Group Labs, Beijing 100036; 2. Graduate School, Chinese Academy of Sciences; 3. Institute of Computing Technology, Chinese Academy of Sciences

Received:2004-12-24 Revised:2005-03-11 Online:2005-07-01 Published:2005-07-01

网页查重技术在企业数据仓库中的应用

白广慧^1,2，连浩²，刘悦³，程学旗³

1.中国网通集团研究院，北京100036； 2.中国科学院研究生院，北京 100039；
3.中国科学院计算技术研究所，北京100080

作者简介:白广慧(1976-)，女（回），山东德州人，工程师，主要研究方向：计算机及应用、网络安全；连浩（1980-)，女，湖北武汉人，硕士研究生，主要研究方向：中文处理、网络安全；刘悦(1971-)，女，山东泰安人，博士，主要研究方向：P2P网络、Web搜索引擎；程学旗(1971-)，男，安徽安庆人，研究员，主要研究方向：网络与信息安全、大规模内容计算、P2P网络、信息网格.

Abstract

Abstract:

Three general methods to detect duplicate Web pages were introduced. The similarity search technique was used to detect duplicate information automatically in enterprise data warehouse. The results indicate that the similarity search method is fit for intelligent pretreatment of enterprise intelligence data.

Key words: data warehouse, online duplicate documents detection, Support Vector Machine (SVM, Vector Space Model（VSM）

摘要：

介绍了处理网页排重的三类通用方法，并介绍了在企业数据仓库系统中，通过利用相似性检索技术实现情报资料自动排重的应用。通过对测试结果的评估表明，这种基于相似性检索技术的自动排重的方法能够达到较好的效果，实现了企业情报资料智能化预处理的应用。

关键词: 数据仓库, 网页查重, 支持向量机, 向量空间模型

CLC Number:

TP391.3

BAI Guang-hui，LIAN Hao，LIU Yue，CHENG Xue-qi. Automatic detection of online duplication documents and its application in enterprise data warehouse[J]. Journal of Computer Applications, 2005, 25(07): 1713-1715.

白广慧，连浩，刘悦，程学旗. 网页查重技术在企业数据仓库中的应用[J]. 计算机应用, 2005, 25(07): 1713-1715.

[1]	Min SUN, Qian CHENG, Xining DING. CBAM-CGRU-SVM based malware detection method for Android [J]. Journal of Computer Applications, 2024, 44(5): 1539-1545.
[2]	Xueyu HUANG, Huaiyu HE, Huimin LIN, Jinshui CHEN. Classification and recognition method of copper alloy metallograph based on feature aggregation [J]. Journal of Computer Applications, 2023, 43(8): 2593-2601.
[3]	Lei YANG, Hongdong ZHAO, Kuaikuai YU. End-to-end speech emotion recognition based on multi-head attention [J]. Journal of Computer Applications, 2022, 42(6): 1869-1875.
[4]	Zhen QU, Kunting LI, Zhixi FENG. Remote sensing image scene classification based on effective channel attention [J]. Journal of Computer Applications, 2022, 42(5): 1431-1439.
[5]	Guifang QIAO, Shouming HOU, Yanyan LIU. Facial expression recognition algorithm based on combination of improved convolutional neural network and support vector machine [J]. Journal of Computer Applications, 2022, 42(4): 1253-1259.
[6]	Wang TAN, Yi LI. Synthesis of loop bound functions for loop programs [J]. Journal of Computer Applications, 2022, 42(2): 565-573.
[7]	Qian GE, Guangbin ZHANG, Xiaofeng ZHANG. Automatic feature selection algorithm based on interaction of ReliefF with maximum information coefficient and SVM [J]. Journal of Computer Applications, 2022, 42(10): 3046-3053.
[8]	Hongfei JIA, Xi LIU, Yu WANG, Hongbing XIAO, Suxia XING. Application of 3DPCANet in image classification of functional magnetic resonance imaging for Alzheimer’s disease [J]. Journal of Computer Applications, 2022, 42(1): 310-315.
[9]	JIA Heming, JIANG Zichao, LI Yao, SUN Kangjian. Simultaneous feature selection optimization based on improved spotted hyena optimizer algorithm [J]. Journal of Computer Applications, 2021, 41(5): 1290-1298.
[10]	YUAN Qianqian, DENG Hongmin, WANG Xiaohang. Citrus disease and insect pest area segmentation based on superpixel fast fuzzy C-means clustering and support vector machine [J]. Journal of Computer Applications, 2021, 41(2): 563-570.
[11]	Hongliang CAO, Ying ZHANG, Bin WU, Fanyu LI, Xubo NA. Prediction method of liver transplantation complications based on transfer component analysis and support vector machine [J]. Journal of Computer Applications, 2021, 41(12): 3608-3613.
[12]	Kai LI, Jie LI. Structure-fuzzy multi-class support vector machine algorithm based on pinball loss [J]. Journal of Computer Applications, 2021, 41(11): 3104-3112.
[13]	TONG Lin, GUAN Zheng. Fuzzy granulation prediction of traffic flow based on improved whale optimization support vector machine [J]. Journal of Computer Applications, 2021, 41(10): 2919-2927.
[14]	ZHANG Jianming, SHI Yuanhao, XU Zhengyi, WEI Jianming. Adaptive UWB/PDR fusion positioning algorithm based on error prediction [J]. Journal of Computer Applications, 2020, 40(6): 1755-1762.
[15]	WANG Yang, ZHAO Hongdong. Human activity recognition based on improved particle swarm optimization-support vector machine and context-awareness [J]. Journal of Computer Applications, 2020, 40(3): 665-671.

Automatic detection of online duplication documents and its application in enterprise data warehouse

网页查重技术在企业数据仓库中的应用

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics