Download performance optimization in Hadoop distributed file system based on P2P

doi:10.3724/SP.J.1087.2011.02317

Journal of Computer Applications ›› 2011, Vol. 31 ›› Issue (09): 2317-2320.DOI: 10.3724/SP.J.1087.2011.02317

• Network and communications • Previous Articles Next Articles

Download performance optimization in Hadoop distributed file system based on P2P

LIAO Bin,YU Jiong,ZHANG Tao,YANG Xing-yao

College of Information Science and Technology, Xinjiang University, Urumqi Xingjiang 830046, China

Received:2011-03-07 Revised:2011-05-13 Online:2011-09-01 Published:2011-09-01
Contact: LIAO Bin

基于P2P的分布式文件系统下载效率优化

廖彬,于炯,张陶,杨兴耀

新疆大学信息科学与工程学院，乌鲁木齐 830046

通讯作者: 廖彬
作者简介:廖彬(1986-),男,四川内江人，硕士研究生,主要研究方向：数据库、网格与云计算；
于炯(1964-),男,北京人，教授,博士，主要研究方向：网络安全、网格与分布式计算；
张陶(1988-),女,新疆乌鲁木齐人，硕士研究生,主要研究方向：分布式计算、网格计算；
杨兴耀(1984-)，男，新疆乌鲁木齐人，博士研究生，主要研究方向：分布式计算、网格计算。
基金资助:
国家自然科学基金资助项目(61003131;61003138;61073116);新疆大学博士科研启动基金资助项目(BS090153)

Abstract

Abstract: The data block storage mechanism and downloading process in Hadoop Distributed File System (HDFS) cluster were analyzed. In combination with multi-point and multi-threaded Peer-to-Peer (P2P) download idea, an efficiency optimization algorithm was proposed from the aspects of data-block, file and cluster. Concerning the possible imbalanced load problem caused by multi-thread download in HDFS cluster, a download-point selection algorithm was put forward to optimize the download-point selection. The mathematical analysis and experiments prove that the three methods can improve the download efficiency and download-point selection algorithm can achieve loading balance among DataNodes in HDFS cluster.

Key words: cloud computing, Hadoop Distributed File System (HDFS), Peer-to-Peer (P2P), parallel download, load balance

摘要： 对分布式文件系统(HDFS)集群内部数据块存储机制与下载流程进行分析研究，结合P2P多点与多线程下载思想，从数据块、文件、集群三个方面提出了数据下载效率优化算法。考虑到集群内部可能因多线程下载出现的负载均衡问题，提出下载点选择算法以优化下载点的选择。实验结果表明，三种优化算法都能提高下载效率，下载点选择算法能够很好地实现集群内部DataNode负载均衡。

关键词: 云计算, 分布式文件系统, 对等网, 并行下载, 负载均衡

CLC Number:

TP393.027

LIAO Bin YU Jiong ZHANG Tao YANG Xing-yao. Download performance optimization in Hadoop distributed file system based on P2P[J]. Journal of Computer Applications, 2011, 31(09): 2317-2320.

廖彬于炯张陶杨兴耀. 基于P2P的分布式文件系统下载效率优化[J]. 计算机应用, 2011, 31(09): 2317-2320.

[1]	Jiaxing LU, Hua DAI, Yuanlong LIU, Qian ZHOU, Geng YANG. Dictionary partition vector space model for ciphertext ranked search in cloud environment [J]. Journal of Computer Applications, 2023, 43(7): 1994-2000.
[2]	Kun YOU, Qinhui WANG, Xin LI. General multi-unit false-name-proof auction mechanism for cloud computing [J]. Journal of Computer Applications, 2023, 43(11): 3351-3357.
[3]	Jingyu SUN, Jiayu ZHU, Ziqiang TIAN, Guozhen SHI, Chuanjiang GUAN. Attribute based encryption scheme based on elliptic curve cryptography and supporting revocation [J]. Journal of Computer Applications, 2022, 42(7): 2094-2103.
[4]	Hanlun LI, Jianguo REN. Malware propagation model based on characteristic behavior detection in P2P networks [J]. Journal of Computer Applications, 2022, 42(7): 2125-2131.
[5]	Jinquan ZHANG, Shouwei XU, Xincheng LI, Chongyang WANG, Jingzhi XU. Cloud computing task scheduling based on orthogonal adaptive whale optimization [J]. Journal of Computer Applications, 2022, 42(5): 1516-1523.
[6]	Qi GAO, Na LYU, Jingcheng MIAO. Wireless virtual network embedding algorithm based on load balance [J]. Journal of Computer Applications, 2022, 42(10): 3148-3153.
[7]	CHEN Jiahao, YIN Xinchun. Traceable and revocable ciphertext-policy attribute-based encryption scheme based on cloud-fog computing [J]. Journal of Computer Applications, 2021, 41(6): 1611-1620.
[8]	GE Lina, HU Yugu, ZHANG Guifen, CHEN Yuanyuan. Reverse hybrid access control scheme based on object attribute matching in cloud computing environment [J]. Journal of Computer Applications, 2021, 41(6): 1604-1610.
[9]	YANG Ling, JIANG Chunmao. Strategy of energy-aware virtual machine migration based on three-way decision [J]. Journal of Computer Applications, 2021, 41(4): 990-998.
[10]	Xiaoling SUN, Guang YANG, Yanping SHEN, Qiuge YANG, Tao CHEN. Searchable encryption scheme based on splittable inverted index [J]. Journal of Computer Applications, 2021, 41(11): 3288-3294.
[11]	LYU Jiayu, ZHU Zhirong, YAO Zhiqiang. Two-channel dynamic data encryption strategy in cloud computing environment [J]. Journal of Computer Applications, 2020, 40(8): 2268-2273.
[12]	GUO Shujie, LI Zhihua, LIN Kaiqing. Fuzzy membership degree based virtual machine placement algorithmin cloud environment [J]. Journal of Computer Applications, 2020, 40(5): 1374-1381.
[13]	CHEN Chengjun, MAO Yingchi, WANG Yichao. CNN model compression based on activation-entropy based layer-wise iterative pruning strategy [J]. Journal of Computer Applications, 2020, 40(5): 1260-1265.
[14]	XU Yingxin, SUN Lei, ZHAO Jiancheng, GUO Songhui. Virtual field programmable gate array placement strategy based on ant colony optimization algorithm [J]. Journal of Computer Applications, 2020, 40(3): 747-752.
[15]	XIANG Xiong, TIAN Jian. P2P transmission scheduling optimization based on software defined network [J]. Journal of Computer Applications, 2020, 40(3): 777-782.

Download performance optimization in Hadoop distributed file system based on P2P

基于P2P的分布式文件系统下载效率优化

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics