Journal of Computer Applications ›› 2011, Vol. 31 ›› Issue (09): 2317-2320.DOI: 10.3724/SP.J.1087.2011.02317
• Network and communications • Previous Articles Next Articles
LIAO Bin,YU Jiong,ZHANG Tao,YANG Xing-yao
Received:
Revised:
Online:
Published:
Contact:
廖彬,于炯,张陶,杨兴耀
通讯作者:
作者简介:
基金资助:
Abstract: The data block storage mechanism and downloading process in Hadoop Distributed File System (HDFS) cluster were analyzed. In combination with multi-point and multi-threaded Peer-to-Peer (P2P) download idea, an efficiency optimization algorithm was proposed from the aspects of data-block, file and cluster. Concerning the possible imbalanced load problem caused by multi-thread download in HDFS cluster, a download-point selection algorithm was put forward to optimize the download-point selection. The mathematical analysis and experiments prove that the three methods can improve the download efficiency and download-point selection algorithm can achieve loading balance among DataNodes in HDFS cluster.
Key words: cloud computing, Hadoop Distributed File System (HDFS), Peer-to-Peer (P2P), parallel download, load balance
摘要: 对分布式文件系统(HDFS)集群内部数据块存储机制与下载流程进行分析研究,结合P2P多点与多线程下载思想,从数据块、文件、集群三个方面提出了数据下载效率优化算法。考虑到集群内部可能因多线程下载出现的负载均衡问题,提出下载点选择算法以优化下载点的选择。实验结果表明,三种优化算法都能提高下载效率,下载点选择算法能够很好地实现集群内部DataNode负载均衡。
关键词: 云计算, 分布式文件系统, 对等网, 并行下载, 负载均衡
CLC Number:
TP393.027
LIAO Bin YU Jiong ZHANG Tao YANG Xing-yao. Download performance optimization in Hadoop distributed file system based on P2P[J]. Journal of Computer Applications, 2011, 31(09): 2317-2320.
廖彬 于炯 张陶 杨兴耀. 基于P2P的分布式文件系统下载效率优化[J]. 计算机应用, 2011, 31(09): 2317-2320.
0 / Recommend
Add to citation manager EndNote|Ris|BibTeX
URL: https://www.joca.cn/EN/10.3724/SP.J.1087.2011.02317
https://www.joca.cn/EN/Y2011/V31/I09/2317