基于Kademlia的负载平衡云存储算法

doi:10.11772/j.issn.1001-9081.2015.03.643

计算机应用 ›› 2015, Vol. 35 ›› Issue (3): 643-647.DOI: 10.11772/j.issn.1001-9081.2015.03.643

基于Kademlia的负载平衡云存储算法

郑凯, 朱林, 陈优广

华东师范大学信息科学技术学院, 上海 200062

收稿日期:2014-10-11 修回日期:2014-11-06 出版日期:2015-03-10 发布日期:2015-03-13
通讯作者: 郑凯
作者简介:郑凯(1968-),男,浙江宁波人,副教授,博士,主要研究方向:计算机网络、云计算;朱林(1990-),男,山东聊城人,硕士研究生,主要研究方向:计算机网络、云计算;陈优广(1971-),男,山东临沂人,高级工程师,博士,主要研究方向:图像处理、人工智能
基金资助:
国家863计划项目(2013AA01A211)

Load balancing cloud storage algorithm based on Kademlia

ZHENG Kai, ZHU Lin, CHEN Youguang

School of Information Science Technology, East China Normal University, Shanghai 200062, China

Received:2014-10-11 Revised:2014-11-06 Online:2015-03-10 Published:2015-03-13

摘要/Abstract

摘要：

针对采用主从式结构的主流云存储系统可能出现的性能瓶颈和可扩展问题,基于分布式哈希表(DHT)技术的完全分布式云存储系统成为一种新的选择。解决好节点的负载平衡问题,是此类技术获得推广的关键。研究了Kademlia算法应用于云存储系统的负载平衡性能。考虑到算法在异构环境下负载平衡性能有明显下降,改进算法在Kademlia找出的候选存储节点中根据节点的存储能力来分配负载。仿真结果表明,改进后算法的负载平衡性能有非常明显的提高,在系统模拟运行时间足够长(如1500 h以上)时,过载节点平均下降7.0%(轻载)和33.7%(重载);文件保存成功率平均提高27.2%(轻载)和35.1%(重载),而增加的通信开销可接受。

关键词: 云存储, 负载平衡, 分布式文件系统, Kademlia, peersim

Abstract:

Prevailing cloud storage systems normally use master/slave structure, which may cause performance bottlenecks and scalability problems in some extreme cases. So, fully distributed cloud storage system based on Distributed Hash Table (DHT) technology is becoming a new choice. How to solve load balancing problem for nodes, is the key for this technology to be applicable. The Kademlia algorithm was used to locate storage target in cloud storage system and its load balancing performance was investigated. Considering the load balancing performance of the algorithm significantly decreased in heterogeneous environment, an improved algorithm was proposed, which considered heterogeneous nodes and their storage capacities and distributed loads according to the storage capacity of each node. The simulation results show that the proposed algorithm can effectively improve load balance performance of the system. Compared with the original algorithm, after running a long period (more than 1500 hours in simulation), the number of overloaded nodes in system dropped at an average percentage 7.0%(light load) to 33.7%(heavy load), file saving success rate increased at an average percentage 27.2%(light load) to 35.1%(heavy load), and also its communication overhead is acceptable.

Key words: cloud storage, load balancing, distributed file system, Kademlia, peersim

中图分类号:

郑凯, 朱林, 陈优广. 基于Kademlia的负载平衡云存储算法[J]. 计算机应用, 2015, 35(3): 643-647.

ZHENG Kai, ZHU Lin, CHEN Youguang. Load balancing cloud storage algorithm based on Kademlia[J]. Journal of Computer Applications, 2015, 35(3): 643-647.

参考文献

[1] GHEMAWAT S, GOBIOFF H, LEUNG S-T. The Google file system[C]//Proceedings of the 19th ACM Symposium on Operating Systems Principles. New York: ACM, 2003:29-43.
[2] Hadoop community. Hadoop disributed file system[EB/OL].[2014-07-11]. http://hadoop.apache.org/.
[3] ZHANG C, YIN J. Dynamic load balancing algorithm of distributed file system[J]. Journal of Chinese Computer Systems, 2011,32(7):1424-1426.(张聪萍,尹建伟.分布式文件系统的动态负载均衡算法[J].小型微型计算机系统,2011,32(7):1424-1426.)
[4] LIU K, NIU W. An improved data balancing algorithm for Hadoop[J]. Journal of Hennan Polytechnic University: Natural Science, 2013,32(3):332-336.(刘琨,钮文良.一种改进的Hadoop数据负载均衡算法[J].河南理工大学学报:自然科学版,2013,32(3):332-336.)
[5] STOICA I, MORRIS R, KARGER D, et al. Chord: a scalable peer-to-peer lookup service for Internet applications[C]//Proceedings of the International Conference of the Special Interest Group on Data Communication. New York: ACM, 2001:149-160.
[6] ROWSTRON A, DRUSCHEL P. Pastry: scalable, decentralized object location and routing for large-scale peer-to-peer systems[C]//Proceedings of IFIP/ACM International Conference on Distributed Systems Platforms. New York: ACM, 2001:329-350.
[7] HILDRUM K, KUBIATOWICZ J D, RAO S, et al. Distributed object location in a dynamic network[J]. Theory of Computer Systems, 2002,37(3):405-440.
[8] MAYMOUNKOV P, MAZTERES D. Kademlia: a peer-to-peer information system based on the XOR metric[C]//Proceedings of the 1st International Workshop on Peer-to-Peer Systems. Berlin: Springer, 2002:153-161.
[9] HUANG Q, CHENG Y, CHEN G. Research on hash algorithm for distributed storage system[J]. Computer Engineering and Applications, 2014,50(1):1-4.(黄秋兰,程耀东,陈刚. 分布式存储系统的哈希算法研究[J].计算机工程与应用,2014,50(1):1-4.)
[10] WU J, FU J, PING L, et al. Study on the P2P cloud storage system[J]. Acta Electronica Sinica, 2011,39(5):1100-1107.(吴吉义,傅建庆,平玲娣,等.一种对等结构的云存储系统研究[J].电子学报,2011,39(5):1100-1107.)
[11] YIN X, YANG J, QU C. Research on load balancing of distributed file system in cloud computing[J]. Computer Science, 2014,41(3):141-144.(尹向东,杨杰,屈长青.云计算环境下分布式文件系统的负载平衡研究[J].计算机科学,2014,41(3):141-144.)
[12] JESI G P. Peersim howto: build a new protocol for the peersim 1.0 simulator[EB/OL].[2011-08-04]. http://peersim.sourceforge.net/tutorial1/tutorial1.pdf.
[13] BROBERG J, BUYYA R, TARI Z. Creating a ‘cloud storage' mashup for high performance, low cost content delivery[C]//Proceedings of the 6th International Conference on Service-Oriented Computing, LNCS 5472. Berlin: Springer, 2009:178-183.

[1]	赵全, 汤小春, 朱紫钰, 毛安琪, 李战怀. 大规模短时间任务的低延迟集群调度框架[J]. 计算机应用, 2021, 41(8): 2396-2405.
[2]	李秀艳, 刘明曦, 史闻博, 董国芳. 面向资源受限用户的高效动态数据审计方案[J]. 计算机应用, 2021, 41(2): 422-432.
[3]	苟子安, 张晓, 吴东南, 王艳秋. 分布式存储系统中的日志分析与负载特征提取[J]. 计算机应用, 2020, 40(9): 2586-2593.
[4]	唐鑫, 周琳娜. 基于响应模糊化的抗附加块攻击云数据安全去重方法[J]. 计算机应用, 2020, 40(4): 1085-1090.
[5]	付伟, 顾晨阳, 高强. 基于属性加密的多用户共享ORAM方案[J]. 计算机应用, 2020, 40(2): 497-502.
[6]	董聪, 张晓, 程文迪, 石佳. 基于新型存储器件的分布式文件系统性能优化[J]. 计算机应用, 2020, 40(12): 3594-3603.
[7]	王海勇, 彭垚, 郭凯璇. 云存储中基于代理重加密的CP-ABE访问控制方案[J]. 计算机应用, 2019, 39(9): 2611-2616.
[8]	柳玉东, 王绪安, 涂广升, 王涵. 全生命周期的云外包数据安全审计协议[J]. 计算机应用, 2019, 39(7): 1954-1958.
[9]	明洋, 何宝康. 支持属性撤销的可验证外包的多授权属性加密方案[J]. 计算机应用, 2019, 39(12): 3556-3562.
[10]	周坚, 金瑜, 何亨, 李鹏. 基于嵌套Merkle Hash tree区块链的云数据动态审计模型[J]. 计算机应用, 2019, 39(12): 3575-3583.
[11]	李静, 刘冬实. 主动容错云存储系统的可靠性评价模型[J]. 计算机应用, 2018, 38(9): 2631-2636.
[12]	陈博, 何连跃, 严巍巍, 徐照淼, 徐俊. 海量小文件系统的可移植操作系统接口兼容技术[J]. 计算机应用, 2018, 38(5): 1389-1392.
[13]	付朝江, 王天奇, 林悦荣. 基于有效并行求解策略的显式有限元分析并行算法[J]. 计算机应用, 2018, 38(4): 1072-1077.
[14]	杨宏宇, 王玥. 云存储环境下的多关键字密文搜索方法[J]. 计算机应用, 2018, 38(2): 343-347.
[15]	邰滢滢, 庞影, 段苛苛, 付云鹏. 基于改进权重的D-S证据理论的动态负载平衡算法[J]. 计算机应用, 2018, 38(10): 2976-2981.

基于Kademlia的负载平衡云存储算法

Load balancing cloud storage algorithm based on Kademlia

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics