面向大规模数据接入系统的负载平衡机制

doi:10.11772/j.issn.1001-9081.2017071836

计算机应用 ›› 2018, Vol. 38 ›› Issue (1): 50-55.DOI: 10.11772/j.issn.1001-9081.2017071836

• 2017年全国开放式分布与并行计算学术年会(DPCS 2017)论文 • 上一篇下一篇

面向大规模数据接入系统的负载平衡机制

周岳, 陈庆奎

上海理工大学光电信息与计算机工程学院, 上海 200093

收稿日期:2017-07-25 修回日期:2017-08-04 出版日期:2018-01-10 发布日期:2018-01-22
通讯作者: 陈庆奎
作者简介:周岳(1991-),男,江苏宿迁人,硕士研究生,主要研究方向:分布式计算、物联网;陈庆奎(1966-),男,黑龙江哈尔滨人,教授,博士生导师,博士,CCF会员,主要研究方向:网络计算、并行计算、物联网。
基金资助:
国家自然科学基金资助项目（60970012，61572325）；上海重点科技攻关项目（14511107902，16DZ1203603）；上海市工程技术研究中心建设项目（GCZX14014）；上海市一流学科建设项目（XTKX2012）。

Load balancing mechanism for large-scale data access system

ZHOU Yue, CHEN Qingkui

School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, China

Received:2017-07-25 Revised:2017-08-04 Online:2018-01-10 Published:2018-01-22
Supported by:
This work is partially supported by the National Natural Science Foundation of China (60970012, 61572325), Shanghai Key Science and Technology Project (14511107902, 16DZ1203603), Shanghai Engineering Research Center Construction Project (GCZX14014), Shanghai Leading Academic Discipline Project (XTKX2012).

摘要/Abstract

摘要： 当前分布式系统负载平衡算法存在问题：1）算法建立的系统中各节点角色固定，系统不具有自适应性；2）算法的通用性不高；3）负载迁移任务巨大，且负载平衡周期过长等。针对这些问题，提出了混合式负载平衡算法。首先，设计了一个分布式系统接收模型。模型将系统任务分为三层：接收层、处理层和存储层。在接收层使用了自定义的通信协议提高系统的接收性能。然后，负载平衡算法采用随机负载迁移策略，根据系统中节点的负载状态，对负载任务进行随机迁移。通过这种策略解决负载平衡周期过长和负载回迁问题。最后，通过分布式控制节点选择策略，使系统中节点具有自适应性。实验结果显示，在百万数据源以下，系统各层平均延迟处于毫秒级，系统负载平衡平均耗时在3 min以下。实验证明了所提出的负载平衡机制具有周期短、任务响应迅速等特点，能够提高分布式系统的接收性能。

关键词: 大规模数据接入系统, 分布式系统, 混合式负载平衡, 随机负载迁移, 数据接入系统

Abstract: Some problems of the current load balancing algorithms for distributed systems include:1) The role of each node in the system is fixed, and the system has no adaptability. 2) The load balancing algorithm is not universal. 3) The migration task is too large, and the load balance cycle is too long. To solve these problems, a hybrid load balancing algorithm was proposed. Firstly, a distributed receiving system model was designed, by which the system tasks were divided into three parts:receiving level, handling level and storing level. In receiving level, a home-made transmission protocol was used to improve the reception capability of the system. And then, in the load balancing algorithm, random load migration strategy was used. According to the status of the nodes, the tasks of load were randomly migrated. The problems of long load balance cycle and load moving back were solved by this strategy. Finally, the distributed control node selecting strategy was adopted to make the nodes adaptable. The experimental results show that the average delay in each layer of the system is in milliseconds, and the system load balancing takes less than 3 minutes, which proves that the load balancing mechanism has short load balance cycle and fast response, and can improve the reception capability of the distributed system.

Key words: large-scale data receiving system, distributed system, hybrid load balancing, random load migration, data access system

中图分类号:

TP302.1

周岳, 陈庆奎. 面向大规模数据接入系统的负载平衡机制[J]. 计算机应用, 2018, 38(1): 50-55.

ZHOU Yue, CHEN Qingkui. Load balancing mechanism for large-scale data access system[J]. Journal of Computer Applications, 2018, 38(1): 50-55.

参考文献

[1] 廖锋,成静静.大数据环境下Hadoop分布式系统的研究与设计[J].广东通信技术,2013(10):22-27.(LIAO F, CHENG J J. Research and design of Hadoop distributed system under big data environment[J]. Guangdong Communication Technology, 2013(10):22-27.)
[2] HASHEM I A T, YAQOOB I, ANUAR N B, et al. The rise of "big data" on cloud computing:review and open research issues[J]. Information Systems, 2015, 47:98-115.
[3] 房俊华,王晓桐,张蓉,等.分布式数据流上的高性能分发策略[J].软件学报,2017,28(3):563-578.(FANG J H, WANG X T, ZHANG R, et al. A high performance distribution policy on distributed data streams[J]. Journal of Software, 2017, 28(3):563-578.)
[4] JIANG Y, ZHOU Y, WANG W. Task allocation for undependable multiagent systems in social networks[J]. IEEE Transactions on Parallel and Distributed Systems, 2013, 24(8):1671-1681.
[5] PENMATSA S, CHRONOPOULOS A T. Game-theoretic static load balancing for distributed systems[J]. Journal of Parallel and Distributed Computing, 2011, 71(4):537-555.
[6] TONG Z, XIAO Z, LI K, et al. Proactive scheduling in distributed computing-A reinforcement learning approach[J]. Journal of Parallel and Distributed Computing, 2014, 74(7):2662-2672.
[7] JIANG Y. A survey of task allocation and load balancing in distributed systems[J]. IEEE Transactions on Parallel & Distributed Systems, 2016, 27(2):585-599.
[8] VAN DER HORST J, NOBLE J. Distributed and centralized task allocation:when and where to use them[C]//SASOW 2010:Proceedings of the 2010 Fourth IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshop. Piscataway, NJ:IEEE, 2010:1-8.
[9] IZAKIAN H, ABRAHAM A, LADANI B T. An auction method for resource allocation in computational grids[J]. Future Generation Computer Systems, 2010, 26(2):228-235.
[10] WANG W, JIANG Y. Community-aware task allocation for social networked multiagent systems[J]. IEEE Transactions on Cybernetics, 2014, 44(9):1529-1543.
[11] DI S, WANG C L. Decentralized proactive resource allocation for maximizing throughput of P2P grid[J]. Journal of Parallel and Distributed Computing, 2012, 72(2):308-321.
[12] GARG S K, VENUGOPAL S, BROBERG J, et al. Double auction-inspired meta-scheduling of parallel applications on global grids[J]. Journal of Parallel and Distributed Computing, 2013, 73(4):450-464.
[13] DUONG-BA T, NGUYEN T, BOSE B, et al. Distributed client-server assignment for online social network applications[J]. IEEE Transactions on Emerging Topics in Computing, 2014, 2(4):422-435.
[14] FARAGARDI H R, SHOJAEE R, KESHTKAR M A, et al. Optimal task allocation for maximizing reliability in distributed real-time systems[C]//ICIS 2013:Proceedings of the 2013 IEEE/ACIS 12th International Conference on Computer and Information Science. Piscataway, NJ:IEEE, 2013:513-519.
[15] DUSSO P M, SAUER C, HÄRDER T. Optimizing sort in Hadoop using replacement selection[C]//Proceedings of the 2015 East European Conference on Advances in Databases and Information Systems. Berlin:Springer, 2015:365-379.
[16] KOTA R, GIBBINS N, JENNINGS N R. Decentralized approaches for self-adaptation in Agent organizations[J]. ACM Transactions on Autonomous and Adaptive Systems, 2012, 7(1):1-28.
[17] AN B, LESSER V, SIM K M. Strategic Agents for multi-resource negotiation[J]. Autonomous Agents and Multi-Agent Systems, 2011, 23(1):114-153.
[18] CARPIO F, ENGELMANN A, JUKAN A. DiffFlow:differentiating short and long flows for load balancing in data center networks[C]//GLOBECOM 2016:Proceedings of the 2016 IEEE Global Communications Conference. Piscataway, NJ:IEEE, 2016:1-6.
[19] JIANG Y, HUANG Z. The rich get richer:Preferential attachment in the task allocation of cooperative networked multiagent systems with resource caching[J]. IEEE Transactions on Systems, Man, and Cybernetics-Part A:Systems and Humans, 2012, 42(5):1040-1052.
[20] EUGSTER P, KOGAN K, NIKOLENKO S I, et al. Heterogeneous packet processing in shared memory buffers[J]. Journal of Parallel & Distributed Computing, 2016, 99(8):1-13.
[21] WANG T, SU Z, XIA Y, et al. Rethinking the data center networking:architecture, network protocols, and resource sharing[J]. IEEE Access, 2014, 2(7):1481-1496.

面向大规模数据接入系统的负载平衡机制

Load balancing mechanism for large-scale data access system

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	吴志军, 王航. 基于订阅/发布服务的广域信息管理系统应急响应机制[J]. 计算机应用, 2020, 40(5): 1340-1347.
[2]	肖圣龙, 陈昕, 李卓. 面向社会安全事件的分布式神经网络攻击行为分类方法[J]. 计算机应用, 2017, 37(10): 2794-2798.
[3]	邱桂, 闫仁武. 基于灰色关联分析的分布式协同过滤推荐算法[J]. 计算机应用, 2016, 36(4): 1054-1059.
[4]	魏敏, 刘以安, 吴鸿雁. 分布式系统中局部处理机的设计与实现[J]. 计算机应用, 2015, 35(5): 1290-1295.
[5]	李勇, 吴立慧, 黄宁, 吴维刚. 基于HBase的地理分布副本管理机制[J]. 计算机应用, 2015, 35(11): 3097-3101.
[6]	朱强孙玉强. 高通信效率的分布式流数据聚类算法[J]. 计算机应用, 2014, 34(9): 2505-2509.
[7]	胡晓辉肖知屹陈永李欣. 列车安全距离控制形式化建模与验证[J]. 计算机应用, 2014, 34(3): 851-856.
[8]	侯金奎王磊. 基于Agent的分布式模型形式化语义描述[J]. 计算机应用, 2013, 33(12): 3423-3427.
[9]	郭栋王伟曾国荪. 基于一致性树分布的数据分布式存储方法[J]. 计算机应用, 2013, 33(12): 3432-3436.
[10]	范敏徐胜才. 基于Hadoop的海量医学图像检索系统[J]. 计算机应用, 2013, 33(12): 3345-3349.
[11]	吴晋张国良汤文俊孙一杰. 基于改进人工协调场的多机器人避碰算法[J]. 计算机应用, 2013, 33(11): 3123-3128.
[12]	陈雅琳黄宏光李燕斌. CORBA分布式系统中网络分割协议可行度分析[J]. 计算机应用, 2013, 33(08): 2124-2127.
[13]	邵婧陈性元杜学绘曹利峰. 基于无干扰理论的分布式多级安全核心架构[J]. 计算机应用, 2013, 33(03): 712-716.
[14]	耿技陈非聂鹏陈伟秦志光. 可靠的分布式系统生存性保障模型[J]. 计算机应用, 2012, 32(10): 2748-2751.
[15]	石磊侯垚森. 基于消息延迟预测的自适应失效检测模型[J]. 计算机应用, 2010, 30(05): 1312-1315.