分布式文件系统数据块聚类存储节能策略

doi:10.11772/j.issn.1001-9081.2015.02.0378

计算机应用 ›› 2015, Vol. 35 ›› Issue (2): 378-382.DOI: 10.11772/j.issn.1001-9081.2015.02.0378

分布式文件系统数据块聚类存储节能策略

王政英¹, 于炯^1,2, 英昌甜¹, 鲁亮¹

1. 新疆大学信息科学与工程学院, 乌鲁木齐 830046;
2. 新疆大学软件学院, 乌鲁木齐 830008

收稿日期:2014-09-15 修回日期:2014-11-19 出版日期:2015-02-10 发布日期:2015-02-12
通讯作者: 王政英
作者简介:王政英(1984-),女,山西晋城人,硕士研究生,CCF会员,主要研究方向:云计算、绿色计算、数据挖掘; 于炯(1964-),男,北京人,博士,教授,博士生导师,CCF高级会员,主要研究方向:网络安全、网格与分布式计算; 英昌甜(1989-),女,新疆乌鲁木齐人,博士研究生,主要研究方向:云计算、分布式存储; 鲁亮(1990-),男,湖南湘潭人,博士研究生,CCF会员,主要研究方向:分布式计算、云计算。
基金资助:
国家自然科学基金资助项目(61462079,61262088,61363083);新疆维吾尔族自治区自然科学基金资助项目(2013211A011)。

Energy-efficient strategy of distributed file system based on data block clustering storage

WANG Zhengying¹, YU Jiong^1,2, YING Changtian¹, LU Liang¹

1. School of Information Science and Engineering, Xinjiang University, Urumqi Xinjiang 830046, China;
2. School of Software, Xinjiang University, Urumqi Xinjiang 830008, China

Received:2014-09-15 Revised:2014-11-19 Online:2015-02-10 Published:2015-02-12

摘要/Abstract

摘要：

针对分布式文件系统中由于数据块随机放置而导致的服务器利用率低、能耗管理复杂的问题,建立了数据块访问特征向量模型描述用户对数据块的随机访问,运用K-means算法对数据块进行聚类计算,根据计算结果将数据节点划分为多个区域以存储不同聚类簇的数据块,在系统负载较低时进行数据块动态重配置,关闭不必要节点达到节能的目的。为使得策略适用于对能耗和资源利用率有不同要求的场景,算法中聚类簇间隔参数可灵活设置。实验通过和冷热区划分算法进行比较表明:按照聚类结果进行数据块重配置后,能耗节省效率优于冷热区划分算法,节省能耗35%~38%。

关键词: 云计算, 分布式文件系统, 数据聚类, 动态重配置, 节能计算

Abstract:

Concerning the low server utilization and complicated energy management caused by block random placement strategy in distributed file systems, the vector of the visiting feature on data block was built to depict the behavior of the random block accessing. K-means algorithm was adopted to do the clustering calculation according to the calculation result, then the datanodes were divided into multiple regions to store different cluster data blocks. The data blocks were dynamic reconfigured according to the clustering calculation results when the system load is low. The unnecessary datanodes could sleep to reduce the energy consumption. The flexible set of distance parameters between clusters made the strategy be suitable for different scenarios that has different requests for the energy consumption and utilization. Compared with hot-cold zoning strategies, the mathematical analysis and experimental results prove that the proposed method has a higher energy saving efficiency, the energy consumption reduces by 35% to 38%.

Key words: cloud computing, distributed file system, data clustering, dynamic reconfiguration, energy-efficient computing

中图分类号:

TP393.09

王政英, 于炯, 英昌甜, 鲁亮. 分布式文件系统数据块聚类存储节能策略[J]. 计算机应用, 2015, 35(2): 378-382.

WANG Zhengying, YU Jiong, YING Changtian, LU Liang. Energy-efficient strategy of distributed file system based on data block clustering storage[J]. Journal of Computer Applications, 2015, 35(2): 378-382.

参考文献

[1] VOUK M A. Cloud computing — issues, research and implementations [J]. Journal of Computing and Information Technology, 2004, 16(4):235-246.
[2] CHEMAWAT S,GOBIFF H, LEUNG S-T. The Goole file system [C]//SOSP'03: Proceeding of 19th ACM Symposium on Operating System Principles. New York: ACM, 2003: 29-43.
[3] BORTHAKUR D. The Hadoop distributed file system: architecture and design [EB/OL]. (2007-07-01) [2014-07-14]. http://hadoop.apache.org/common/docs/r0.18.2/hdfs_design.pdf.
[4] HOOPER A. Green computing [J]. Communications of the ACM, 2008, 51(10): 11-13.
[5] Global Action Plan. An inefficient truth [R/OL]. [2014-04-20]. http://www.it-energy.co.uk/pdf/GAP%20An%20Inefficient%20Truth%20Dec%202007.pdf.
[6] LEVERICH J, KOZYRAKIS C. On the energy in efficiency of Hadoop clusters [J]. ACM SIGOPS Operating Systems Review, 2010,44(1):61-65.
[7] WANG Z, YU J, YING C, et al. Energy-efficient strategy for dynamic management of cloud storage replica based on user visiting characteristic [J]. Journal of Computer Applications, 2014, 34(8): 2256-2259. (王政英,于炯,英昌甜,等. 基于用户访问特征的云存储副本动态管理节能策略[J]. 计算机应用,2014,34(8): 2256-2259.)
[8] MAHESHWARI N, NANDURI R, VARMA V. Dynamic energy efficient data placement and cluster reconfiguration algorithm for MapReduce framework [J]. Future Generation Computer Systems, 2011, 28(1): 119-127.
[9] LIU Q, Todman T, LUK W. Combining optimizations in automated low power design [C]//DATE '10: Proceedings of the Conference on Design, Automation and Test in Europe. Leuven, Belgium: European Design and Automation Association, 2010: 1791-1796.
[10] LIN B, LI S, LIAO X, et al. Seadown: SLA-aware size-scaling power management in heterogeneous MapReduce cluster [J]. Chinese Journal of Computers, 2013, 36(5): 977-987. (林彬,李姗姗,廖湘科,等.Seadown:一种异构MapReduce集群中面向SLA的能耗管理方法[J]. 计算机学报,2013,36(5):977-987.)
[11] HARNIK D, NAOR D, SEGALL I. Low power mode in cloud storage systems [C]//IPDPS 2009: Proceedings of the IEEE International Symposium on Paraller & Distributed Processing. Piscataway: IEEE, 2009: 1-8.
[12] VASIC N, BARISITS M, SALZGEBER V, et al. Making cluster applications energy-aware [C]//ACDC '09: Proceedings of the 1st Workshop on Automated Control for Datacenters and Clouds. New York: ACM, 2009: 37-42.
[13] KAUSHIK R T, BHANDARKAR M, NAHRSTEDT K. Evaluation and analysis of GreenHDFS: a self-adaptive, energy conserving variant of the Hadoop distributed file system [C]//Proceedings of the IEEE 2nd International Conference on Cloud Computing Technology and Science. Piscataway: IEEE, 2010: 274-287.
[14] KAUSHIK R T, BHANDARKAR M. GreenHDFS: towards an energy-conserving, storage-efficient, hybrid Hadoop compute cluster [C]//HotPower'10: Proceedings of the 2010 International Conference on Power Aware Computing and Systems. Piscataway: IEEE, 2010: Article No. 1-9.
[15] LIAO B, YU J, ZHANG T, et al. Energy-efficient algorithms for distributed file system HDFS [J]. Chinese Journal of Computers, 2013,5:1-18. (廖彬,于炯,张陶,等.基于分布式文件系统HDFS的节能算法[J]. 计算机学报,2013,36(5):1047-1064.)
[16] MacQUEEN J. Some methods for classification and analysis of multivariate observations [C]//Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability. Berkeley: University of California Press, 1967: 281-297.

分布式文件系统数据块聚类存储节能策略

Energy-efficient strategy of distributed file system based on data block clustering storage

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	陈家豪, 殷新春. 基于云雾计算的可追踪可撤销密文策略属性基加密方案[J]. 计算机应用, 2021, 41(6): 1611-1620.
[2]	葛丽娜, 胡雨谷, 张桂芬, 陈园园. 云计算环境基于客体属性匹配的逆向混合访问控制方案[J]. 计算机应用, 2021, 41(6): 1604-1610.
[3]	杨翎, 姜春茂. 基于三支决策的虚拟机节能迁移策略[J]. 计算机应用, 2021, 41(4): 990-998.
[4]	孙晓玲, 杨光, 沈焱萍, 杨秋格, 陈涛. 基于可拆分倒排索引的可搜索加密方案[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3288-3294.
[5]	苟子安, 张晓, 吴东南, 王艳秋. 分布式存储系统中的日志分析与负载特征提取[J]. 计算机应用, 2020, 40(9): 2586-2593.
[6]	吕佳玉, 竺智荣, 姚志强. 云计算环境下的双通道数据动态加密策略[J]. 计算机应用, 2020, 40(8): 2268-2273.
[7]	陈程军, 毛莺池, 王绎超. 基于激活-熵的分层迭代剪枝策略的CNN模型压缩[J]. 计算机应用, 2020, 40(5): 1260-1265.
[8]	郭曙杰, 李志华, 蔺凯青. 云环境下基于模糊隶属度的虚拟机放置算法[J]. 计算机应用, 2020, 40(5): 1374-1381.
[9]	许英鑫, 孙磊, 赵建成, 郭松辉. 基于蚁群优化算法的虚拟现场可编程门阵列部署策略[J]. 计算机应用, 2020, 40(3): 747-752.
[10]	刘福鑫, 李劲巍, 王熠弘, 李琳. 基于Kubernetes的云原生海量数据存储系统设计与实现[J]. 计算机应用, 2020, 40(2): 547-552.
[11]	王庆永, 毛莺池, 王绎超, 王龙宝. 基于多微云协作的计算任务卸载[J]. 计算机应用, 2020, 40(2): 328-334.
[12]	董聪, 张晓, 程文迪, 石佳. 基于新型存储器件的分布式文件系统性能优化[J]. 计算机应用, 2020, 40(12): 3594-3603.
[13]	杨哂哂, 吴慧珍, 庄黎丽, 吕宏武. 基于Markov过程的IaaS系统可用性建模与分析方法[J]. 计算机应用, 2020, 40(10): 3013-3018.
[14]	王巧玲, 乔非, 蒋友好. 基于聚合距离参数的改进K-means算法[J]. 计算机应用, 2019, 39(9): 2586-2590.
[15]	林立, 熊金波, 肖如良, 林铭炜, 陈秀华. Gaming@Edge:基于边缘节点的低延迟云游戏系统[J]. 计算机应用, 2019, 39(7): 2001-2007.