Energy-efficient strategy of distributed file system based on data block clustering storage

doi:10.11772/j.issn.1001-9081.2015.02.0378

Journal of Computer Applications ›› 2015, Vol. 35 ›› Issue (2): 378-382.DOI: 10.11772/j.issn.1001-9081.2015.02.0378

Previous Articles Next Articles

Energy-efficient strategy of distributed file system based on data block clustering storage

WANG Zhengying¹, YU Jiong^1,2, YING Changtian¹, LU Liang¹

1. School of Information Science and Engineering, Xinjiang University, Urumqi Xinjiang 830046, China;
2. School of Software, Xinjiang University, Urumqi Xinjiang 830008, China

Received:2014-09-15 Revised:2014-11-19 Online:2015-02-10 Published:2015-02-12

分布式文件系统数据块聚类存储节能策略

王政英¹, 于炯^1,2, 英昌甜¹, 鲁亮¹

1. 新疆大学信息科学与工程学院, 乌鲁木齐 830046;
2. 新疆大学软件学院, 乌鲁木齐 830008

通讯作者: 王政英
作者简介:王政英(1984-),女,山西晋城人,硕士研究生,CCF会员,主要研究方向:云计算、绿色计算、数据挖掘; 于炯(1964-),男,北京人,博士,教授,博士生导师,CCF高级会员,主要研究方向:网络安全、网格与分布式计算; 英昌甜(1989-),女,新疆乌鲁木齐人,博士研究生,主要研究方向:云计算、分布式存储; 鲁亮(1990-),男,湖南湘潭人,博士研究生,CCF会员,主要研究方向:分布式计算、云计算。
基金资助:
国家自然科学基金资助项目(61462079,61262088,61363083);新疆维吾尔族自治区自然科学基金资助项目(2013211A011)。

Abstract

Abstract:

Concerning the low server utilization and complicated energy management caused by block random placement strategy in distributed file systems, the vector of the visiting feature on data block was built to depict the behavior of the random block accessing. K-means algorithm was adopted to do the clustering calculation according to the calculation result, then the datanodes were divided into multiple regions to store different cluster data blocks. The data blocks were dynamic reconfigured according to the clustering calculation results when the system load is low. The unnecessary datanodes could sleep to reduce the energy consumption. The flexible set of distance parameters between clusters made the strategy be suitable for different scenarios that has different requests for the energy consumption and utilization. Compared with hot-cold zoning strategies, the mathematical analysis and experimental results prove that the proposed method has a higher energy saving efficiency, the energy consumption reduces by 35% to 38%.

Key words: cloud computing, distributed file system, data clustering, dynamic reconfiguration, energy-efficient computing

摘要：

针对分布式文件系统中由于数据块随机放置而导致的服务器利用率低、能耗管理复杂的问题,建立了数据块访问特征向量模型描述用户对数据块的随机访问,运用K-means算法对数据块进行聚类计算,根据计算结果将数据节点划分为多个区域以存储不同聚类簇的数据块,在系统负载较低时进行数据块动态重配置,关闭不必要节点达到节能的目的。为使得策略适用于对能耗和资源利用率有不同要求的场景,算法中聚类簇间隔参数可灵活设置。实验通过和冷热区划分算法进行比较表明:按照聚类结果进行数据块重配置后,能耗节省效率优于冷热区划分算法,节省能耗35%~38%。

关键词: 云计算, 分布式文件系统, 数据聚类, 动态重配置, 节能计算

CLC Number:

TP393.09

WANG Zhengying, YU Jiong, YING Changtian, LU Liang. Energy-efficient strategy of distributed file system based on data block clustering storage[J]. Journal of Computer Applications, 2015, 35(2): 378-382.

王政英, 于炯, 英昌甜, 鲁亮. 分布式文件系统数据块聚类存储节能策略[J]. 计算机应用, 2015, 35(2): 378-382.

References

[1] VOUK M A. Cloud computing — issues, research and implementations [J]. Journal of Computing and Information Technology, 2004, 16(4):235-246.
[2] CHEMAWAT S,GOBIFF H, LEUNG S-T. The Goole file system [C]//SOSP'03: Proceeding of 19th ACM Symposium on Operating System Principles. New York: ACM, 2003: 29-43.
[3] BORTHAKUR D. The Hadoop distributed file system: architecture and design [EB/OL]. (2007-07-01) [2014-07-14]. http://hadoop.apache.org/common/docs/r0.18.2/hdfs_design.pdf.
[4] HOOPER A. Green computing [J]. Communications of the ACM, 2008, 51(10): 11-13.
[5] Global Action Plan. An inefficient truth [R/OL]. [2014-04-20]. http://www.it-energy.co.uk/pdf/GAP%20An%20Inefficient%20Truth%20Dec%202007.pdf.
[6] LEVERICH J, KOZYRAKIS C. On the energy in efficiency of Hadoop clusters [J]. ACM SIGOPS Operating Systems Review, 2010,44(1):61-65.
[7] WANG Z, YU J, YING C, et al. Energy-efficient strategy for dynamic management of cloud storage replica based on user visiting characteristic [J]. Journal of Computer Applications, 2014, 34(8): 2256-2259. (王政英,于炯,英昌甜,等. 基于用户访问特征的云存储副本动态管理节能策略[J]. 计算机应用,2014,34(8): 2256-2259.)
[8] MAHESHWARI N, NANDURI R, VARMA V. Dynamic energy efficient data placement and cluster reconfiguration algorithm for MapReduce framework [J]. Future Generation Computer Systems, 2011, 28(1): 119-127.
[9] LIU Q, Todman T, LUK W. Combining optimizations in automated low power design [C]//DATE '10: Proceedings of the Conference on Design, Automation and Test in Europe. Leuven, Belgium: European Design and Automation Association, 2010: 1791-1796.
[10] LIN B, LI S, LIAO X, et al. Seadown: SLA-aware size-scaling power management in heterogeneous MapReduce cluster [J]. Chinese Journal of Computers, 2013, 36(5): 977-987. (林彬,李姗姗,廖湘科,等.Seadown:一种异构MapReduce集群中面向SLA的能耗管理方法[J]. 计算机学报,2013,36(5):977-987.)
[11] HARNIK D, NAOR D, SEGALL I. Low power mode in cloud storage systems [C]//IPDPS 2009: Proceedings of the IEEE International Symposium on Paraller & Distributed Processing. Piscataway: IEEE, 2009: 1-8.
[12] VASIC N, BARISITS M, SALZGEBER V, et al. Making cluster applications energy-aware [C]//ACDC '09: Proceedings of the 1st Workshop on Automated Control for Datacenters and Clouds. New York: ACM, 2009: 37-42.
[13] KAUSHIK R T, BHANDARKAR M, NAHRSTEDT K. Evaluation and analysis of GreenHDFS: a self-adaptive, energy conserving variant of the Hadoop distributed file system [C]//Proceedings of the IEEE 2nd International Conference on Cloud Computing Technology and Science. Piscataway: IEEE, 2010: 274-287.
[14] KAUSHIK R T, BHANDARKAR M. GreenHDFS: towards an energy-conserving, storage-efficient, hybrid Hadoop compute cluster [C]//HotPower'10: Proceedings of the 2010 International Conference on Power Aware Computing and Systems. Piscataway: IEEE, 2010: Article No. 1-9.
[15] LIAO B, YU J, ZHANG T, et al. Energy-efficient algorithms for distributed file system HDFS [J]. Chinese Journal of Computers, 2013,5:1-18. (廖彬,于炯,张陶,等.基于分布式文件系统HDFS的节能算法[J]. 计算机学报,2013,36(5):1047-1064.)
[16] MacQUEEN J. Some methods for classification and analysis of multivariate observations [C]//Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability. Berkeley: University of California Press, 1967: 281-297.

Energy-efficient strategy of distributed file system based on data block clustering storage

分布式文件系统数据块聚类存储节能策略

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	CHEN Jiahao, YIN Xinchun. Traceable and revocable ciphertext-policy attribute-based encryption scheme based on cloud-fog computing [J]. Journal of Computer Applications, 2021, 41(6): 1611-1620.
[2]	GE Lina, HU Yugu, ZHANG Guifen, CHEN Yuanyuan. Reverse hybrid access control scheme based on object attribute matching in cloud computing environment [J]. Journal of Computer Applications, 2021, 41(6): 1604-1610.
[3]	YANG Ling, JIANG Chunmao. Strategy of energy-aware virtual machine migration based on three-way decision [J]. Journal of Computer Applications, 2021, 41(4): 990-998.
[4]	Xiaoling SUN, Guang YANG, Yanping SHEN, Qiuge YANG, Tao CHEN. Searchable encryption scheme based on splittable inverted index [J]. Journal of Computer Applications, 2021, 41(11): 3288-3294.
[5]	GOU Zi'an, ZHANG Xiao, WU Dongnan, WANG Yanqiu. Log analysis and workload characteristic extraction in distributed storage system [J]. Journal of Computer Applications, 2020, 40(9): 2586-2593.
[6]	LYU Jiayu, ZHU Zhirong, YAO Zhiqiang. Two-channel dynamic data encryption strategy in cloud computing environment [J]. Journal of Computer Applications, 2020, 40(8): 2268-2273.
[7]	CHEN Chengjun, MAO Yingchi, WANG Yichao. CNN model compression based on activation-entropy based layer-wise iterative pruning strategy [J]. Journal of Computer Applications, 2020, 40(5): 1260-1265.
[8]	GUO Shujie, LI Zhihua, LIN Kaiqing. Fuzzy membership degree based virtual machine placement algorithmin cloud environment [J]. Journal of Computer Applications, 2020, 40(5): 1374-1381.
[9]	XU Yingxin, SUN Lei, ZHAO Jiancheng, GUO Songhui. Virtual field programmable gate array placement strategy based on ant colony optimization algorithm [J]. Journal of Computer Applications, 2020, 40(3): 747-752.
[10]	WANG Qingyong, MAO Yingchi, WANG Yichao, WANG Longbao. Computing task offloading based on multi-cloudlet collaboration [J]. Journal of Computer Applications, 2020, 40(2): 328-334.
[11]	LIU Fuxin, LI Jingwei, WANG Yihong, LI Lin. Design and implementation of cloud native massive data storage system based on Kubernetes [J]. Journal of Computer Applications, 2020, 40(2): 547-552.
[12]	DONG Cong, ZHANG Xiao, CHENG Wendi, SHI Jia. Performance optimization of distributed file system based on new type storage devices [J]. Journal of Computer Applications, 2020, 40(12): 3594-3603.
[13]	YANG Shenshen, WU Huizhen, ZHUANG Lili, LYU Hongwu. Markov process-based availability modeling and analysis method of IaaS system [J]. Journal of Computer Applications, 2020, 40(10): 3013-3018.
[14]	WANG Qiaoling, QIAO Fei, JIANG Youhao. Improved K-means algorithm with aggregation distance coefficient [J]. Journal of Computer Applications, 2019, 39(9): 2586-2590.
[15]	LIN Li, XIONG Jinbo, XIAO Ruliang, LIN Mingwei, CHEN Xiuhua. Gaming@Edge: low latency cloud gaming system based on edge nodes [J]. Journal of Computer Applications, 2019, 39(7): 2001-2007.