Optimization of small files storage and accessing on Hadoop distributed file system

doi:10.11772/j.issn.1001-9081.2014.11.3091

Journal of Computer Applications ›› 2014, Vol. 34 ›› Issue (11): 3091-3095.DOI: 10.11772/j.issn.1001-9081.2014.11.3091

Previous Articles Next Articles

Optimization of small files storage and accessing on Hadoop distributed file system

LI Tie,YAN Cairong,HUANG Yongfeng,Song Yalong

School of Computer Science and Technology, Donghua University, Shanghai 201620, China

Received:2014-07-18 Revised:2014-07-30 Online:2014-11-01 Published:2014-12-01
Contact: YAN Cairong

面向Hadoop分布式文件系统的小文件存取优化方法

李铁,燕彩蓉,黄永锋,宋亚龙

东华大学计算机科学与技术学院,上海 201620

通讯作者: 燕彩蓉
作者简介:李铁(1989-),男,湖南永州人,硕士研究生,主要研究方向:分布式存储、分布式计算;燕彩蓉(1978-),女,湖北仙桃人,副教授,博士,主要研究方向:并行计算、分布式计算、大数据处理;黄永锋(1971-),男,山东泰安人,副教授,博士,主要研究方向:数据挖掘、机器学习、图像处理;宋亚龙(1988-),男,河南鹤壁人,硕士研究生,主要研究方向:分布式存储、分布式计算。
基金资助:
国家自然科学基金资助项目;中央高校基本科研业务费专项资金资助项目;上海市自然科学基金资助项目

Abstract

Abstract:

In order to improve the efficiency of processing small files in Hadoop Distributed File System (HDFS), a new efficient approach named SmartFS was proposed. By analyzing the file accessing log to obtain the accessing behavior of users, SmartFS established a probability model of file associations. This model was the reference of merging algorithm to merge the relevant small files into large files which would be stored on HDFS. When a file was accessed, SmartFS prefetched the related files according to the prefetching algorithm to accelerate the access speed. To guarantee the enough cache space, a cache replacement algorithm was put forward. The experimental results show that SmartFS can save the metadata space of NameNode in HDFS, reduce the interaction between users and HDFS, and improve the storing and accessing efficiency of small files on HDFS.

摘要：

为提高Hadoop分布式文件系统(HDFS)的小文件处理效率,提出了一种面向HDFS的智能小文件存取优化方法——SmartFS。SmartFS通过分析小文件访问日志,获取用户访问行为,建立文件关联概率模型,并根据基于文件关联关系的合并算法将小文件组装成大文件之后存至HDFS;当从HDFS获取文件时,根据基于文件关联关系的预取算法来提高文件访问效率,并提出基于预取的缓存替换算法来管理缓存空间,从而提高文件的命中率。实验结果表明,SmartFS有效减少了HDFS中NameNode的元数据空间,减少了用户与HDFS的交互次数,提高了小文件的存储效率和访问速度。

CLC Number:

TP316.4

LI Tie YAN Cairong HUANG Yongfeng Song Yalong. Optimization of small files storage and accessing on Hadoop distributed file system[J]. Journal of Computer Applications, 2014, 34(11): 3091-3095.

李铁燕彩蓉黄永锋宋亚龙. 面向Hadoop分布式文件系统的小文件存取优化方法[J]. 计算机应用, 2014, 34(11): 3091-3095.

References

[1]DONG B, QIU J, ZHENG Q, et al.A novel approach to improving the efficiency of storing and accessing small files on Hadoop: a case study by PowerPoint files [C]// SCC 2010: Proceedings of the 7th IEEE International Conference on Services Computing. Piscataway: IEEE Press, 2010: 65-72.
[2]LIU X, HAN J, ZHONG Y, et al.Implementing WebGIS on Hadoop: a case study of improving small file I/O performance on HDFS [C]// CLUSTER 2009: Proceedings of the 2009 IEEE International Conference on Cluster Computing and Workshops. Piscataway: IEEE Press, 2009: 1-8.
[3]JIANG L, LI B, SONG M. The optimization of HDFS based on small files [C]// IC-BNMT 2010: Proceedings of the 3rd IEEE International Conference on Broadband Network and Multimedia Technology. Piscataway: IEEE Press, 2010: 912-915.
[4]DONG B, ZHENG Q, TIAN F, et al.An optimized approach for storing and accessing small files on cloud storage[J]. Journal of Network and Computer Applications, 2012, 35(6): 1847-1862.
[5]GOHIL P, PANCHAL B. Efficient ways to improve the performance of HDFS for small files[J]. Computer Engineering and Intelligent Systems, 2014, 5(1): 45-49.
[6]CHEN J, WANG D, FU L, et al.An improved small file processing method for HDFS[J]. International Journal of Digital Content Technology and its Applications, 2012, 6(20): 296-304.
[7]HUA X, WU H, LI Z, et al.Enhancing throughput of the Hadoop distributed file system for interaction-intensive tasks[J]. Journal of Parallel and Distributed Computing, 2014, 74(8): 2770-2779.
[8]CHANDRASEKAR S, DAKSHINAMURTHY R, SESHAKUMAR P, et al.A novel indexing scheme for efficient handling of small files in Hadoop distributed file system [C]// ICCCI 2013: Proceedings of the 2013 International Conference on Computer Communication and Informatics. Piscataway: IEEE Press, 2013: 1-8.
[9]MACKEY G, SEHRISH S, WANG J. Improving metadata management for small files in HDFS [C]// CLUSTER 2009: Proceedings of the 2009 IEEE International Conference on Cluster Computing and Workshops. Piscataway: IEEE Press, 2009: 1-4.
[10]HAO Q, ZHU M, HAO J. New proxy cache replacement policy[J]. Journal of Computer Research and Development, 2002, 39(10): 1178-1185.(郝沁汾,祝明发,郝继升.一种新的代理缓存替换策略[J].计算机研究与发展,2002,39(10):1178-1185.)
[11]SHI L, MENG C, HAN Y. Web replacement policy based on prediction[J]. Journal of Computer Applications, 2007, 27(8): 1842-1845.(石磊,孟彩霞,韩英杰.基于预测的Web缓存替换策略[J].计算机应用,2007,27(8):1842-1845.)
[12]JIN Z, ZHANG G. Intelligent prefetch and cache techniques based on network performance[J]. Journal of Computer Research and Development, 2001, 38(8): 1000-1004.(金志刚,张钢.基于网络性能的智能Web加速技术——缓存与预取[J].计算机研究与发展,2001,38(8):1000-1004.)

[1]	. Adaptive video analysis in edge computing [J]. Journal of Computer Applications, 0, (): 0-0.
[2]	LI Yefei, XU Chao, XU Daoqiang, ZOU Yunfeng, ZHANG Xiaoda, QIAN Zhuzhong. Online task scheduling algorithm for big data analytics based on cumulative running work [J]. Journal of Computer Applications, 2019, 39(8): 2431-2437.
[3]	CAI Tao, CHEN Zhipeng, NIU Dejiao, WANG Jie, ZHAN Bisheng. New NVM storage system supporting high concurrent access [J]. Journal of Computer Applications, 2019, 39(1): 51-56.
[4]	WANG Peng, ZHOU Yan. MPI big data processing for high performance applications [J]. Journal of Computer Applications, 2018, 38(12): 3496-3499.
[5]	GENG Haijun, LIU Jieqi. Distributed load balancing algorithm based on hop-by-hop computing [J]. Journal of Computer Applications, 2018, 38(12): 3524-3528.
[6]	. Research on MPI Big Data Processing for High Performance Applications [J]. Journal of Computer Applications, 0, (): 0-0.
[7]	WEN Liang, JIANG Wei, PAN Xiong, ZHOU Keran, DONG Qi, WANG Junlong. Optimization design of preventing fault injection attack on distributed embedded systems [J]. Journal of Computer Applications, 2016, 36(2): 495-498.
[8]	WEI Min, LIU Yi'an, WU Hongyan. Design and implementation of local processor in a distributed system [J]. Journal of Computer Applications, 2015, 35(5): 1290-1295.
[9]	WANG Mingkun YUAN Shaoguang ZHU Yongli WANG Dewen. Real-time clustering for massive data using Storm [J]. Journal of Computer Applications, 2014, 34(11): 3078-3081.
[10]	ZHANG Qiaolong ZHANG Guizhu WU Delong. Multi-dimensional QoS cloud task scheduling algorithm based on task replication [J]. Journal of Computer Applications, 2014, 34(9): 2527-2531.
[11]	LIU Heng LI Meian SU Meng. Shortest cyclic quorum generation algorithm based on number of repetitions [J]. Journal of Computer Applications, 2014, 34(5): 1263-1266.
[12]	WANG Fang LI Meian DUAN Weijun. Cloud computing task scheduling based on dynamically adaptive ant colony algorithm [J]. Journal of Computer Applications, 2013, 33(11): 3160-3162.
[13]	WU Peng LI Meian. Quorum generation algorithm with time complexity of O(n) [J]. Journal of Computer Applications, 2013, 33(02): 323-360.
[14]	LIU Lian ZHENG Biao GONG Yi-li. Metadata processing optimization in distributed file systems [J]. Journal of Computer Applications, 2012, 32(12): 3271-3273.
[15]	ZHANG Chun-yan LIU Qing-lin MENG Ke. Task allocation based on ant colony optimization in cloud computing [J]. Journal of Computer Applications, 2012, 32(05): 1418-1420.

Optimization of small files storage and accessing on Hadoop distributed file system

面向Hadoop分布式文件系统的小文件存取优化方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics