面向Hadoop分布式文件系统的小文件存取优化方法

doi:10.11772/j.issn.1001-9081.2014.11.3091

计算机应用 ›› 2014, Vol. 34 ›› Issue (11): 3091-3095.DOI: 10.11772/j.issn.1001-9081.2014.11.3091

• 2014年全国开放式分布与并行计算学术年会(DPCS 2014)论文 • 上一篇下一篇

面向Hadoop分布式文件系统的小文件存取优化方法

李铁,燕彩蓉,黄永锋,宋亚龙

东华大学计算机科学与技术学院,上海 201620

收稿日期:2014-07-18 修回日期:2014-07-30 出版日期:2014-11-01 发布日期:2014-12-01
通讯作者: 燕彩蓉
作者简介:李铁(1989-),男,湖南永州人,硕士研究生,主要研究方向:分布式存储、分布式计算;燕彩蓉(1978-),女,湖北仙桃人,副教授,博士,主要研究方向:并行计算、分布式计算、大数据处理;黄永锋(1971-),男,山东泰安人,副教授,博士,主要研究方向:数据挖掘、机器学习、图像处理;宋亚龙(1988-),男,河南鹤壁人,硕士研究生,主要研究方向:分布式存储、分布式计算。
基金资助:
国家自然科学基金资助项目;中央高校基本科研业务费专项资金资助项目;上海市自然科学基金资助项目

Optimization of small files storage and accessing on Hadoop distributed file system

LI Tie,YAN Cairong,HUANG Yongfeng,Song Yalong

School of Computer Science and Technology, Donghua University, Shanghai 201620, China

Received:2014-07-18 Revised:2014-07-30 Online:2014-11-01 Published:2014-12-01
Contact: YAN Cairong

摘要/Abstract

摘要：

为提高Hadoop分布式文件系统(HDFS)的小文件处理效率,提出了一种面向HDFS的智能小文件存取优化方法——SmartFS。SmartFS通过分析小文件访问日志,获取用户访问行为,建立文件关联概率模型,并根据基于文件关联关系的合并算法将小文件组装成大文件之后存至HDFS;当从HDFS获取文件时,根据基于文件关联关系的预取算法来提高文件访问效率,并提出基于预取的缓存替换算法来管理缓存空间,从而提高文件的命中率。实验结果表明,SmartFS有效减少了HDFS中NameNode的元数据空间,减少了用户与HDFS的交互次数,提高了小文件的存储效率和访问速度。

Abstract:

In order to improve the efficiency of processing small files in Hadoop Distributed File System (HDFS), a new efficient approach named SmartFS was proposed. By analyzing the file accessing log to obtain the accessing behavior of users, SmartFS established a probability model of file associations. This model was the reference of merging algorithm to merge the relevant small files into large files which would be stored on HDFS. When a file was accessed, SmartFS prefetched the related files according to the prefetching algorithm to accelerate the access speed. To guarantee the enough cache space, a cache replacement algorithm was put forward. The experimental results show that SmartFS can save the metadata space of NameNode in HDFS, reduce the interaction between users and HDFS, and improve the storing and accessing efficiency of small files on HDFS.

中图分类号:

TP316.4

李铁燕彩蓉黄永锋宋亚龙. 面向Hadoop分布式文件系统的小文件存取优化方法[J]. 计算机应用, 2014, 34(11): 3091-3095.

LI Tie YAN Cairong HUANG Yongfeng Song Yalong. Optimization of small files storage and accessing on Hadoop distributed file system[J]. Journal of Computer Applications, 2014, 34(11): 3091-3095.

参考文献

[1]DONG B, QIU J, ZHENG Q, et al.A novel approach to improving the efficiency of storing and accessing small files on Hadoop: a case study by PowerPoint files [C]// SCC 2010: Proceedings of the 7th IEEE International Conference on Services Computing. Piscataway: IEEE Press, 2010: 65-72.
[2]LIU X, HAN J, ZHONG Y, et al.Implementing WebGIS on Hadoop: a case study of improving small file I/O performance on HDFS [C]// CLUSTER 2009: Proceedings of the 2009 IEEE International Conference on Cluster Computing and Workshops. Piscataway: IEEE Press, 2009: 1-8.
[3]JIANG L, LI B, SONG M. The optimization of HDFS based on small files [C]// IC-BNMT 2010: Proceedings of the 3rd IEEE International Conference on Broadband Network and Multimedia Technology. Piscataway: IEEE Press, 2010: 912-915.
[4]DONG B, ZHENG Q, TIAN F, et al.An optimized approach for storing and accessing small files on cloud storage[J]. Journal of Network and Computer Applications, 2012, 35(6): 1847-1862.
[5]GOHIL P, PANCHAL B. Efficient ways to improve the performance of HDFS for small files[J]. Computer Engineering and Intelligent Systems, 2014, 5(1): 45-49.
[6]CHEN J, WANG D, FU L, et al.An improved small file processing method for HDFS[J]. International Journal of Digital Content Technology and its Applications, 2012, 6(20): 296-304.
[7]HUA X, WU H, LI Z, et al.Enhancing throughput of the Hadoop distributed file system for interaction-intensive tasks[J]. Journal of Parallel and Distributed Computing, 2014, 74(8): 2770-2779.
[8]CHANDRASEKAR S, DAKSHINAMURTHY R, SESHAKUMAR P, et al.A novel indexing scheme for efficient handling of small files in Hadoop distributed file system [C]// ICCCI 2013: Proceedings of the 2013 International Conference on Computer Communication and Informatics. Piscataway: IEEE Press, 2013: 1-8.
[9]MACKEY G, SEHRISH S, WANG J. Improving metadata management for small files in HDFS [C]// CLUSTER 2009: Proceedings of the 2009 IEEE International Conference on Cluster Computing and Workshops. Piscataway: IEEE Press, 2009: 1-4.
[10]HAO Q, ZHU M, HAO J. New proxy cache replacement policy[J]. Journal of Computer Research and Development, 2002, 39(10): 1178-1185.(郝沁汾,祝明发,郝继升.一种新的代理缓存替换策略[J].计算机研究与发展,2002,39(10):1178-1185.)
[11]SHI L, MENG C, HAN Y. Web replacement policy based on prediction[J]. Journal of Computer Applications, 2007, 27(8): 1842-1845.(石磊,孟彩霞,韩英杰.基于预测的Web缓存替换策略[J].计算机应用,2007,27(8):1842-1845.)
[12]JIN Z, ZHANG G. Intelligent prefetch and cache techniques based on network performance[J]. Journal of Computer Research and Development, 2001, 38(8): 1000-1004.(金志刚,张钢.基于网络性能的智能Web加速技术——缓存与预取[J].计算机研究与发展,2001,38(8):1000-1004.)

[1]	戴炀陈彧祝永晋张胜. 边缘环境下的自适应视频分析[J]. 计算机应用, 0, (): 0-0.
[2]	李叶飞, 徐超, 许道强, 邹云峰, 张晓达, 钱柱中. 基于累计工作量的在线大数据分析作业调度算法[J]. 计算机应用, 2019, 39(8): 2431-2437.
[3]	蔡涛, 陈志鹏, 牛德姣, 王杰, 詹毕晟. 支持高并发访问的新型NVM存储系统[J]. 计算机应用, 2019, 39(1): 51-56.
[4]	王鹏, 周岩. 面向高性能应用的MPI大数据处理[J]. 计算机应用, 2018, 38(12): 3496-3499.
[5]	耿海军, 刘洁琦. 基于逐跳计算的分布式负载均衡算法[J]. 计算机应用, 2018, 38(12): 3524-3528.
[6]	王鹏周岩. 面向高性能应用的MPI大数据处理研究[J]. 计算机应用, 0, (): 0-0.
[7]	文亮, 江维, 潘雄, 周可染, 董琪, 王峻龙. 分布式系统中抵御错误注入攻击的优化设计[J]. 计算机应用, 2016, 36(2): 495-498.
[8]	魏敏, 刘以安, 吴鸿雁. 分布式系统中局部处理机的设计与实现[J]. 计算机应用, 2015, 35(5): 1290-1295.
[9]	王铭坤袁少光朱永利王德文. 基于Storm的海量数据实时聚类[J]. 计算机应用, 2014, 34(11): 3078-3081.
[10]	张巧龙张桂珠吴德龙. 基于任务复制的多维QoS云计算任务调度[J]. 计算机应用, 2014, 34(9): 2527-2531.
[11]	刘恒李美安苏萌. 基于重复数的最短循环请求集生成算法[J]. 计算机应用, 2014, 34(5): 1263-1266.
[12]	王芳李美安段卫军. 基于动态自适应蚁群算法的云计算任务调度[J]. 计算机应用, 2013, 33(11): 3160-3162.
[13]	武鹏李美安. 具有O(n)时间复杂度的分布式请求集生成算法[J]. 计算机应用, 2013, 33(02): 323-360.
[14]	刘恋郑彪龚奕利. 分布式文件系统中元数据操作的优化研究[J]. 计算机应用, 2012, 32(12): 3271-3273.
[15]	张春艳刘清林孟珂. 基于蚁群优化算法的云计算任务分配[J]. 计算机应用, 2012, 32(05): 1418-1420.

面向Hadoop分布式文件系统的小文件存取优化方法

Optimization of small files storage and accessing on Hadoop distributed file system

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics