海量小文件系统的可移植操作系统接口兼容技术

doi:10.11772/j.issn.1001-9081.2017102934

计算机应用 ›› 2018, Vol. 38 ›› Issue (5): 1389-1392.DOI: 10.11772/j.issn.1001-9081.2017102934

海量小文件系统的可移植操作系统接口兼容技术

陈博¹, 何连跃^1,2, 严巍巍², 徐照淼¹, 徐俊¹

1. 国防科技大学计算机学院, 长沙 410073;
2. 北京网云飞信息技术有限公司, 北京 100067

收稿日期:2017-12-13 修回日期:2017-12-14 出版日期:2018-05-10 发布日期:2018-05-24
通讯作者: 陈博
作者简介:陈博(1993-),男,湖南湘潭人,硕士研究生,主要研究方向:分布式文件系统、云计算;何连跃(1971-),男,浙江武义人,研究员,博士,主要研究方向:分布式文件系统、信息安全;严巍巍(1991-),男,江苏南通人,硕士,主要研究方向:分布式文件系统;徐照淼(1994-),男,湖南常德人,硕士研究生,主要研究方向:分布式文件系统;徐俊(1993-),男,浙江武义人,硕士研究生,主要研究方向:分布式文件系统。

Portable operating system interface of UNIX compatibility technology in mass small distributed file system

CHEN Bo¹, HE Lianyue^1,2, YAN Weiwei², XU Zhaomiao¹, XU Jun¹

1. College of Computer, National University of Defense Technology, Changsha Hunan 410073, China;
2. Beijing Netclouds Information Technology Corporation Limited, Beijing 100070, China

Received:2017-12-13 Revised:2017-12-14 Online:2018-05-10 Published:2018-05-24
Contact: 陈博

摘要/Abstract

摘要： 基于Hadoop分布式文件系统（HDFS）研发的海量小文件系统（SMDFS）遗留了HDFS不兼容可移植操作系统接口（POSIX）约束的问题，为解决SMDFS的这一问题，提出基于本地缓存的POSIX兼容技术和基于数据暂存区的元数据高效管理技术。首先，通过设置数据暂存区来实现读写模式文件流的重定向，然后建立异步线程池模型，实现数据暂存区镜像文件的同步，从而完成用户层到存储层的所有POSIX相关的文件操作。此外，借助跳表结构的元数据缓存实现List目录等元数据操作效率优化。测试表明，相较于HDFS的Linux客户端，基于技术成果实现的SMDFS3.0的随机读性能有10倍以上的性能提升，顺序读和顺序写性能有约3~4倍的提升，随机写性能可以达到本地文件系统的20%，基于目录的元数据缓存的设计使目录的List操作效率提升近10倍。但是，由于用户空间文件系统（FUSE）挂载的客户端会引入额外的内核态和用户态切换等带来的开销，因此SMDFS3.0的Linux客户端相对于系统的Java接口会有大约50%的性能损耗。

关键词: 海量小文件系统, 分布式文件系统, 可移植操作系统接口兼容, 元数据缓存, 云存储

Abstract: Focused on the issue that the mass small file system developed based on HDFS (Hadoop Distributed File System), SMDFS (Mass Small Distributed File System), is not compatible with POSIX (Portable Operating System Interface of UNIX) constraints, a POSIX compatible technology based on local cache and an efficient metadata management technology based on temporary data cache were proposed. Firstly, the data storage area was set to realize the redirection of the file flow in the read-write mode, and then an asynchronous thread pool model was established to synchronize the data in temporary cache, thereby completing all POSIX-related file operations from the user layer to the storage layer. In addition, with the help of the metadata cache of the skip list structure, the efficiency of metadata operations such as the List directory was optimized. The test results show that, compared to the Linux client of HDFS, the performance of random read improves ten times more, the sequential read and sequential write improves about three to four times. The performance of random write can reach 20% of the local file system. Besides, the List operation efficiency of the directory improves about 10 times. However, due to the additional switching of kernel-mode and user-mode introduced by FUSE (Filesystem in Userspace), the Linux client of SMDFS3.0 has a performance penalty of about 50% compared to Java interface.

Key words: mass small file system, distributed file system, Portable Operating System Interface of UNIX (POSIX) compatibility, metadata cache, cloud storage

中图分类号:

TP311

陈博, 何连跃, 严巍巍, 徐照淼, 徐俊. 海量小文件系统的可移植操作系统接口兼容技术[J]. 计算机应用, 2018, 38(5): 1389-1392.

CHEN Bo, HE Lianyue, YAN Weiwei, XU Zhaomiao, XU Jun. Portable operating system interface of UNIX compatibility technology in mass small distributed file system[J]. Journal of Computer Applications, 2018, 38(5): 1389-1392.

参考文献

[1] HDFS users guide[EB/OL].[2017-06-25]. http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html.
[2] POSIX[EB/OL].[2017-06-26]. https://en.wikipedia.org/wiki/POSIX.
[3] 赵跃龙, 谢晓玲, 蔡咏才. 一种性能优化的小文件存储访问策略[J].计算机研究与发展, 2012, 49(7):1579-1586. (ZHAO Y L, XIE X L, CAI Y C. A strategy of small file storage access with performance optimization[J]. Journal of Computer Research and Development, 2012, 49(7):1579-1586.)
[4] LIU X H, HAN J Z, YUN Q, et al. Implementing WebGIS on Hadoop:a case study of improving small file I/O performance on HDFS[C]//Proceedings of the 2009 IEEE International Conference on Cluster Computing and Workshops. Piscataway, NJ:IEEE, 2009:1-8.
[5] 张春明, 芮建武, 何婷婷. 一种Hadoop小文件存储和读取的方法[J].计算机应用与软件, 2012, 29(11):95-100.(ZHANG C M,RUI J W, HE T T. An approach for storing and accessing small files on Hadoop[J]. Computer Applications and Software, 2012, 29(11):95-100.)
[6] DONG B, QIU J, ZHENG Q, et a1. A novel approach to improving the efficiency of storing and accessing small files on Hadoop:a case study by powerpoint files[C]//Proceedings of the 2010 IEEE International Conference on Services Computing. Piscataway, NJ:IEEE, 2010:65-72.
[7] YANG Z, DAN L. Improving the efficiency of storing for small files in HDFS[C]//Proceedings of the 2012 International Conference on Computer Science and Service System. Washington, DC:IEEE Computer Society, 2012:2239-2242.
[8] 严巍巍, 何连跃, 李三霞, 等. SMDFS分布式海量小文件系统的大空间聚合存储技术[J]. 计算机研究与发展, 2015, 52(S2):29-34.(YAN W W, HE L Y, LI S X, et al. Large aggregation space technology in mass small files system SMDFS[J]. Journal of Computer Research and Development, 2015, 52(S2):29-34.)
[9] 杨开杰,刘秋菊,徐汀荣.线程池的多线程并发控制技术研究[J].计算机应用与软件,2010,27(1):168-170.(YANG K J, LIU Q J, XU T R. On multi-threaded concurrent control technology for thread pool[J]. Computer Applications and Software, 2010,27(1):168-170.)
[10] LING Y, MULLEN T, LIN X. Analysis of optimal thread pool size[J]. ACM SIGOPS Operating Systems Review, 2000, 34(2):42-55.
[11] Skip list[EB/OL].[2017-07-01].https://en.wikipedia.org/wiki/Skip_list.
[12] Red-black tree[EB/OL].[2017-07-06].https://en.wikipedia.org/wiki/Red-black_tree.

[1]	李秀艳, 刘明曦, 史闻博, 董国芳. 面向资源受限用户的高效动态数据审计方案[J]. 计算机应用, 2021, 41(2): 422-432.
[2]	苟子安, 张晓, 吴东南, 王艳秋. 分布式存储系统中的日志分析与负载特征提取[J]. 计算机应用, 2020, 40(9): 2586-2593.
[3]	唐鑫, 周琳娜. 基于响应模糊化的抗附加块攻击云数据安全去重方法[J]. 计算机应用, 2020, 40(4): 1085-1090.
[4]	付伟, 顾晨阳, 高强. 基于属性加密的多用户共享ORAM方案[J]. 计算机应用, 2020, 40(2): 497-502.
[5]	董聪, 张晓, 程文迪, 石佳. 基于新型存储器件的分布式文件系统性能优化[J]. 计算机应用, 2020, 40(12): 3594-3603.
[6]	王海勇, 彭垚, 郭凯璇. 云存储中基于代理重加密的CP-ABE访问控制方案[J]. 计算机应用, 2019, 39(9): 2611-2616.
[7]	柳玉东, 王绪安, 涂广升, 王涵. 全生命周期的云外包数据安全审计协议[J]. 计算机应用, 2019, 39(7): 1954-1958.
[8]	明洋, 何宝康. 支持属性撤销的可验证外包的多授权属性加密方案[J]. 计算机应用, 2019, 39(12): 3556-3562.
[9]	周坚, 金瑜, 何亨, 李鹏. 基于嵌套Merkle Hash tree区块链的云数据动态审计模型[J]. 计算机应用, 2019, 39(12): 3575-3583.
[10]	李静, 刘冬实. 主动容错云存储系统的可靠性评价模型[J]. 计算机应用, 2018, 38(9): 2631-2636.
[11]	杨宏宇, 王玥. 云存储环境下的多关键字密文搜索方法[J]. 计算机应用, 2018, 38(2): 343-347.
[12]	王敏燊, 熊金波, 林倩, 王丽丽. 基于密钥分发和密文抽样的云数据确定性删除方案[J]. 计算机应用, 2018, 38(1): 194-200.
[13]	李强, 刘晓峰. 基于Hopfield神经网络的云存储负载均衡策略[J]. 计算机应用, 2017, 37(8): 2214-2217.
[14]	李昊宇, 张龙军, 李庆鹏. 基于B⁺树的动态数据持有性证明方案[J]. 计算机应用, 2017, 37(7): 1931-1935.
[15]	曹景源, 李立新, 李全良, 丁永善. 云存储环境下生命周期可控的数据销毁模型[J]. 计算机应用, 2017, 37(5): 1335-1340.

海量小文件系统的可移植操作系统接口兼容技术

Portable operating system interface of UNIX compatibility technology in mass small distributed file system

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics