DFS-Cache： memory-efficient and persistent client cache for distributed file systems

doi:10.11772/j.issn.1001-9081.2023050590

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (4): 1172-1180.DOI: 10.11772/j.issn.1001-9081.2023050590

Special Issue: 先进计算

• Advanced computing • Previous Articles Next Articles

DFS-Cache： memory-efficient and persistent client cache for distributed file systems

Ruixuan NI¹, Miao CAI¹^,²(), Baoliu YE¹^,²^,³

^1.College of Computer and Information，Hohai University，Nanjing Jiangsu 211100，China
^2.Key Laboratory of Water Big Data Technology of Ministry of Water Resources （Hohai University），Nanjing Jiangsu 211100，China
^3.State Key Laboratory for Novel Software Technology （Nanjing University），Nanjing Jiangsu 210023，China

Received:2023-05-16 Revised:2023-07-29 Accepted:2023-07-31 Online:2023-08-10 Published:2024-04-10
Contact: Miao CAI
About author:NI Ruixuan， born in 1999， M. S. candidate. His research interests include distributed system.
CAI Miao， born in 1991， Ph. D.， assistant researcher. His research interests include memory/storage system， operating system.
YE Baoliu， born in 1976， Ph. D.， professor. His research interests include distributed system， cloud computing， wireless network.
Supported by:
National Natural Science Foundation of China(61832005);Fundamental Research Funds for Central Universities(B220202073);Natural Science Foundation of Jiangsu Province(BK20220973);China Postdoctoral Science Foundation(2022M711014);Jiangsu Planned Projects for Postdoctoral Research Funds(2021K635C)

内存高效的持久性分布式文件系统客户端缓存DFS-Cache

倪瑞轩¹, 蔡淼¹^,²(), 叶保留¹^,²^,³

^1.河海大学计算机与信息学院，南京 211100
^2.水利部水利大数据重点实验室（河海大学），南京 211100
^3.计算机软件新技术国家重点实验室（南京大学），南京 210023

通讯作者: 蔡淼
作者简介:倪瑞轩（1999—），男，江苏南京人，硕士研究生，主要研究方向：分布式系统
蔡淼（1991—），男，江苏盐城人，助理研究员，博士，CCF会员，主要研究方向：内存/存储系统、操作系统 mcai@hhu.edu.cn
叶保留（1976—），男，江苏南通人，教授，博士，CCF高级会员，主要研究方向：分布式系统、云计算、无线网络。
基金资助:
国家自然科学基金资助项目(61832005);中央高校业务经费资助项目(B220202073);江苏省自然科学基金资助项目(BK20220973);中国博士后科学基金资助项目(2022M711014);江苏省博士后科研资助计划项目(2021K635C)

Abstract

Abstract:

To effectively reduce cache defragmentation overhead and improve cache hit radio in data-intensive workflows， a persistent client cache for distributed file system was proposed， namely DFS-Cache （Distributed File System Cache）， which was designed and implemented based on Non-Volatile Memory （NVM） and was able to ensure data persistence and crash consistency with significantly reducing cold start time. DFS-Cache was consisted of a cache defragmentation mechanism based on virtual memory remapping and a cache space management strategy based on Time-To-Live （TTL）. The former was based on the characteristic that NVM could be directly addressed by the memory controller. By dynamically modifying the mapping relationship between virtual addresses and physical addresses， zero-copy memory defragmentation was achieved. The latter was a cold-hot separated grouping management strategy that could enhance cache space management efficiency with the support of the remapping-based cache defragmentation mechanism. Experiments were conducted using real Intel Optane persistent memory devices. Compared with commercial distributed file systems MooseFS and GlusterFS， while employing standard benchmarking programs like Fio and Filebench， the proposed client cache can increase the system throughput by up to 5.73 times and 1.89 times.

Key words: Non-Volatile Memory (NVM), distributed file system, client cache, cache defragmentation, cold-hot data grouping, cache design

摘要：

为了在数据密集型工作流下有效降低缓存碎片整理开销并提高缓存命中率，提出一种持久性分布式文件系统客户端缓存DFS-Cache（Distributed File System Cache）。DFS-Cache基于非易失性内存（NVM）设计实现，能够保证数据的持久性和崩溃一致性，并大幅减少冷启动时间。DFS-Cache包括基于虚拟内存重映射的缓存碎片整理机制和基于生存时间（TTL）的缓存空间管理策略。前者基于NVM可被内存控制器直接寻址的特性，动态修改虚拟地址和物理地址之间的映射关系，实现零拷贝的内存碎片整理；后者是一种冷热分离的分组管理策略，借助重映射的缓存碎片整理机制，提升缓存空间的管理效率。实验采用真实的Intel傲腾持久性内存设备，对比商用的分布式文件系统MooseFS和GlusterFS，采用Fio和Filebench等标准测试程序，DFS-Cache最高能提升5.73倍和1.89倍的系统吞吐量。

关键词: 非易失性内存, 分布式文件系统, 客户端缓存, 缓存碎片整理, 冷热数据分组, 缓存设计

CLC Number:

TP316.4

Ruixuan NI, Miao CAI, Baoliu YE. DFS-Cache： memory-efficient and persistent client cache for distributed file systems[J]. Journal of Computer Applications, 2024, 44(4): 1172-1180.

倪瑞轩, 蔡淼, 叶保留. 内存高效的持久性分布式文件系统客户端缓存DFS-Cache[J]. 《计算机应用》唯一官方网站, 2024, 44(4): 1172-1180.

Figures/Tables 12

Fig. 1 Distributed file system architecture

Fig. 2 DFS-Cache architecture

Fig. 3 Defragmentation of cache based on virtual memory remapping

Fig. 4 TTL-based cache space management strategy

Fig. 5 Change of number of idle blocks in cache， space utilization and hit ratio

Fig. 6 Number of cache fragments defragmented each time

Fig. 7 Comparison of cache space utilization with different I/O sizes

Fig. 8 Comparison of cache hit ratios with different I/O sizes

Fig. 9 Comparison of cache space utilization with different α

Fig. 10 Comparison of cache hit ratios with different α

Fig. 11 Throughput varies with α of Zipf and I/O size range

Tab. 1 Comparison of average latency under various workflows of Filebench

DFS-Cache与对比对象	随机读写的平均延时	随机读的平均延时
DFS-Cache	2.649	0.203
MooseFS	3.288	0.430
MooseFS+pmdk	2.947	0.244
GlusterFS	2.926	0.455

References 33

1	罗军舟，吴文甲，杨明.移动互联网：终端、网络与服务［J］.计算机学报，2011，34（11）：2029-2051. 10.3724/sp.j.1016.2011.02029
	LUO J Z， WU W J， YANG M. Mobile internet： terminal devices，networks and services［J］.Chinese Journal of Computers，2011，34（11）：2029-2051. 10.3724/sp.j.1016.2011.02029
2	ATZORI L， IERA A， MORABITO G. The internet of things： a survey computer networks［J］. Computer Networks， 2010， 54（15）： 2787-2805. 10.1016/j.comnet.2010.05.010
3	BOYD D M， ELLISON N B. Social network sites： definition， history， and scholarship［J］. Journal of Computer-Mediated Communication， 2007， 13（1）： 210-230. 10.1111/j.1083-6101.2007.00393.x
4	ARMBRUST M， FOX A， GRIFFITH R， et al. A view of cloud computing［J］. Communications of the ACM， 2010， 53（4）： 50-58. 10.1145/1721654.1721672
5	MULLENDER S. Distributed systems［M］. New York：ACM Press， 1990： 1-26. 10.1145/90417
6	ANDERSON T E， CANINI M， KIM J， et al. Assise： performance and availability via client-local NVM in a distributed file system［C］// Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation. Berkeley： USENIX Association， 2020： 1011-1027.
7	ISLAM N S， WASI-UR-RAHMAN M， LU X， et al. High performance design for HDFS with byte-addressability of NVM and RDMA［C］// Proceedings of the 2016 International Conference on Supercomputing. New York： ACM， 2016： No.8. 10.1145/2925426.2926290
8	LU Y， SHU J， CHEN Y， et al. Octopus： an RDMA-enabled distributed persistent memory file system［C］// Proceedings of the 2017 USENIX Annual Technical Conference. Berkeley： USENIX Association， 2017： 773-785.
9	QIAN Y， LI X， IHARA S， et al. LPCC： hierarchical persistent client caching for Lustre［C］// Proceedings of the 2019 International Conference for High Performance Computing， Networking， Storage and Analysis. New York： ACM， 2019： No. 88. 10.1145/3295500.3356139
10	CHENG W， LI C， ZENG L， et al. NVMM-oriented hierarchical persistent client caching for Lustre［J］. ACM Transactions on Storage， 2021， 17（1）： No. 6. 10.1145/3404190
11	LIAO J， TRAHAY F， CAI Z， et al. Fine granularity and adaptive cache update mechanism for client caching［J］. IEEE Systems Journal， 2019， 13（2）： 1587-1598. 10.1109/jsyst.2018.2866905
12	BAO N， CHAI Y， QIN X. A write-efficient cache algorithm based on macroscopic trend for NVM-based read cache［C］// Proceedings of the 2019 Design， Automation & Test in Europe Conference & Exhibition. Piscataway： IEEE， 2019： 1245-1248. 10.23919/date.2019.8715276
13	XU Y， YE C， SOLIHIN Y， et al. FFCCD： fence-free crash-consistent concurrent defragmentation for persistent memory［C］// Proceedings of the 49th Annual International Symposium on Computer Architecture. New York： ACM， 2022： 274-288. 10.1145/3470496.3527406
14	LI L， JIANG W， PAN R， et al. On-line memorry defragmentation for NVM-based persistent heaps［C］// Proceedings of the 2017 IEEE 6th Non-Volatile Memory Systems and Applications Symposium. Piscataway： IEEE， 2017： 1-4. 10.1109/nvmsa.2017.8064471
15	GÖTZE P， VAN RENEN A， LERSCH L， et al. Data management on non-volatile memory： a perspective［J］. Datenbank-Spektrum， 2018， 18： 171-182. 10.1007/s13222-018-0301-1
16	OUKID I， BOOSS D， LESPINASSE A， et al. Memory management techniques for large-scale persistent-main-memory systems［J］. Proceedings of the VLDB Endowment， 2017， 10（11）： 1166-1177. 10.14778/3137628.3137629
17	QURESHI M K， SRINIVASAN V， RIVERS J A. Scalable high performance main memory system using phase-change memory technology［C］// Proceedings of the 36th Annual International Symposium on Computer architecture. New York： ACM， 2009： 24-33. 10.1145/1555754.1555760
18	BURR G W， BREITWISCH M J， FRANCESCHINI M， et al. Phase change memory technology［J］. Journal of Vacuum Science & Technology B， 2010， 28： 223-262. 10.1116/1.3301579
19	LEE B C， IPEK E， MUTLU O， et al. Architecting phase change memory as a scalable dram alternative［J］. ACM SIGARCH Computer Architecture News， 2009， 37（3）： 2-13. 10.1145/1555815.1555758
20	AKINAGA H， SHIMA H. Resistive Random Access Memory （ReRAM） based on metal oxides［J］. Proceedings of the IEEE， 2010， 98（12）： 2237-2251. 10.1109/jproc.2010.2070830
21	JUNG M， SHALF J， KANDEMIR M. Design of a large-scale storage-class RRAM system［C］// Proceedings of the 27th International ACM Conference on International Conference on Supercomputing. New York： ACM， 2013： 103-114. 10.1145/2464996.2465004
22	WEIL S A， BRANDT S A， MILLER E L， et al. Ceph： a scalable， high-performance distributed file system［C］// Proceedings of the 7th Symposium on Operating Systems Design and Implementation. Berkeley： USENIX Association， 2006： 307-320.
23	GHEMAWAT S， GOBIOFF H， S-T LEUNG. The Google file system［J］. ACM SIGOPS Operating Systems Review， 2003， 37（5）： 29-43. 10.1145/1165389.945450
24	YANG J， YUE Y， RASHMI K V. A large scale analysis of hundreds of in-memory cache clusters at Twitter［C］// Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation. Berkeley： USENIX Association， 2020： 191-208. 10.1145/3468521
25	SUNG H， BANG J， KIM C， et al. BBOS： efficient HPC storage management via burst buffer over-subscription［C］// Proceedings of the 2020 20th IEEE/ACM International Symposium on Cluster， Cloud and Internet Computing. Piscataway： IEEE， 2020： 142-151. 10.1109/ccgrid49817.2020.00-79
26	RAMASWAMY L， IYENGAR A， LIU L， et al. Automatic fragment detection in dynamic web pages and its impact on caching［J］. IEEE Transactions on Knowledge and Data Engineering， 2005， 17（6）： 859-874. 10.1109/tkde.2005.89
27	HERZBERG A， SHULMAN H. Fragmentation considered poisonous， or： one-domain-to-rule-them-all. org［C］// Proceedings of the 2013 IEEE Conference on Communications and Network Security. Piscataway： IEEE， 2013： 224-232. 10.1109/cns.2013.6682711
28	GUERRERO C， JUIZ C， PUIGJANER R. Improving web cache performance via adaptive content fragmentation design［C］// Proceedings of the 2011 IEEE 10th International Symposium on Network Computing and Applications. Piscataway： IEEE， 2011： 310-313. 10.1109/nca.2011.55
29	SANBERG R， GOLDBERG D， KLEIMAN S， et al. Design and implementation of the Sun network filesystem［C］// Proceedings of the Summer 1985 USENIX Conference. Berkeley： USENIX Association， 1985： 119-130.
30	HOWARD J H， KAZAR M L， MENEES S G， et al. Scale and performance in a distributed file system［J］. ACM Transactions on Computer Systems， 1988， 6（1）： 51-81. 10.1145/35037.35059
31	NIGHTINGALE E B， CHEN P M， FLINN J. Speculative execution in a distributed file system［J］. ACM SIGOPS Operating Systems Review， 2005， 39（5）： 191-205. 10.1145/1095809.1095829
32	BZOCH P， AFARÍK J. Maintaining cache consistency for mobile clients in distributed file system［C］// Proceedings of the 2013 3rd Eastern European Regional Conference on the Engineering of Computer Based Systems. Washington， DC： IEEE Computer Society， 2013： 55-62. 10.1109/ecbs-eerc.2013.16
33	SUN J， HU C， WO T， et al. HCFS2： a file storage service with weak consistency in the hybrid cloud［C］// Proceedings of the 2018 IEEE Symposium on Service-Oriented System Engineering. Piscataway： IEEE， 2018： 228-233. 10.1109/sose.2018.00038

[1]	Xu LI, Yulin HE, Laizhong CUI, Zhexue HUANG, Fournier‑Viger PHILIPPE. Distributed observation point classifier for big data with random sample partition [J]. Journal of Computer Applications, 2024, 44(6): 1727-1733.
[2]	TANG Chen, HUANG Guorui, JIN Peiquan. NVM-LH: non-volatile memory-friendly linear hash index [J]. Journal of Computer Applications, 2021, 41(3): 623-629.
[3]	GOU Zi'an, ZHANG Xiao, WU Dongnan, WANG Yanqiu. Log analysis and workload characteristic extraction in distributed storage system [J]. Journal of Computer Applications, 2020, 40(9): 2586-2593.
[4]	DONG Cong, ZHANG Xiao, CHENG Wendi, SHI Jia. Performance optimization of distributed file system based on new type storage devices [J]. Journal of Computer Applications, 2020, 40(12): 3594-3603.
[5]	CAI Tao, CHEN Zhipeng, NIU Dejiao, WANG Jie, ZHAN Bisheng. New NVM storage system supporting high concurrent access [J]. Journal of Computer Applications, 2019, 39(1): 51-56.
[6]	CHEN Bo, HE Lianyue, YAN Weiwei, XU Zhaomiao, XU Jun. Portable operating system interface of UNIX compatibility technology in mass small distributed file system [J]. Journal of Computer Applications, 2018, 38(5): 1389-1392.
[7]	LI Qiang, LIU Xiaofeng. Load balancing strategy of cloud storage based on Hopfield neural network [J]. Journal of Computer Applications, 2017, 37(8): 2214-2217.
[8]	YANG Shun, CHEN Zhiguang, XIAO Nong. Implementation of directory index for Pmfs [J]. Journal of Computer Applications, 2017, 37(5): 1241-1245.
[9]	YANG Junjie, LIAO Zhuofan, FENG Chaochao. Survey on big data storage framework and algorithm [J]. Journal of Computer Applications, 2016, 36(9): 2465-2471.
[10]	LIU Qing, FU Yinjin, NI Guiqiang, MEI Jianmin. Distributed deduplication storage system based on Hadoop platform [J]. Journal of Computer Applications, 2016, 36(2): 330-335.
[11]	WU Jinbo, SONG Jie, ZHANG Li, BAO Yubin. Probery: probability-based data query system for big data [J]. Journal of Computer Applications, 2016, 36(1): 8-12.
[12]	SHAO Tian, CHEN Guangsheng, JING Weipeng. Cut-GAR: solution to determine cut-off point in cloud storage system [J]. Journal of Computer Applications, 2015, 35(9): 2497-2502.
[13]	YANG Wenhui, LI Guoqiang, MIAO Fang. Metadata management mechanism of massive spatial data storage [J]. Journal of Computer Applications, 2015, 35(5): 1276-1279.
[14]	ZHENG Kai, ZHU Lin, CHEN Youguang. Load balancing cloud storage algorithm based on Kademlia [J]. Journal of Computer Applications, 2015, 35(3): 643-647.
[15]	WANG Zhengying, YU Jiong, YING Changtian, LU Liang. Energy-efficient strategy of distributed file system based on data block clustering storage [J]. Journal of Computer Applications, 2015, 35(2): 378-382.

DFS-Cache： memory-efficient and persistent client cache for distributed file systems

内存高效的持久性分布式文件系统客户端缓存DFS-Cache

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 33

Related Articles 15

Recommended Articles

Metrics