基于Linux内核的Key-Value存储系统——KStore

doi:10.11772/j.issn.1001-9081.2015.01.0099

计算机应用 ›› 2015, Vol. 35 ›› Issue (1): 99-102.DOI: 10.11772/j.issn.1001-9081.2015.01.0099

基于Linux内核的Key-Value存储系统——KStore

谢沛东^1,2, 武延军^1,3

1. 中国科学院软件研究所基础软件国家工程研究中心, 北京100190;
2. 中国科学院大学, 北京100190;
3. 中国科学院软件研究所计算机科学国家重点实验室, 北京100190

收稿日期:2014-07-16 修回日期:2014-08-04 发布日期:2015-01-26 出版日期:2015-01-01
通讯作者: 谢沛东
作者简介:谢沛东(1988-),男,山东菏泽人,硕士研究生,主要研究方向:分布式存储系统、操作系统;武延军(1979-),男,陕西延安人,高级工程师,博士生导师,主要研究方向:云计算、操作系统.
基金资助:
中国科学院战略性科技先导专项(XDA06010600);核高基重大专项(2012ZX01039-004).

KStore: Linux kernel-based Key-Value store system

XIE Peidong^1,2, WU Yanjun^1,3

1. National Engineering Research Center of Fundamental Software, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China;
2. University of Chinese Academy of Sciences, Beijing 100190, China;
3. State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China

Received:2014-07-16 Revised:2014-08-04 Online:2015-01-26 Published:2015-01-01

摘要/Abstract

摘要：

Key-Value存储系统在各种互联网服务中被广泛使用,但现有的Key-Value存储系统通常在用户态空间设计和实现,因为频繁的模式切换和上下文切换,导致访问接口、事务处理效率不高,在高并发、低延迟的数据存储需求中尤为突出.针对该问题,给出了一个内核态Key-Value存储系统的实现——KStore:提供内核空间的索引和内存分配机制,并在此基础上,通过基于内核Socket的远程接口以及基于文件系统的本地接口,保证了KStore的低延迟;同时,通过基于内核多线程的并发处理机制,保证了KStore的并发性.实验结果表明,与Memcached相比,KStore在实时性和并发性方面都取得显著优势.

关键词: Key-Value存储系统, Linux内核, 文件系统, 内核Socket, 内核线程, Slab内存分配

Abstract:

Nowadays Key-Value store system is widely used in various Internet services. However, the existing Key-Value store systems, mostly run in user-mode, can not meet the demands of high-concurrency and low-latency. It is mainly because user-mode usually provides inefficient access interfaces and transaction processing due to mode switch or context switch. To solve these problems, an in-kernel implementation of Key-Value store system, called KStore, was proposed in this paper. It had an in-kernel index and an in-kernel memory allocator, which were used to manage Key-Value data efficiently. To guarantee the low-latency response, KStore provided a remote interface based on in-kernel Socket, and a local interface based on file system. In addition, KStore processed concurrent requests with a novel mechanism based on in-kernel multi-thread. The experimental results show that KStore gains a remarkable advantage over Memcached in the characteristics of real-time and concurrency.

Key words: Key-Value store system, Linux kernel, file system, kernel Socket, kernel thread, Slab memory allocation

中图分类号:

TP311.52

谢沛东, 武延军. 基于Linux内核的Key-Value存储系统——KStore[J]. 计算机应用, 2015, 35(1): 99-102.

XIE Peidong, WU Yanjun. KStore: Linux kernel-based Key-Value store system[J]. Journal of Computer Applications, 2015, 35(1): 99-102.

参考文献

[1] SADALAGE P J, FOWLER M. NoSQL distilled: a brief guide to the emerging world of polyglot persistence [M]. Boston: Addison Wesley, 2012: 31-33.
[2] NISHATALA R, FUGAL H, GRIMM S, et al. Scaling Memcache at Facebook [C]// NSDI'13: Proceedings of the 10th USENIX Conference on Networked Systems Design and Implementation. Berkeley: USENIX Association, 2013: 385-398.
[3] ZHANG S, HAN J, LIU Z, et al. Accelerating MapReduce with distributed memory cache [C]// Proceedings of the 15th International Conference on Parallel and Distributed Systems. Piscataway: IEEE, 2009: 472-478.
[4] DORMANDO. Memcached [EB/OL]. [2014-06-20]. http://memcached.org/.
[5] VMWARE. Redis [EB/OL]. [2014-06-10]. http://redis.io/.
[6] MICHAEL M M. High performance dynamic lock-free hash tables and list-based sets [C]// Proceedings of the 14th ACM Symposium on Parallel Algorithms and Architectures. New York: ACM, 2002: 73-82.
[7] ANDREWS B. In-Kernel Berkeley DB [EB/OL]. [2014-06-15]. http://www.fsl.cs.sunysb.edu/project-kbdb.html.
[8] KLEIMAN S R. Vnodes: an architecture for multiple file system types in Sun UNIX [C]// Proceedings of the 1986 Summer USENIX Conference. Berkeley: USENIX Association, 1986: 238-247.
[9] LOVE R. Kernel korner: CPU affinity [J]. Linux Journal, 2003, 2003(111): 8.
[10] PUGH W. Skip lists: a probabilistic alternative to balanced trees [J]. Communications of the ACM, 1990, 33(6): 668-676.
[11] BONWICK J. The slab allocator: an object-caching kernel memory allocator [C]// USTC'94: Proceedings of the USENIX Summer 1994 Technical Conference on USENIX Summer 1994 Technical Conference. Berkeley: USENIX Association, 1994: 6-6.
[12] WIGGINS A, LAGNSTON J. Enhancing the scalability of Memcached [EB/OL]. [2014-06-05]. https://software.intel.com/en-us/articles/enhancing-the-scalability-of-memcached-0.

[1]	李旭, 何玉林, 崔来中, 黄哲学, PHILIPPE Fournier‑Viger. 基于大数据随机样本划分的分布式观测点分类器[J]. 《计算机应用》唯一官方网站, 2024, 44(6): 1727-1733.
[2]	陈美宏, 袁凌云, 夏桐. 基于主从多链的数据分类分级访问控制模型[J]. 《计算机应用》唯一官方网站, 2024, 44(4): 1148-1157.
[3]	倪瑞轩, 蔡淼, 叶保留. 内存高效的持久性分布式文件系统客户端缓存DFS-Cache[J]. 《计算机应用》唯一官方网站, 2024, 44(4): 1172-1180.
[4]	杜晓玉, 刘帅起, 韩志杰, 霍振祥, 王玉璟. 以患者为中心基于IPFS和区块链的医疗信息共享方案[J]. 《计算机应用》唯一官方网站, 2024, 44(10): 3122-3133.
[5]	张国潮, 唐华云, 陈建海, 沈睿, 何钦铭, 黄步添. 基于区块链的数字音乐版权管理系统[J]. 计算机应用, 2021, 41(4): 945-955.
[6]	苟子安, 张晓, 吴东南, 王艳秋. 分布式存储系统中的日志分析与负载特征提取[J]. 计算机应用, 2020, 40(9): 2586-2593.
[7]	董豪宇, 陈康. 纯用户态的网络文件系统——RUFS[J]. 计算机应用, 2020, 40(9): 2577-2585.
[8]	刘福鑫, 李劲巍, 王熠弘, 李琳. 基于Kubernetes的云原生海量数据存储系统设计与实现[J]. 《计算机应用》唯一官方网站, 2020, 40(2): 547-552.
[9]	贺庆建, 蔡涛, 王杰, 牛德姣. 面向NVM存储系统的快速文件访问系统[J]. 《计算机应用》唯一官方网站, 2020, 40(2): 541-546.
[10]	董聪, 张晓, 程文迪, 石佳. 基于新型存储器件的分布式文件系统性能优化[J]. 计算机应用, 2020, 40(12): 3594-3603.
[11]	王可可, 陈志德, 徐健. 基于联盟区块链的农产品质量安全高效追溯体系[J]. 计算机应用, 2019, 39(8): 2438-2443.
[12]	陈博, 何连跃, 严巍巍, 徐照淼, 徐俊. 海量小文件系统的可移植操作系统接口兼容技术[J]. 计算机应用, 2018, 38(5): 1389-1392.
[13]	李强, 刘晓峰. 基于Hopfield神经网络的云存储负载均衡策略[J]. 计算机应用, 2017, 37(8): 2214-2217.
[14]	杨顺, 陈志广, 肖侬. Pmfs中目录项索引的实现[J]. 计算机应用, 2017, 37(5): 1241-1245.
[15]	杨俊杰, 廖卓凡, 冯超超. 大数据存储架构和算法研究综述[J]. 计算机应用, 2016, 36(9): 2465-2471.

基于Linux内核的Key-Value存储系统——KStore

KStore: Linux kernel-based Key-Value store system

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics