High performance key-value storage system based on remote direct memory access

doi:10.11772/j.issn.1001-9081.2019091635

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (2): 316-320.DOI: 10.11772/j.issn.1001-9081.2019091635

• DPCS 2019 • Previous Articles Next Articles

High performance key-value storage system based on remote direct memory access

Cheng WANG¹, Baoliu YE¹(), Feng MEI², Wenda LU²

^1.State Key Laboratory for Novel Software Technology （Nanjing University），Nanjing Jiangsu 210023，China
^2.State Grid Zhejiang Electric Power Company Limited，Hangzhou Zhejiang 310007，China

Received:2019-07-31 Revised:2019-09-25 Accepted:2019-09-29 Online:2019-11-18 Published:2020-02-10
Contact: Baoliu YE
About author:WANG Cheng， born in 1994， M. S. candidate. His research interests include high-performance network.
MEI Feng， born in 1977， M. S.， senior engineer. His research interests include power information system， big data.
LU Wenda， born in 1989， M. S.， assistant engineer. His research interests include data mining， cloud computing.
Supported by:
National Key Technology Research and Development Program(2018YFB1004704);the National Natural Science Foundation of China(61832005);the Science and Technology Project of State Grid Corporation of China(52110418001M)

基于远程直接内存访问的高性能键值存储系统

王成¹, 叶保留¹(), 梅峰², 卢文达²

^1.计算机软件新技术国家重点实验室（南京大学），南京 210023
^2.国网浙江省电力有限公司，杭州 310007

通讯作者: 叶保留
作者简介:王成（1994—），男，江苏泗洪人，硕士研究生，主要研究方向：高性能网络
梅峰（1977—），男，浙江湖州人，高级工程师，硕士，主要研究方向：电力信息系统、大数据
卢文达（1989—），男，吉林松原人，助理工程师，硕士，主要研究方向：数据挖掘、云计算。
基金资助:
国家重点研发计划项目(2018YFB1004704);国家自然科学基金资助项目(61832005);国家电网公司科技项目(52110418001M)

Abstract

Abstract:

With the continuous increment of data and system size， network communication becomes a performance bottleneck of key-value storage systems. Meanwhile， Remote Direct Memory Access （RDMA） technique can support high bandwidth， low latency data transmission， which provides a new idea for designing key-value storage systems. Based on RDMA technique in the high performance network， a key-value storage system named Chequer with high performance and low CPU overhead was designed and implemented. By combining the characteristics of RDMA primitives， the basic operation workflow of key-value storage system was redesigned. And a linear probing based shared hash table was designed to reduce the number of client reading rounds by solving the problem of client cache invalidation as well as increasing the hash hit rate， which can further improve the performance of the system. The Chequer system was implemented on the small-scale cluster， and its performance was demonstrated by experiments.

Key words: Remote Direct Memory Access (RDMA), hash table, key-value storage, high performance network

摘要：

随着数据与系统规模的不断扩大，网络传输成为了键值存储系统的性能瓶颈。同时，远程直接内存访问（RDMA）技术能够支持高带宽和低时延的数据传输，为键值存储系统设计提供了新的思路。结合高性能网络中的RDMA技术，设计并实现了高性能、低CPU负载的键值存储系统Chequer；结合RDMA原语的特性，重新设计了键值存储系统的基本操作工作流程；并设计了基于线性探测的共享hash表，解决客户端缓存失效的问题以及提高hash命中率来减少客户端的读取轮数，进一步提高了系统的性能。在小规模集群上实现了Chequer系统，并通过实验验证了其性能。

关键词: 远程直接内存访问, 哈希表, 键值存储, 高性能网络

CLC Number:

TP311

Cheng WANG, Baoliu YE, Feng MEI, Wenda LU. High performance key-value storage system based on remote direct memory access[J]. Journal of Computer Applications, 2020, 40(2): 316-320.

王成, 叶保留, 梅峰, 卢文达. 基于远程直接内存访问的高性能键值存储系统[J]. 《计算机应用》唯一官方网站, 2020, 40(2): 316-320.

Figures/Tables 9

Fig. 1 Overall architecture of Chequer

Fig. 2 Module of memory pool caching

Fig. 3 Operation process of put

Fig. 4 Operation process of get

Fig. 5 Structure diagram of LRU

Fig. 6 Design of hash record structure

Tab.1 Configuration information of nodes in cluster

软硬件配置	参数
CPU	Intel Xeon CPU E5-2620 v2 @ 2.10 GHz
硬盘	2 TB
内存	64 GB
网卡	Mellanox ConnectX-5
操作系统	CentOS7.1
GCC Version	7.2.0
OFED Version	4.3-1.0.1

Fig. 7 Throughput of server under workload of 90% gets + 10% puts

Fig. 8 Throughput of server under workolad of 50% gets + 50% puts

References 14

1	RECIO R， METZLER B， CULLEY P， et al. A remote direct memory access protocol specification： RFC5040［S］. Fremont， CA： Internet Engineering Task Force， 2007-10. 10.17487/rfc5040
2	BUYYA R， CORTES T， JIN H. An introduction to the InfiniBand architecture［M］// High Performance Mass Storage and Parallel I/O： Technologies and Applications. Piscataway： IEEE， 2002： 616-632.
3	JIA C， LIU J， JIN X， et al. Improving the performance of distributed TensorFlow with RDMA［J］. International Journal of Parallel Programming， 2018， 46（4）：674-685. 10.1007/s10766-017-0520-3
4	LI M， WEN K， LIN H， et al. Improving the performance of distributed MXNet with RDMA［J］. International Journal of Parallel Programming， 2019， 47（3）：467-480. 10.1007/s10766-018-00623-w
5	MITCHELL C， GENG Y， LI J. Using one-sided RDMA reads to build a fast， CPU-efficient key-value store［C］// Proceedings of the 2013 USENIX Annual Technical Conference. Berkeley， CA： USENIX Association， 2013： 103-114.
6	DRAGOJEVIĆ A， NARAYANAN D， HODSON O， et al. FaRM： fast remote memory［C］// Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation. Berkeley， CA： USENIX Association， 2014： 401-414.
7	KALIA A， KAMINSKY M， ANDERSEN D G. Using RDMA efficiently for key-value services［J］. ACM SIGCOMM Computer Communication Review， 2015， 44（4）： 295-306. 10.1145/2740070.2626299
8	CASSELL B， SZEPESI T， WONG B， et al. Nessie： a decoupled， client-driven key-value store using RDMA［J］. IEEE Transactions on Parallel and Distributed Systems， 2017， 28（12）： 3537-3552. 10.1109/tpds.2017.2729545
9	CHEN W， YU S， WANG Z. Fast in-memory key-value cache system with RDMA［J］. Journal of Circuits， Systems and Computers， 2018， 28（5）： No.19500749-. 10.1142/s0218126619500749
10	王成. 基于RDMA的键值存储系统性能优化［D］. 南京：南京大学， 2019：56-70.
	WANG C. Performance optimization of key value storage system based on RDMA［D］. Nanjing： Nanjing University， 2019： 56-70.
11	ZHENG L， KURODA S I， LIU H， et al. Buddy algorithm optimization in Linux memory management［J］. Applied Mechanics and Materials， 2013， 423/426：2746-2750. 10.4028/www.scientific.net/amm.423-426.2746
12	COOPER B F， SILBERSTEIN A， TAM E， et al. Benchmarking cloud serving systems with YCSB［C］// Proceedings of the 1st ACM Symposium on Cloud Computing. New York： ACM， 2010： 143-154. 10.1145/1807128.1807152
13	CHEN H， ZHANG H， DONG M， et al. Efficient and available in-memory KV-store with hybrid erasure coding and replication［J］. ACM Transactions on Storage， 2017， 13（3）： No.25. 10.1145/3129900
14	YIU M M T， CHAN H H W， LEE P P C. Erasure coding for small objects in in-memory KV storage［C］// Proceedings of the 10th ACM International Systems and Storage Conference. New York： ACM， 2017： No.14. 10.1145/3078468.3078470

Metrics

Viewed

Full text

289

HTTP404 无法找到页面

http404错误

没有找到您要访问的页面，请检查您是否输入正确url。

请尝试以下操作：

·如果您已经在地址栏中输入该网页的地址，请确认其拼写正确。
·打开主页，然后查找指向您感兴趣信息的链接。
·单击后退链接，尝试其他链接。

HTTP404 无法找到页面

http404错误

没有找到您要访问的页面，请检查您是否输入正确url。

请尝试以下操作：

·如果您已经在地址栏中输入该网页的地址，请确认其拼写正确。
·打开主页，然后查找指向您感兴趣信息的链接。
·单击后退链接，尝试其他链接。

Abstract

300

HTTP404 无法找到页面

http404错误

没有找到您要访问的页面，请检查您是否输入正确url。

请尝试以下操作：

·如果您已经在地址栏中输入该网页的地址，请确认其拼写正确。
·打开主页，然后查找指向您感兴趣信息的链接。
·单击后退链接，尝试其他链接。

HTTP404 无法找到页面

http404错误

没有找到您要访问的页面，请检查您是否输入正确url。

请尝试以下操作：

·如果您已经在地址栏中输入该网页的地址，请确认其拼写正确。
·打开主页，然后查找指向您感兴趣信息的链接。
·单击后退链接，尝试其他链接。

[1]	Dong SUN, Biao WANG, Yun XU. Design and implementation of block transmission mechanism based on remote direct memory access [J]. Journal of Computer Applications, 2023, 43(2): 484-489.
[2]	DONG Haoyu, CHEN Kang. RUFS: a pure userspace network file system [J]. Journal of Computer Applications, 2020, 40(9): 2577-2585.
[3]	WANG Minshen, XIONG Jinbo, LIN Qian, WANG Lili. Cloud data assured deletion scheme based on key distribution and ciphertext sampling [J]. Journal of Computer Applications, 2018, 38(1): 194-200.
[4]	YANG Yanxia, FENG Lin. Data analysis method for parallel DHP based on Hadoop [J]. Journal of Computer Applications, 2016, 36(12): 3280-3284.
[5]	WANG Dong, XIONG Jinbo, ZHANG Xiaoying. Trust evaluation mechanism for distributed Hash table network nodes in cloud data secure self-destruction system [J]. Journal of Computer Applications, 2016, 36(10): 2715-2722.
[6]	PENG Chengzhang JIANG Zejun CAI Xiaobin ZHANG Zhike. Chord protocol and algorithm in distributed programming language [J]. Journal of Computer Applications, 2013, 33(07): 1885-1889.
[7]	Du Fei DONG Zhiguo MIAO Lin TUO YupengYupeng. Two-stage IPv6 route search algorithm based on perfect-Hash table and multibit-trie [J]. Journal of Computer Applications, 2013, 33(05): 1194-1202.
[8]	CHEN Xi ZENG Huashen WU Tao. Collaborative Web service composition based on distributed Hash table [J]. Journal of Computer Applications, 2013, 33(05): 1197-1202.
[9]	ZHONG Yan WANG Peng. Design and implementation of secret communication based on layered service provider [J]. Journal of Computer Applications, 2011, 31(12): 3340-3342.
[10]	FAN Xiao HUANG Qing-song. Message generation and store model based on HL7V2.x protocol [J]. Journal of Computer Applications, 2011, 31(12): 3418-3421.
[11]	Bo LIU Shi-ming FAN Hua LIU. Design and implementation of hybrid index mechanism for real-time database [J]. Journal of Computer Applications, 2011, 31(08): 2265-2269.
[12]	hua yanjiang. Decentralized approach for metadata management in computing resource sharing platform [J]. Journal of Computer Applications, 2011, 31(02): 462-465.
[13]	. Hash table based on Trie-tree [J]. Journal of Computer Applications, 2010, 30(8): 2193-2196.
[14]	. Paralleled packet classification based on network processor [J]. Journal of Computer Applications, 2010, 30(2): 306-308.
[15]	Yuan LI Shiping CHEN. An efficient RP2P network based on hierarchical dividing [J]. Journal of Computer Applications, 2009, 29(3): 646-648.

High performance key-value storage system based on remote direct memory access

基于远程直接内存访问的高性能键值存储系统

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 14

Related Articles 15

Recommended Articles 0

Metrics