Distributed data storage method based on consistent tree distribution

Journal of Computer Applications ›› 2013, Vol. 33 ›› Issue (12): 3432-3436.

• 2013 National Open Distributed and Parallel Computing Confernece of China (DPCS2013) • Previous Articles Next Articles

Distributed data storage method based on consistent tree distribution

GUO Dong¹,²,WANG Wei¹,²,ZENG Guoshun¹,²

1. Department of Computer Science and Technology, Tongji University, Shanghai 200092, China
2. Tongji Branch, National Engineering and Technology Center of High Performance, Shanghai 200092, China

Received:2013-07-29 Online:2013-12-31 Published:2013-12-01
Contact: GUO Dong

基于一致性树分布的数据分布式存储方法

郭栋¹,²,³,王伟¹,²,³,曾国荪¹,²,³

1. 国家高性能计算机工程技术研究中心同济大学分中心，上海 200092
2. 同济大学计算机科学与技术系，上海 200092；
3. 国家高性能计算机工程技术研究中心同济大学分中心，上海 200092

通讯作者: 郭栋
作者简介:郭栋(1991-),男,上海人,主要研究方向:分布式系统、云计算;
王伟(1976-),男,上海人,副教授,博士,CCF会员,主要研究方向:并行计算、分布式系统;
曾国荪(1964-),男,上海人,教授,博士生导师,CCF会员,主要研究方向:计算机软件、分布式系统、信息安全。
基金资助:
国家自然科学基金资助项目;上海市优秀学科带头人计划项目;教育部网络时代的科技论文快速共享专项研究课题资助项目;教育部博士点基金资助项目;浙江大学CAD&CG国家重点实验室开放课题资助项目;南京大学计算机软件新技术国家重点实验室开放课题资助项目;同济大学中央高校基本科研业务费专项资金资助项目;中国科学院模式识别国家重点实验室开放课题资助项目

Abstract

Abstract: With the development of cloud computing and big data, traditional single storage medium based data storage cannot meet the demands of large data processing, thus distributed data storage has been widely used recently. However, some existing distributed ways of storage cannot perfectly meet the needs of the distribution system. In order to achieve the distributed data storage and redundancy more effectively, a new distributed method and backup strategy based on Consistent Tree Distribution (CTD) was proposed to achieve the location mapping of data indexing and storage. The new method featured load balancing, no single point of failure, high scalability and easy programming. In addition, a method based on Consistent Binary Tree Distribution (CBTD) was also put forward. Analysis of the application system verifies that the method can satisfy the data balance, load balance and high scalability requirements of the distributed system.

Key words: distributed system, distributed storage, Consistent Tree Distribution (CTD), Consistent Binary Tree Distribution (CBTD)

摘要： 随着云计算和大数据技术的发展，传统的单一存储介质的数据存储方式已经不能满足大数据处理的需求，在这样的背景下，分布式数据存储得到了广泛的应用。然而，目前存在的几种分布式存储方式并不能够完美地满足分布系统的需求。为了更有效地实现数据的分布式存储和冗余备份，采用一种新的基于一致性树分布(CTD)的分布式存储方法，并提出基于该方法的备份策略，实现数据索引与存储位置的映射。该方案具有负载平衡、无单点故障问题、扩展性高且易于实现的优点。同时提出了基于一致性二叉树分布(CBTD)的应用方案。通过对应用系统实例的分析，验证该方法能够很好地满足分布式系统的数据平衡、负载均衡和扩展性需求。

关键词: 分布式系统, 分布式存储, 一致性树分布, 一致性二叉树分布

CLC Number:

TP311.12

GUO Dong WANG Wei ZENG Guoshun. Distributed data storage method based on consistent tree distribution[J]. Journal of Computer Applications, 2013, 33(12): 3432-3436.

郭栋王伟曾国荪. 基于一致性树分布的数据分布式存储方法[J]. 计算机应用, 2013, 33(12): 3432-3436.

[1]	Li YANG, Jianting CHEN, Yang XIANG. Performance optimization strategy of distributed storage for industrial time series big data based on HBase [J]. Journal of Computer Applications, 2023, 43(3): 759-766.
[2]	Yunbo LONG, Dan TANG. Load balancing method based on local repair code in distributed storage [J]. Journal of Computer Applications, 2023, 43(3): 767-775.
[3]	Na ZHOU, Ming CHENG, Menglin JIA, Yang YANG. Medical image privacy protection based on thumbnail encryption and distributed storage [J]. Journal of Computer Applications, 2023, 43(10): 3149-3155.
[4]	QING Xinyi, CHEN Yuling, ZHOU Zhengqiang, TU Yuanchao, LI Tao. Blockchain storage expansion model based on Chinese remainder theorem [J]. Journal of Computer Applications, 2021, 41(7): 1977-1982.
[5]	WU Zhijun, WANG Hang. System wide information management emergency response mechanism based on subscribe/publish service [J]. Journal of Computer Applications, 2020, 40(5): 1340-1347.
[6]	ZHANG Hang, LIU Shanzheng, TANG Dan, CAI Hongliang. Erasure code with low recovery-overhead in distributed storage systems [J]. Journal of Computer Applications, 2020, 40(10): 2942-2950.
[7]	ZHANG Guochao, WANG Ruijin. Blockchain shard storage model based on threshold secret sharing [J]. Journal of Computer Applications, 2019, 39(9): 2617-2622.
[8]	ZHOU Yue, CHEN Qingkui. Load balancing mechanism for large-scale data access system [J]. Journal of Computer Applications, 2018, 38(1): 50-55.
[9]	ZHOU Weibo, ZHONG Yong, LI Zhendong. Storage load balancing algorithm based on storage entropy [J]. Journal of Computer Applications, 2017, 37(8): 2209-2213.
[10]	XU Zhiqiang, YUAN Dezhai, CHEN Liang. Regenerating codes construction method based on sparse random matrix [J]. Journal of Computer Applications, 2017, 37(7): 1948-1952.
[11]	WU Jiagao, XIA Xuan, LIU Linfeng. Parallel trajectory compression method based on MapReduce [J]. Journal of Computer Applications, 2017, 37(5): 1282-1286.
[12]	XIAO Shenglong, CHEN Xin, LI Zhuo. Distributed neural network for classification of attack behavior to social security events [J]. Journal of Computer Applications, 2017, 37(10): 2794-2798.
[13]	QIU Gui, YAN Renwu. Distributed collaborative filtering recommendation algorithm based on gray association analysis [J]. Journal of Computer Applications, 2016, 36(4): 1054-1059.
[14]	LIU Qing, FU Yinjin, NI Guiqiang, MEI Jianmin. Distributed deduplication storage system based on Hadoop platform [J]. Journal of Computer Applications, 2016, 36(2): 330-335.
[15]	WENG Haixing, GONG Xueqing, ZHU Yanchao, HU Hualiang. Implementation of distributed index in cluster environment [J]. Journal of Computer Applications, 2016, 36(1): 1-7.

Distributed data storage method based on consistent tree distribution

基于一致性树分布的数据分布式存储方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics