Distributed storage solution based on parity coding

doi:10.3724/SP.J.1087.2013.00211

Journal of Computer Applications ›› 2013, Vol. 33 ›› Issue (01): 211-214.DOI: 10.3724/SP.J.1087.2013.00211

• Network and distributed techno • Previous Articles Next Articles

Distributed storage solution based on parity coding

CHEN Dongxiao¹,²,WANG Peng¹,³

1. Chengdu Institute of Computer Applications, Chinese Academy of Sciences, Chengdu Sichuan 610041, China
2. Graduate University of Chinese Academy of Sciences, Beijing 100190, China
3. Parallel Computing Laboratory, Chengdu University of Information Technology, Chengdu Sichuan 610025, China

Received:2012-07-09 Revised:2012-08-07 Online:2013-01-01 Published:2013-01-09
Contact: WANG Peng
Supported by:
the National Nature Science Fundation of China under Grant

基于校验编码备份的分布存储方案

陈冬晓¹,²,王鹏¹,³

1. 中国科学院成都计算机应用研究所，成都 610041
2. 中国科学院研究生院，北京 100049
3. 成都信息工程学院并行计算实验室，成都 610225

通讯作者: 王鹏
作者简介:陈冬晓(1986-)，男，四川德阳人，硕士研究生，主要研究方向：并行计算、云计算；王鹏(1975-)，男，四川乐山人，教授，博士生导师，博士，CCF高级会员，主要研究方向：云计算、并行计算。
基金资助:
国家自然科学基金资助项目(60872064);成都市科技局创新发展战略研究项目(软科学)(11RKYB016ZF)

Abstract

Abstract: To guarantee reliability, traditional cloud storage solutions generally backup data through mirror redundancy, which influences the usage efficiency of storage data space. A storage solution was proposed to reduce the usage of storage data space for redundancy-backup data. The solution introduced: 1) the parity coding backup instead of mirror backup, which reduced the size of backup data; 2) the conflict-jump mechanism to confirm the backup data, which guaranteed reliability while number of backup data copies was reduced. The contrast between running result of simulation program and performance of mainstream cloud storage solutions shows that, by using the proposed solution, the usage of storage space for distributed storage is significantly reduced while the reliability gets guaranteed.

Key words: cloud storage, consistent hash, Hadoop Distributed File System (HDFS), data backup, data recovery

摘要： 传统的云计算存储系统为保障可用性，一般使用镜像冗余备份而产生大量冗余备份数据，影响了存储数据空间的利用效率。针对此情况，为减少备份数据对存储空间的占用，提出一种存储方案。放弃了镜像冗余备份，引入校验编码的方式进行备份，以减少备份数据；同时采用了冲突跳转的机制对备份进行验证，在保证备份数据有效性的前提下减少备份数量。通过模拟程序运行结果与主流云存储方案的对比表明，所提存储方案在保证数据可靠性的同时，显著地降低了分布存储对磁盘空间的占用。

关键词: 云存储, 一致性哈希, Hadoop分布式文件系统, 数据备份, 数据恢复

CLC Number:

TP301.6

CHEN Dongxiao WANG Peng. Distributed storage solution based on parity coding[J]. Journal of Computer Applications, 2013, 33(01): 211-214.

陈冬晓王鹏. 基于校验编码备份的分布存储方案[J]. 计算机应用, 2013, 33(01): 211-214.

References

［1］Storage Networking Industry Association (SNIA). Cloud storage reference model: Version 0.3 rev ［EB/OL］. ［2012-03-15］. http://www.snia.org/sites/default/files/CloudStorageReferenceModelV03.pdf.

［2］CALLAGHAN B. NFS over RDMA ［EB/OL］. ［2012-04-05］. http://static.usenix.org/events/fast02/wips/callagcal.pdf.

［3］BORTHAKUR D. HDFS architecture guide ［EB/OL］. ［2012-03-18］. http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.pdf.

［4］SHVACHKO K, KUANG H, RADIA S, et al. The Hadoop distributed file system ［C］// Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies. Washington, DC: IEEE Computer Society, 2010: 1-10.

［5］许春玲,张广泉.分布式文件系统Hadoop HDFS与传统文件系统Linux FS的比较与分析［J］.苏州大学学报:工科版,2010,30(4):5-19.

［6］WHITE T. HDFS reliability ［EB/OL］. ［2012-04-20］. http://www.cloudera.com/wp-content/uploads/2010/03/HDFS_Reliability.pdf.

［7］DECANDIA G, HASTORUN D, JAMPANI M, et al. Dynamo: Amazon's highly available key-value store ［C］// SOSP'07: Proceedings of the 21st ACM Symposium on Operating Systems Principles. New York: ACM Press, 2007: 205-220.

［8］Amazon.com Inc. Amazon simple storage service (Amazon S3) ［EB/OL］. ［2012-03-15］. http://aws.amazon.com/s3.

［9］LEWIN D. Consistent hashing and random trees: algorithms for caching in distributed networks ［D］. Cambridge, Massachusetts: Massachusetts Institute of Technology, 1998.

［10］PAI V S, ARON M, BANGA G, et al. Locality-aware request distribution in cluster-based network servers ［C］// Proceedings of the 8th International Conference on Architectural Support for Programming Languages and Operating Systems. New York: ACM Press,1998:205-216.

［11］中国移动通信集团设计院有限公司.综合布线系统工程设计规范, GB50311—2007 ［S/OL］. ［2012-04-22］. http://www.sc-cic.org.cn/zywyh/GB50311.htm.

［12］KUROSE J F, ROSS K W. Computer networking: a top-down approach ［M］. 4th ed.北京:高等教育出版社,2010.

［13］周敬利,周正达.改进的云存储系统数据分布策略［J］.计算机应用,2012,32(2):309-312.

[1]	LI Xiuyan, LIU Mingxi, SHI Wenbo, DONG Guofang. Efficient dynamic data audit scheme for resource-constrained users [J]. Journal of Computer Applications, 2021, 41(2): 422-432.
[2]	TANG Xin, ZHOU Linna. Response obfuscation based secure deduplication method for cloud data with resistance against appending chunk attack [J]. Journal of Computer Applications, 2020, 40(4): 1085-1090.
[3]	FU Wei, GU Chenyang, GAO Qiang. Multi-user sharing ORAM scheme based on attribute encryption [J]. Journal of Computer Applications, 2020, 40(2): 497-502.
[4]	DONG Cong, ZHANG Xiao, CHENG Wendi, SHI Jia. Performance optimization of distributed file system based on new type storage devices [J]. Journal of Computer Applications, 2020, 40(12): 3594-3603.
[5]	ZHANG Hang, LIU Shanzheng, TANG Dan, CAI Hongliang. Erasure code with low recovery-overhead in distributed storage systems [J]. Journal of Computer Applications, 2020, 40(10): 2942-2950.
[6]	WANG Haiyong, PENG Yao, GUO Kaixuan. CP-ABE access control scheme based on proxy re-encryption in cloud storage [J]. Journal of Computer Applications, 2019, 39(9): 2611-2616.
[7]	LIU Yudong, WANG Xu'an, TU Guangsheng, WANG Han. Cloud outsourcing data secure auditing protocol throughout whole lifecycle [J]. Journal of Computer Applications, 2019, 39(7): 1954-1958.
[8]	ZHOU Jian, JIN Yu, HE Heng, LI Peng. Dynamic cloud data audit model based on nest Merkle Hash tree block chain [J]. Journal of Computer Applications, 2019, 39(12): 3575-3583.
[9]	MING Yang, HE Baokang. Attribute revocation and verifiable outsourcing supported multi-authority attribute-based encryption scheme [J]. Journal of Computer Applications, 2019, 39(12): 3556-3562.
[10]	LI Jing, LIU Dongshi. Reliability evaluation model for cloud storage systems with proactive fault tolerance [J]. Journal of Computer Applications, 2018, 38(9): 2631-2636.
[11]	CHEN Bo, HE Lianyue, YAN Weiwei, XU Zhaomiao, XU Jun. Portable operating system interface of UNIX compatibility technology in mass small distributed file system [J]. Journal of Computer Applications, 2018, 38(5): 1389-1392.
[12]	YANG Hongyu, WANG Yue. Multi-keyword ciphertext search method in cloud storage environment [J]. Journal of Computer Applications, 2018, 38(2): 343-347.
[13]	WANG Minshen, XIONG Jinbo, LIN Qian, WANG Lili. Cloud data assured deletion scheme based on key distribution and ciphertext sampling [J]. Journal of Computer Applications, 2018, 38(1): 194-200.
[14]	LI Qiang, LIU Xiaofeng. Load balancing strategy of cloud storage based on Hopfield neural network [J]. Journal of Computer Applications, 2017, 37(8): 2214-2217.
[15]	LI Haoyu, ZHANG Longjun, LI Qingpeng. Dynamic provable data possession scheme based on B⁺ tree [J]. Journal of Computer Applications, 2017, 37(7): 1931-1935.

Distributed storage solution based on parity coding

基于校验编码备份的分布存储方案

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics