Metadata management mechanism of massive spatial data storage

doi:10.11772/j.issn.1001-9081.2015.05.1276

Journal of Computer Applications ›› 2015, Vol. 35 ›› Issue (5): 1276-1279.DOI: 10.11772/j.issn.1001-9081.2015.05.1276

Previous Articles Next Articles

Metadata management mechanism of massive spatial data storage

YANG Wenhui¹, LI Guoqiang¹, MIAO Fang^1,2,3

1. Key Laboratory of Earth Exploration and Information Techniques of Ministry of Education (Chengdu University of Technology), Chengdu Sichuan 610059, China;
2. State Key Laboratory of Geohazard Prevention and Geoenvironment Protection (Chengdu University of Technology), Chengdu Sichuan 610059, China;
3. Key Laboratory of Pattern Recognition and Intelligent Information Processing of University of Sichuan Province (Chengdu University), Chengdu Sichuan 610106, China

Received:2014-12-11 Revised:2015-01-17 Online:2015-05-10 Published:2015-05-14

面向海量空间数据存储的元数据管理机制

杨文晖¹, 李国强¹, 苗放^1,2,3

1. 地球探测与信息技术教育部重点实验室(成都理工大学), 成都 610059;
2. 地质灾害防治与地质环境保护国家重点实验室(成都理工大学), 成都 610059;
3. 模式识别与智能信息处理四川省高校重点实验室(成都大学), 成都 610106

通讯作者: 李国强
作者简介:杨文晖(1969-),女,上海人,副教授,主要研究方向:虚拟现实与仿真、分布式并行计算; 李国强(1990-),男,四川泸州人,硕士研究生,主要研究方向:分布式存储与计算、空间信息技术; 苗放(1958-),男,北京人,教授,博士,主要研究方向:计算机网络、空间信息技术.
基金资助:
国家自然科学基金资助项目(61071121);成都市经信委科技专项(201102153).

Abstract

Abstract:

In order to manage the metadata of massive spatial data storage effectively, a distributed metadata server management structure based on consistent hashing was introduced, and on this basis, a metadata wheeled backup strategy was proposed in this paper, which stored Hash metadata node after excuting a consistent Hash algorithm according to the method of data backup, and it effectively alleviated the single point of metadata management and access bottleneck problems. Finally testing wheel backup strategy, it obtained the optimum number of metadata node backup solution. Compared with the single point of metadata servers, the proposed strategy improves the metadata safety, reduces the access delay, and improves the load balance of distributed metadata server combined with virtual nodes.

Key words: consistent hashing, metadata, distributed file system, wheeled backup strategy

摘要：

为了有效管理海量空间数据存储的元数据,引入了一种基于一致性哈希的分布式元数据服务器管理架构,并在此基础上提出了一种元数据轮式备份策略,将经过一致性哈希算法散列后存储元数据的节点按轮转方式进行数据备份,有效缓解了元数据管理的单点问题与访问瓶颈.最后对轮式备份策略进行测试,得出最佳元数据节点个数备份方案,与单点元数据服务器相比提高了元数据的安全性,降低了访问延迟,并结合虚拟节点改善了分布式元数据服务器的负载均衡.

关键词: 一致性哈希, 元数据, 分布式文件系统, 轮式备份策略

CLC Number:

TP301

YANG Wenhui, LI Guoqiang, MIAO Fang. Metadata management mechanism of massive spatial data storage[J]. Journal of Computer Applications, 2015, 35(5): 1276-1279.

杨文晖, 李国强, 苗放. 面向海量空间数据存储的元数据管理机制[J]. 计算机应用, 2015, 35(5): 1276-1279.

References

[1] LU J. Distributed system and introduction to cloud computing[M]. Beijing: Tsinghua University Press,2011:3-4.(陆嘉恒,分布式系统及云计算概论[M].北京:清华大学出版社,2011:3-4.)
[2] FEI X,WANG S. Storage and management of massive data in cloud computing[J].Computer Measurement and Control, 2014,22(7):2259-2261.(费贤举,王树锋. 基于云环境下的海量大数据存储系统设计[J].计算机测量与控制,2014,22(7):2259-2261.)
[3] CUI X. Huge amounts of spatial data distributed storage management, and parallel processing technology research[D]. Changsha: National University of Defense Technology,2010.(崔鑫.海量空间数据的分布式存储管理及并行处理技术研究[D].长沙: 国防科学技术大学,2010.)
[4] KARGER D,LEHMAN E,LEIGHTON T, et al. Consistent Hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web[C]// Proceedings of the 29th Annual ACM Symposium on Theory of Computing. New York: ACM, 1997:654-663.
[5] WEN A, WANG L. High availability HDFS: deep practice of Hadoop distributed file system[M]. Beijing: Tsinghua University Press, 2012:3-5.(文艾,王磊.高可用性的HDFS: Hadoop分布式文件系统深度实践[M].北京:清华大学出版社,2012:3-5.)
[6] WEI Q. The research on large-scale distributed storage technology[D]. Chengdu: University of Electronic Science and Technology of China,2004.(魏青松.大规模分布式存储技术研究[D].成都: 电子科技大学,2004.)
[7] XIE P,YU J. The metadata management strategy based on CWM[J]. Microcomputer and Its Applications, 2011,30(23):12-15.(谢培基,余金山.基于CWM 的元数据管理策略[J]. 微型机与应用,2011,30(23):12-15.)
[8] MNEG A. Study of metadata management mechanism in distributed file system[J].Computer Knowledge and Technology, 2011,7(35):9038-9040.(蒙安泰.分布式文件系统中元数据管理机制的研究[J].电脑知识与技术,2011,7(35):9038-9040.)
[9] WU W. Research on metadata management in large-seale storage systems[D].Wuhan: Huazhong University of Science and Technology,2010:8-10.(吴伟.海量存储系统元数据管理的研究[D].武汉: 华中科技大学,2010:8-10.)
[10] YAO M,XIE H. Consistency Hash algorithm in the application of the distributed system[J]. Computer Development and Applications, 2012(7):510.(姚墨涵,谢红薇.一致性哈希算法在分布式系统中的应用[J].电脑开发与应用, 2012(7):510.)
[11] FU D. Research and improvement of the massive distributed file system based on HDFS[D].Beijing: Beijing University of Posts and Telecommunications, 2012.(付东华.基于HDFS的海量分布式文件系统的研究与优化[D].北京:北京邮电大学,2012.)
[12] CHENG F,MIAO F,CHEN K. Self-adaptive metadata management model for distributed file system[J].Computer Engineering and Design,2014,35(3):867-868.(程付超,苗放,陈垦.自适应的分布式文件系统元数据管理模型[J].计算机工程与设计,2014,35(3):867-868.)

[1]	GOU Zi'an, ZHANG Xiao, WU Dongnan, WANG Yanqiu. Log analysis and workload characteristic extraction in distributed storage system [J]. Journal of Computer Applications, 2020, 40(9): 2586-2593.
[2]	DONG Cong, ZHANG Xiao, CHENG Wendi, SHI Jia. Performance optimization of distributed file system based on new type storage devices [J]. Journal of Computer Applications, 2020, 40(12): 3594-3603.
[3]	CHEN Bo, HE Lianyue, YAN Weiwei, XU Zhaomiao, XU Jun. Portable operating system interface of UNIX compatibility technology in mass small distributed file system [J]. Journal of Computer Applications, 2018, 38(5): 1389-1392.
[4]	LI Qiang, LIU Xiaofeng. Load balancing strategy of cloud storage based on Hopfield neural network [J]. Journal of Computer Applications, 2017, 37(8): 2214-2217.
[5]	YANG Junjie, LIAO Zhuofan, FENG Chaochao. Survey on big data storage framework and algorithm [J]. Journal of Computer Applications, 2016, 36(9): 2465-2471.
[6]	LIU Chao, HU Chengyu, YAO Hong, LIANG Qingzhong, YAN Xuesong. Non-relational data storage management mechanism for massive unstructured data [J]. Journal of Computer Applications, 2016, 36(3): 670-674.
[7]	LIU Qing, FU Yinjin, NI Guiqiang, MEI Jianmin. Distributed deduplication storage system based on Hadoop platform [J]. Journal of Computer Applications, 2016, 36(2): 330-335.
[8]	WU Jinbo, SONG Jie, ZHANG Li, BAO Yubin. Probery: probability-based data query system for big data [J]. Journal of Computer Applications, 2016, 36(1): 8-12.
[9]	SHAO Tian, CHEN Guangsheng, JING Weipeng. Cut-GAR: solution to determine cut-off point in cloud storage system [J]. Journal of Computer Applications, 2015, 35(9): 2497-2502.
[10]	ZHENG Kai, ZHU Lin, CHEN Youguang. Load balancing cloud storage algorithm based on Kademlia [J]. Journal of Computer Applications, 2015, 35(3): 643-647.
[11]	WANG Zhengying, YU Jiong, YING Changtian, LU Liang. Energy-efficient strategy of distributed file system based on data block clustering storage [J]. Journal of Computer Applications, 2015, 35(2): 378-382.
[12]	CHEN Jirong LE Jiajin. Programming model based on MapReduce for importing big table into HDFS [J]. Journal of Computer Applications, 2013, 33(09): 2486-2489.
[13]	ZHU Yuanyuan WANG Xiaojing. HDFS optimization program based on GE coding [J]. Journal of Computer Applications, 2013, 33(03): 730-733.
[14]	CHEN Dongxiao WANG Peng. Distributed storage solution based on parity coding [J]. Journal of Computer Applications, 2013, 33(01): 211-214.
[15]	LIU Lian ZHENG Biao GONG Yi-li. Metadata processing optimization in distributed file systems [J]. Journal of Computer Applications, 2012, 32(12): 3271-3273.

Metadata management mechanism of massive spatial data storage

面向海量空间数据存储的元数据管理机制

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics