基于HBase的海量地形数据存储

doi:10.11772/j.issn.1001-9081.2015.07.1849

计算机应用 ›› 2015, Vol. 35 ›› Issue (7): 1849-1853.DOI: 10.11772/j.issn.1001-9081.2015.07.1849

基于HBase的海量地形数据存储

李振举¹, 李学军¹, 谢剑薇¹, 李雁南²

1. 装备学院信息装备系, 北京 101416;
2. 96275部队, 河南洛阳 471003

收稿日期:2015-02-05 修回日期:2015-04-05 出版日期:2015-07-10 发布日期:2015-07-17
通讯作者: 李振举(1987-),男,河南安阳人,助理工程师,博士研究生,CCF会员,主要研究方向:云计算、遥感数据管理,belovings@163.com
作者简介:李学军(1967-),男,湖北监利人,教授,博士,CCF会员,主要研究方向:通信与信息系统、计算机图形学; 谢剑薇(1970-),女,安徽黄山人,副教授,硕士,主要研究方向:遥感数据管理; 李雁南(1988-),男,河南叶县人,助理工程师,硕士,主要研究方向:遥感数据管理。

Massive terrain data storage based on HBase

LI Zhenju¹, LI Xuejun¹, XIE Jianwei¹, LI Yannan²

1. Department of Information Equipment, Equipment Academy, Beijing 101416, China;
2. 96275 Troops, Luoyang Henan 471003, China

Received:2015-02-05 Revised:2015-04-05 Online:2015-07-10 Published:2015-07-17

摘要/Abstract

摘要：

随着遥感技术的发展,遥感数据的类型和量级发生了巨大变化,对于传统的存储方法产生了挑战。针对HBase中海量地形数据管理效率不高的问题,提出一种四叉树-Hilbert相结合的索引设计方法。首先,对传统地形数据管理方式和基于HBase的数据存储国内外研究现状进行了综述;然后,在基于四叉树对全球数据进行组织的基础上,提出了四叉树和Hilbert编码相结合的设计思想;其次,设计了根据经纬度求地形数据的行列号和根据行列号计算Hilbert编码的算法;最后,对设计的索引的物理存储结构进行了设计。实验结果表明,利用设计的索引进行海量地形数据入库,数据入库速度与单机情况相比,提高了63.79%~78.45%;在地形数据的范围查询中,设计的索引与传统的行序索引相比,查询时间降低了16.13%~39.68%。查询速度最低为14.71 MB/s,可以满足地形数据显示的要求。

关键词: HBase, 地形数据, 云存储, 四叉树-Hilbert索引, 三维地形显示

Abstract:

With the development of remote sensing technology, the data type and data volume of remote sensing data has increased dramatically in the past decades which is a challenge for traditional storage mode. A combination of quadtree and Hilbert spatial index was proposed in this paper to solve the the low storage efficiency in HBase data storage. Firstly, the research status of traditional terrain data storage and data storage based on HBase was reviewed. Secondly the design idea on the combination of quadtree and Hilbert spatial index based on managing global data was proposed. Thirdly the algorithm for calculating the row and column number based on the longitude and latitude of terrain data, and the algorithm for calculating the final Hilbert code was designed. Finally, the physical storage infrastructure for the index was designed. The experimental results illustrate that the data loading speed in Hadoop cluster improved 63.79%-78.45% compared to the single computer, the query time decreases by 16.13%-39.68% compared to the traditional row key index, the query speed is at least 14.71 MB/s which can meet the requirements of terrain data visualization.

Key words: HBase, terrain data, cloud storage, quadtree-Hilbert index, 3-dimensional terrain data visualization

中图分类号:

TP391.1

李振举, 李学军, 谢剑薇, 李雁南. 基于HBase的海量地形数据存储[J]. 计算机应用, 2015, 35(7): 1849-1853.

LI Zhenju, LI Xuejun, XIE Jianwei, LI Yannan. Massive terrain data storage based on HBase[J]. Journal of Computer Applications, 2015, 35(7): 1849-1853.

参考文献

[1] CHEN C L P, ZHANG C-Y. Data-intensive applications, challenges, techniques and technologies: a survey on big data [J]. Information Sciences, 2014, 275: 314-347.
[2] CHANG F D J, DEAN J, GHEMAWAT S, et al. Bigtable: a distributed storage system for structured data [J]. ACM Transactions on Computer Systems, 2008, 26(2): Article No. 4.
[3] O'DRISCOLL A, DAUGELAITE J, SLEATOR R D. 'Big data’, Hadoop and cloud computing in genomics [J]. Journal of Biomedical Informatics, 2013, 46(5): 774-781.
[4] SHANG X. Design and implementation of WebGIS massive tile data management engine [D]. Jinhua: Zhejiang Normal University, 2012.(商秀玉.WebGIS海量瓦片数据管理引擎的设计与实现[D].金华:浙江师范大学,2012.)
[5] YU W. Digital earth oriented key technologies for 3D landscape development [D]. Beijing: University of Chinese Academy of Sciences, 2006.(于文洋.面向数字地球的三维景观构造关键技术研究[D].北京:中国科学院研究生院,2006.)
[6] LUO Z, LI X. High performance tile map service based on database storage scheme [J]. Geography and Geo-Information Science, 2013, 29(3): 48-51.(罗智勇,黎小东.基于数据库存储方案的高性能瓦片地图服务研究[J].地理与地理信息科学,2013,29(3):48-51.)
[7] ZHAO S. Typical Hadoop cloud computing [M]. Beijing: Publishing House of Electronics Industry, 2013: 362.(赵书兰.典型Hadoop云计算[M].北京:电子工业出版社,2013:362.)
[8] CHEN H, LI Y, ZHU M. Research on tile buffer policy supporting large number of concurrent [J]. Computer Engineering & Science, 2012, 34(12): 144-149.(陈华,李艳明,朱美正.一种支持大量并发用户的瓦片缓存方案研究[J].计算机工程与科学,2012,34(12):144-149.)
[9] ZHU X, ZHAO Z. Massive image storage based on HBase [J]. China CIO News, 2013(8): 22-24.(朱晓丽,赵志刚.一种基于HBase的海量图片存储技术[J].信息系统工程,2013(8):22-24.)
[10] ZHOU L, CHEN Q. HBase-based storage system for wireless sensor information of agriculture [J]. Computer Systems & Applications, 2012, 21(8): 6-9.(周利珍,陈庆奎.基于HBase的农业无线传感信息系统[J].计算机系统应用,2012,21(8):6-9.)
[11] CHEN Q, ZHOU L. HBase-based storage system for large-scale data in wireless sensor network [J]. Journal of Computer Applications, 2012, 32(7): 1920-1923.(陈庆奎,周利珍.基于HBase的大规模无线传感网络数据存储系统[J].计算机应用,2012,32(7):1920-1923.)
[12] HUO S. Research on the key techniques of massive image data management based on Hadoop [D]. Changsha: National University of Defense Technology, 2010.(霍树民.基于Hadoop的海量遥感数据管理关键技术研究[D].长沙:国防科学技术大学,2010.)
[13] FAN J, LONG M, XIONG W. Research of vector spatial data distributed storage based on HBase [J]. Geography and Geo-Information Science, 2012, 28(5): 39-41.(范建永,龙明,熊伟.基于HBase的矢量空间数据分布式存储研究[J].地理与地理信息科学,2012,28(5):39-41.)
[14] PAPAILIOU N, TSOUMAKOS D, KONSTANTINOU I, et al. H2RDF+: an efficient data management system for big RDF graphs [C]// Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. New York: ACM, 2014: 909-912.
[15] HAN D, STROULIA E. HGrid: a data model for large geospatial data sets in HBase [C]// Proceedings of the 2013 IEEE Sixth International Conference on Cloud Computing. Piscataway: IEEE, 2013: 910-917.
[16] DeCANDIA G, HASTORUN D, JAMPANI M, et al. Dynamo: Amazon's highly available key-value store [C]// ACM SIGOPS Operating Systems Review, 2007, 41(6): 205-220.
[17] TANG G, ZHANG Q, WANG G. Research of storage and schedule of huge terrain based on 3D GIS [J]. Science of Surveying and Mapping, 2008, 33(3): 110-120.(唐桂文,张庆娟,王功明,等.基于三维GIS的海量地形数据存储和调度的研究[J].测绘科学,2008,33(3):110-120.)

基于HBase的海量地形数据存储

Massive terrain data storage based on HBase

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	崔双双, 王宏志. 基于日志结构合并树的轻量级分布式索引实现方法[J]. 计算机应用, 2021, 41(3): 630-635.
[2]	李秀艳, 刘明曦, 史闻博, 董国芳. 面向资源受限用户的高效动态数据审计方案[J]. 计算机应用, 2021, 41(2): 422-432.
[3]	唐鑫, 周琳娜. 基于响应模糊化的抗附加块攻击云数据安全去重方法[J]. 计算机应用, 2020, 40(4): 1085-1090.
[4]	徐江峰, 谭玉龙. 基于HBase的多维索引查询机制的优化[J]. 计算机应用, 2020, 40(2): 571-577.
[5]	付伟, 顾晨阳, 高强. 基于属性加密的多用户共享ORAM方案[J]. 计算机应用, 2020, 40(2): 497-502.
[6]	王海勇, 彭垚, 郭凯璇. 云存储中基于代理重加密的CP-ABE访问控制方案[J]. 计算机应用, 2019, 39(9): 2611-2616.
[7]	柳玉东, 王绪安, 涂广升, 王涵. 全生命周期的云外包数据安全审计协议[J]. 计算机应用, 2019, 39(7): 1954-1958.
[8]	明洋, 何宝康. 支持属性撤销的可验证外包的多授权属性加密方案[J]. 计算机应用, 2019, 39(12): 3556-3562.
[9]	周坚, 金瑜, 何亨, 李鹏. 基于嵌套Merkle Hash tree区块链的云数据动态审计模型[J]. 计算机应用, 2019, 39(12): 3575-3583.
[10]	李静, 刘冬实. 主动容错云存储系统的可靠性评价模型[J]. 计算机应用, 2018, 38(9): 2631-2636.
[11]	冯钧, 李顶圣, 陆佳民, 张立霞. 基于HBase的路网移动对象时空索引方法[J]. 计算机应用, 2018, 38(6): 1575-1583.
[12]	崔晨, 郑林江, 韩凤萍, 何牧君. 基于内存的HBase二级索引设计[J]. 计算机应用, 2018, 38(6): 1584-1590.
[13]	陈博, 何连跃, 严巍巍, 徐照淼, 徐俊. 海量小文件系统的可移植操作系统接口兼容技术[J]. 计算机应用, 2018, 38(5): 1389-1392.
[14]	吴仁彪, 刘超, 屈景怡. 基于HBase和Hive的航班延误平台的存储方法[J]. 计算机应用, 2018, 38(5): 1339-1345.
[15]	杨宏宇, 王玥. 云存储环境下的多关键字密文搜索方法[J]. 计算机应用, 2018, 38(2): 343-347.