基于HBase和Hive的航班延误平台的存储方法

doi:10.11772/j.issn.1001-9081.2017102475

计算机应用 ›› 2018, Vol. 38 ›› Issue (5): 1339-1345.DOI: 10.11772/j.issn.1001-9081.2017102475

基于HBase和Hive的航班延误平台的存储方法

吴仁彪, 刘超, 屈景怡

中国民航大学天津市智能信号与图像处理重点实验室, 天津 300300

收稿日期:2017-10-19 修回日期:2017-12-29 出版日期:2018-05-10 发布日期:2018-05-24
通讯作者: 屈景怡
作者简介:吴仁彪(1966-),男,湖北武汉人,教授,博士生导师,博士,主要研究方向:自适应信号处理、现代谱分析及其在雷达、卫星导航和空管中的应用;刘超(1991-),男,湖北黄冈人,硕士研究生,主要研究方向:航空运输大数据;屈景怡(1978-),女,天津人,副教授,博士,主要研究方向:航空运输大数据、神经网络。
基金资助:
国家自然科学基金资助项目（11402294）；天津市智能信号与图像处理重点实验室开放基金资助项目（2017ASP-TJ01）。

Storage method for flight delay platform based on HBase and Hive

WU Renbiao, LIU Chao, QU Jingyi

Tianjin Key Laboratory for Advanced Signal Processing, Civil Aviation University of China, Tianjin 300300, China

Received:2017-10-19 Revised:2017-12-29 Online:2018-05-10 Published:2018-05-24
Contact: 屈景怡
Supported by:
This work is partially supported by National Natural Science Foundation of China (11402294), the Tianjin Intelligent Signal and Image Processing Key Laboratory Open Fund Project (2017ASP-TJ01).

摘要/Abstract

摘要： 针对我国目前航班延误平台的移植难、可扩展性差，无法适应民航高速发展所带来的大数据量存储的现状，设计了面向大数据的跨平台、高适用性与高扩展性的航班延误平台。该平台以大数据工具LeafLet为可视化载体，在地图界面实时显示航班轨迹并将轨迹数据加载至HBase数据库中，并且利用信息摘要算法（MD5）重新设计与优化航班数据表的行键，以解决其递增的飞行时间特性产生的"热点"问题；针对HBase过滤器多级查询的缺陷，提出了基于SolrCloud的关联查询算法，利用SolrCloud实现对行键与索引字段的分层存储，从而实现HBase二级快速索引；最后在HBase的历史航班数据与飞行计划数据基础上，构建基于Hive的海量航班信息数据仓库。实验结果显示，航班延误大数据平台的可扩展性与搭建的航班信息数据仓库可以满足民航对数据集中统一存储的需求，而多条件查询的响应速度与无二级索引的集群相比提高了上百倍，并且这种优势随着航班数据量的增长愈发明显。

关键词: 大数据平台, 航班延误, HBase, Hive, SolrCloud, LeafLet

Abstract: In the view of the problem that the portability and expansibility current flight delay platform in China can not adapt to the status of large data storage brought by rapid development of Chinese civil aviation, a flight delay big data platform with cross platform, high availability and high expansion was designed. The platform used a big data tool LeafLet as a visual carrier, displayed the flight trajectory in the map interface, and loaded trajectory data to HBase database. Message-Digest Algorithm (MD5) algorithm was used to redesign and optimize the rowkey of flight data table to solve its "hot spot" problem brought by incremental flight time. Considering the shortcomings of multi-level query of HBase filter, a query algorithm based on SolrCloud was proposed, which utilized SolrCloud to realize hierarchical storage of row and index fields, so as to realize HBase two-level fast indexing. Finally, based on historical flight data and flight plan data of HBase, a massive flight information data warehouse based on Hive was constructed. The experimental results show that the expensibility of large data platform for flight delays and the construction of flight information data warehouse can meet the demand of civil aviation for unified storage of data, and the response speed of the multi-condition query is improved by hundreds of times compared with the cluster without second index, and this advantage becomes more and more obvious as the flight data amount grows.

Key words: big data platform, flight delay, HBase, Hive, SolrCloud, LeafLet

中图分类号:

TP311.1

吴仁彪, 刘超, 屈景怡. 基于HBase和Hive的航班延误平台的存储方法[J]. 计算机应用, 2018, 38(5): 1339-1345.

WU Renbiao, LIU Chao, QU Jingyi. Storage method for flight delay platform based on HBase and Hive[J]. Journal of Computer Applications, 2018, 38(5): 1339-1345.

参考文献

[1] 曹悦琪,贾奇.基于Logistic模型的大面积航班延误预测方法研究[J].交通信息与安全,2017,35(1):86-91.(CAO Y Q, JIA Q. A forecasting method for large-scale flight delays[J]. Journal of Transport Information and Safety, 2017, 35(1):86-91.)
[2] O'LEARY D E. Artificial intelligence and big data[J]. IEEE Intelligent Systems, 2013, 28(2):96-99.
[3] SALOUSTROS G, MAGOUTIS K. Rethinking HBase:design and implementation of an elastic key-value store over log-structured local volumes[C]//Proceedings of the 201514th International Symposium on Parallel and Distributed Computing. Washington, DC:IEEE Computer Society, 2015:225-234.
[4] 房俊,李冬,郭会云,等.面向海量交通数据的HBase时空索引[J].计算机应用,2017,37(2):311-315.(FANG J, LI D, GUO H Y, et al. Spatio-temporal index for massive traffic data based on HBase[J]. Journal of Computer Applications, 2017, 37(2):311-315.)
[5] LIU T, GAO S, CHU X, et al. Storing and querying AIS data in HBase[C]//Proceedings of the 20172nd Asia-Pacific Conference on Intelligent Robot Systems. Piscataway, NJ:IEEE, 2017:88-92.
[6] WANG Y T, CHIANG Y P, WU C H, et al. The implementation of sensor data access cloud service on HBase for intelligent indoor environmental monitoring[C]//Proceedings of the 201615th International Symposium on Parallel and Distributed Computing. Piscataway, NJ:IEEE, 2016:234-239.
[7] ZHANG C, GU P, FENG W. Transform biological data into HBase with MDA[C]//Proceedings of the 20154th International Conference on Computer Science and Network Technology. Piscataway, NJ:IEEE, 2015:474-478.
[8] 丁飞,陈长松,张涛,等.基于协处理器的HBase区域级第二索引研究与实现[J].计算机应用,2014, 34(S1):181-185.(DING F, CHEN C S, ZHANG T, et al. Research and implementation of coprocessor based HBase region-level secondary index[J]. Journal of Computer Applications, 2014, 34(S1):181-185.)
[9] CHANG B R, TSAI H F, WANG Y A, et al. Realization of secondary indexing to NoSQL database with intelligent adaptation[C]//Proceedings of the 2015 Conference on Technologies and Applications of Artificial Intelligence. Piscataway, NJ:IEEE, 2016:449-452.
[10] JIANG D, TUNG A K H, CHEN G. MAP-JOIN-REDUCE:toward scalable and efficient data analysis on large clusters[J]. IEEE Transactions on Knowledge & Data Engineering, 2011, 23(9):1299-1311.
[11] THUSOO A, SARMA J S, JAIN N, et al. Hive:a warehousing solution over a Map-Reduce framework[J]. Proceedings of the VLDB Endowment, 2009, 2(2):1626-1629.
[12] 游进国,奚建清,肖裕洪.基于PC集群的并行数据仓库架构[J].计算机工程,2009,35(20):73-75.(YOU J G, XI J Q, XIAO Y H. Parallel data warehouses architecture based on PC cluster[J]. Computer Engineering, 2009, 35(20):73-75.)
[13] 游进国,杨卓荦,胡建华,等.一种支持大规模数据的多维可视化分析框架[J].计算机工程,2011,37(19):26-27. (YOU J G, YANG Z L, HU J H, et al. Multi-dimensional visualized analysis framework for supporting large scale data[J]. Computer Engineering, 2011, 37(19):26-27.)
[14] 李春晖.基于Web的项目管理平台的设计与开发[D].上海:复旦大学, 2007:12-13. (LI C H. Design and development of project management platform based on Web[D]. Shanghai:Fudan University, 2007:12-13.)
[15] 梅江泽.基于Solr的海量数据并行索引及搜索缓存研究[D].武汉:华中师范大学, 2016:8-11.(MEI J Z. Research on parallel indexing and cache of searching with massive data based on Solr[D]. Wuhan:Central China Normal University, 2016:8-11.)
[16] 任云韬,李毅超,曹跃.基于注册表Hive文件的恶意程序隐藏检测方法[J].电子科技大学学报,2007,36(3):621-624.(REN Y T, LI Y C, CAO Y. A methodology to detect malware based on registry Hive files[J]. Journal of University of Electronic Science and Technology of China, 2007, 36(3):621-624.)
[17] JUNQUEIRA F, REED B. ZooKeeper:Distributed Process Coordination[M]. Sebastopol:O'Reilly Media, Inc. 2013:157-161.
[18] GEORGE L. HBase:the definitive guide[J]. Andre, 2011, 12(1):1-4.
[19] 俞子波.HBase分布式缓存策略的研究与设计[D].北京:北京交通大学, 2017:19-22.(YU Z B. Research and design of distributed caching policy on HBase[D]. Beijing:Beijing Jiaotong University, 2017:19-22.)
[20] 卓海艺.基于HBase的海量数据实时查询系统设计与实现[D].北京:北京邮电大学, 2013:38-43.(ZHUO H Y. The design and implementation of real-time query system for mass data based on HBase[D]. Beijing:Beijing University of Posts and Telecommunications, 2013:38-43.)
[21] 王小森.基于Solr的搜索引擎的设计与实现[D].北京:北京邮电大学,2010:15-18. (WANG X S. Design and implementation of Solr-based search engine[D]. Beijing:Beijing University of Posts and Telecommunications, 2010:15-18.)
[22] DERROUGH J. Instant interactive map designs with Leaflet JavaScript library how-to[EB/OL].[2017-06-20]. http://lcis.com.tw/paper_store/paper_store/Instant%20In-20141213212341828.pdf.

基于HBase和Hive的航班延误平台的存储方法

Storage method for flight delay platform based on HBase and Hive

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 14

编辑推荐

Metrics

[1]	崔双双, 王宏志. 基于日志结构合并树的轻量级分布式索引实现方法[J]. 计算机应用, 2021, 41(3): 630-635.
[2]	屈景怡, 曹磊, 陈敏, 董樑, 曹烨琇. 基于团簇随机连接的CliqueNet航班延误预测模型[J]. 计算机应用, 2020, 40(8): 2420-2427.
[3]	徐江峰, 谭玉龙. 基于HBase的多维索引查询机制的优化[J]. 计算机应用, 2020, 40(2): 571-577.
[4]	吴仁彪, 李佳怡, 屈景怡. 基于双通道卷积神经网络的航班延误预测模型[J]. 计算机应用, 2018, 38(7): 2100-2106.
[5]	冯钧, 李顶圣, 陆佳民, 张立霞. 基于HBase的路网移动对象时空索引方法[J]. 计算机应用, 2018, 38(6): 1575-1583.
[6]	崔晨, 郑林江, 韩凤萍, 何牧君. 基于内存的HBase二级索引设计[J]. 计算机应用, 2018, 38(6): 1584-1590.
[7]	顾军华, 武君艳, 许馨匀, 谢志坚, 张素琪. 基于Spark的并行FP-Growth算法优化及实现[J]. 计算机应用, 2018, 38(11): 3069-3074.
[8]	房俊, 李冬, 郭会云, 王嘉怡. 面向海量交通数据的HBase时空索引[J]. 计算机应用, 2017, 37(2): 311-315.
[9]	刘青, 付印金, 倪桂强, 梅建民. 基于Hadoop平台的分布式重删存储系统[J]. 计算机应用, 2016, 36(2): 330-335.
[10]	邳文君, 宫秀军. 基于Hadoop架构的数据驱动的SVM并行增量学习算法[J]. 计算机应用, 2016, 36(11): 3044-3049.
[11]	李振举, 李学军, 谢剑薇, 李雁南. 基于HBase的海量地形数据存储[J]. 计算机应用, 2015, 35(7): 1849-1853.
[12]	陆婷, 房俊, 乔彦克. 基于HBase的交通流数据实时存储系统[J]. 计算机应用, 2015, 35(1): 103-107.
[13]	陈庆奎周利珍. 基于HBase的大规模无线传感网络数据存储系统[J]. 计算机应用, 2012, 32(07): 1920-1923.
[14]	曹卫东贺国光. 连续航班延误与波及的贝叶斯网络分析[J]. 计算机应用, 2009, 29(2): 606-610.