Three dimensional strom tracking method based on distributed computing architecture

doi:10.11772/j.issn.1001-9081.2017.04.0941

Abstract

Abstract: In recent years, meteorological data increases dramatically, and the amount of data has been TB-per-hour-level. The traditional relational database and file storage system have troubles in the massive data storage and management, thus large-scale and heterogeneous meteorological data cannot also be used effectively in meteorological business. Furthermore, it would be also difficult for scientific researchers to efficiently explore the huge amount of heterogeneous meteorological data. In order to tackle these problems, researchers have developed many types of distributed computing frameworks based on MapReduce and HBase, etc., which provide an effective way to exploit large-scale meteorological data. The distributed computing and storing techniques have been tested separately in applications of meteorology field. However, to our best knowledge, these techniques have not been carefully studied jointly. Therefore, a new 3D storm tracking method based on the combination of MapReduce and Hbase was studied by using a large amount of weather radar data accumulated in recent years. Moreover, based on the original Rest interface, a series of distributed service interfaces were implemented for exploring a variety of point, line and surface data. Compared with the performance of the standard single data storage and access interface based on Rest, the proposed method has better comprehensive performance, and the efficiency is improved about 100%. A practical application for tracking 3D storm in Zhujiang River urban agglomeration from 2007 to 2009 was used to further validate the performance of the proposed method.

Key words: distributed computing framework, storm tracking method, long time series analysis

摘要： 气象数据的增长规模已达到每小时TB级，这使得传统基于关系型数据库和文件存储系统在海量数据存储与管理方面捉襟见肘，进而使得基于大规模异构气象数据的应用无法规模化，同时，也无法满足科研人员对海量气象数据高效探索的需要。为解决这一系列问题，研究者分别基于MapReduce、HBase等分布式框架下的分布式计算和存储技术，尝试为海量气象数据的探索提供有效技术手段，然而，综合性的研究据了解还未开展。因此，利用近年来积累的海量多普勒天气雷达数据，开展了基于MapReduce和HBase相结合的风暴三维追踪方法的研究，并基于传统Rest标准化接口实现了雷达资料的点、线、面、体的多种分布式服务接口，与传统的Rest标准化单机数据存储和访问接口的性能相比，所实现方法在性能方面有100%的效率提升。最后，以2007年至2009年珠江三角洲地区三年雷达数据的风暴追踪回算为例，进一步验证了所提方法在计算和存储管理方面的性能优势。

关键词: 分布式计算框架, 风暴追踪算法, 长时间序列分析

CLC Number:

TP391.7

ZENG Qin, LI Yongsheng. Three dimensional strom tracking method based on distributed computing architecture[J]. Journal of Computer Applications, 2017, 37(4): 941-944.

曾沁, 李永生. 基于分布式计算框架的风暴三维追踪方法[J]. 计算机应用, 2017, 37(4): 941-944.

References

[1] ZHOU S, YANG X, LI X, et al. A Hadoop-based visualization and diagnosis framework for earth science data[C]//Proceedings of the 2015 IEEE International Conference on Big Data. Piscataway, NJ: IEEE, 2015:1972-1977.
[2] DUFFY D Q, SCHNASE J L, THOMPSON J H, et al. Preliminary evaluation of MapReduce for high-performance climate data analysis[EB/OL].[2015-10-10]. https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20120009187.pdf.
[3] 李永生, 曾沁, 徐美红, 等. 基于Hapdoop的数值预报产品服务平台设计与实现[J]. 应用气象学报, 2015, 26(1):122-128.(LI Y S, ZENG Q, XU M S, et al. Design and implementation of NWP data service platform based on Hadoop framework[J]. Journal of Applied Meteorological Science, 2015, 26(1):122-128.)
[4] 李英俊, 韩雷. 基于三维雷达图像数据的风暴体追踪算法研究[J]. 计算机应用, 2008, 28(4):1078-1080.(LI Y J, HAN L. Storm tracking algorithm development based on the three-dimensional radar image data[J]. Journal of Computer Applications, 2008, 28(4): 1078-1080.)
[5] 王海艳, 伏彩航. 基于HBase数据分类的压缩策略选择方法[J]. 通信学报, 2016, 37(4):12-22.(WANG H Y, FU C H. Compression strategies selection method based on classification of HBase data[J]. Journal on Communications, 2016, 37(4):12-22.)
[6] 董春涛, 李文婷, 沈晴霓, 等. Hadoop YARN大数据计算框架及其资源调度机制研究[J]. 信息通信技术, 2015(1):77-84.(DONG C T, LI W T, SHEN Q N, et al. Research on the framework and resource scheduling mechanisms of Hadoop YARN[J]. Information and Communications Technologies, 2015(1):77-84.)
[7] KARUN A K, CHITHARANJAN K. A review on Hadoop-HDFS infrastructure extensions[C]//Proceedings of the 2013 IEEE Conference on Information and Communication Technologies. Piscataway, NJ: IEEE, 2013:132-137.
[8] VORA M N. Hadoop-HBase for large-scale data[C]//Proceedings of the 2011 International Conference on Computer Science and Network Technology. Piscataway, NJ: IEEE, 2011,1:601-605.
[9] DITTRICH J, QUIANÉ-RUIZ J A. Efficient big data processing in Hadoop MapReduce[J]. Proceedings of the VLDB Endowment, 2012, 5(12):2014-2015.
[10] 张海懿, 张国颖, 徐云斌. 软件定义传送网标准化发展探讨与思考[J]. 中兴通讯技术, 2015, 21(6):45-48.(ZHANG H Y, ZHANG G Y, XU Y B. Standardization development of software-defined transport networks[J]. ZTE Communications, 2015,21(6):45-48.)
[11] 王想红,刘纪平,徐胜华,等. 基于NetCDF数据模型的海洋环境数据三维可视化研究[J]. 测绘科学, 2013, 38(2):59-61.(WANG X H, LIU J P, XU S H, et al. Visualization of marine environment data based on NetCDF data model[J]. Science of Surveying and Mapping, 2013, 38(2):59-61.)

[1]	HUANG Xiaoxiang, HU Yongmei, WU Dan, REN Lijie. Early identification and prediction of abnormal carotid arteries based on variational autoencoder [J]. Journal of Computer Applications, 2021, 41(10): 3082-3088.
[2]	ZHU Xincheng, HE Kunjin, NI Na, HAO Bo. Rapid calculation method of orthopedic plate fit based on improved iterative closest point algorithm [J]. Journal of Computer Applications, 2021, 41(10): 3033-3039.
[3]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[4]	WU Zeju, JIAO Cuijuan, CHEN Liang. Tire defect detection method based on improved Faster R-CNN [J]. Journal of Computer Applications, 2021, 41(7): 1939-1946.
[5]	. Optimal preview repetitive control of discrete systems based on information fusion [J]. Journal of Computer Applications, 0, (): 0-0.
[6]	. Difference analysis of event-related potentials under different emotions based on FastICA algorithm [J]. Journal of Computer Applications, 0, (): 0-0.
[7]	LIU Yuliang, CHEN Huaili. Distribution path planning and charging strategy for pure electric vehicles with load constraint [J]. Journal of Computer Applications, 2020, 40(10): 2831-2837.
[8]	ZHANG Xiaobo, YANG Yan, LI Tianrui, LU Fan, PENG Lilan. Early diagnosis and prediction of Parkinson's disease based on clustering medical text data [J]. Journal of Computer Applications, 2020, 40(10): 3088-3094.
[9]	ZHONG Wenbin, SUN Si, LI Xurui, LIU Guangshuai. Point cloud compression method combining density threshold and triangle group approximation [J]. Journal of Computer Applications, 2020, 40(7): 2059-2068.
[10]	SHI Lukui, MA Hongqi, ZHANG Chaozong, FAN Shiyan. Detection method of pulmonary nodules based on improved residual structure [J]. Journal of Computer Applications, 2020, 40(7): 2110-2116.
[11]	SUN Peng, XIAO Jing, ZHAO Haimeng, LIU Fan, YAN Lei, ZHAO Hongying. Design and implementation of SIFT algorithm for UAV remote sensing image based on DSP platform [J]. Journal of Computer Applications, 2020, 40(4): 1237-1242.
[12]	ZHANG Kai, ZHOU Deyun, YANG Zhen, PAN Qian. Constrained multi-objective weapon-target assignment problem [J]. Journal of Computer Applications, 2020, 40(3): 902-911.
[13]	GUO Yihui, HUANG Chenghui, ZHONG Xueling, LU Jiyuan. Direction-perception feature recognition on mesh model [J]. Journal of Computer Applications, 2019, 39(12): 3673-3677.
[14]	WANG Ping, GAO Chen, ZHU Li, ZHAO Jun, ZHANG Jing, KONG Weiming. Segmentation algorithm of ischemic stroke lesion based on 3D deep residual network and cascade U-Net [J]. Journal of Computer Applications, 2019, 39(11): 3274-3279.
[15]	ZHOU Wan, HU Xuemin, SHI Chenyin, WEI Jieling, TONG Xiuchi. Motion planning algorithm of robot for crowd evacuation based on deep Q-network [J]. Journal of Computer Applications, 2019, 39(10): 2876-2882.