Data management based on Hadoop for power geographic information system

doi:10.11772/j.issn.1001-9081.2014.10.2806

Abstract

Abstract:

In consideration of the problem that the traditional power Geographic Information System (GIS) has inabilities in storage, computing and scalability, cloud computing was applied into the power GIS field and a solution which used Hadoop cloud platform to store and manage the massive GIS data was proposed. After analyzing the characteristics of the power GIS data，a data storage strategy was proposed which combined relational database with non-relational database. Based on this strategy, the architecture of power GIS management based on Hadoop was presented, the data model and parallel data analysis based on MapReduce were designed. Finally, a quantity of experiments were carried out including spatial analysis and operation data queries in single-machine and cluster environment to compare and validate the performance. The experimental results show that the average time of data analysis and query declines over 30% after reaching certain amount of data. The proposed scheme has more obvious advantages to deal with large-scale data and has high efficiency and good feasibility.

摘要：

针对传统电力地理信息系统（GIS）在存储能力、分析能力和扩展能力上的不足，将云计算技术应用到电力GIS领域，提出利用Hadoop云平台对电力GIS数据进行高效存储和管理的方案。首先对电力GIS各类数据的特点进行了分析，提出了关系型数据库与非关系型数据库相结合的数据存储策略，并在此基础上设计了基于Hadoop的电力GIS数据管理整体架构、相应的数据模型以及基于MapReduce的数据并行查询分析方法。最后，在单机和集群的环境下，对空间分析与运行数据查询的性能进行了对比与验证。实验结果表明，在数据量达到一定规模时，该方案优势明显，数据分析与查询的平均时间缩短30%以上，具有较高的效率和良好的扩展性。

CLC Number:

TP391

LIN Biying WANG Yanping. Data management based on Hadoop for power geographic information system[J]. Journal of Computer Applications, 2014, 34(10): 2806-2811.

林碧英王艳萍. 基于Hadoop的电力地理信息系统数据管理[J]. 计算机应用, 2014, 34(10): 2806-2811.

References

［1］CHEN Q, DENG Q. Cloud computing and its key techniques ［J］. Journal of Computer Applications, 2009,29(9):254-259.(陈全,邓倩妮.云计算及其关键技术［J］.计算机应用,2009,29(9):254-259.)
［2］RAO W, DING J, LU Q. Cloud computing platform construction for smart grid ［J］. East China Electric Power, 2011,39(9):120-123.(饶威,丁坚勇,路庆凯.智能电网云计算平台构建［J］.华东电力,2011,39(9):120-123.)［3］SONG Y, ZHOU G, ZHU Y. Present status and challenges of big data processing in smart grid ［J］. Power System Technology, 2013,37(4):55-63.(宋亚奇,周国亮,朱永利.智能电网大数据处理技术现状与挑战［J］.电网技术,2013,37(4):55-63.)
［4］WANG X, WU K, XIAO X, et al.A study on power WebGIS platform ［J］. Power System Technology, 2012,36(4):180-184.(王晓辉,吴克河,肖湘宁,等.电力WebGIS平台研究［J］.电网技术,2012,36(4):180-184.)
［5］SADALAGE P J, FOWLER M. NoSQL distilled ［M］. AI F, translated. Beijing: China Machine Press, 2013:3-14.(SADALAGE P J, FOWLER M. NoSQL精粹［M］.爱飞翔，译.北京：机械工业出版社,2013:3-14.)
［6］QIN X, WANG H, LI F, et al.New landscape of data management technologies ［J］. Journal of Software, 2013,36(2):175-197.(覃雄派,王会举,李芙蓉,等.数据管理技术的新格局［J］.软件学报,2013,36(2):175-197.)
［7］WANG D, SONG Y, ZHU Y. Information platform of smart grid based on cloud computing ［J］. Automation of Electric Power Systems, 2010,34(22):12-17.(王德文,宋亚奇,朱永利.基于云计算的智能电网信息平台［J］.电力系统自动化,2010,34(22):12-17.)
［8］Apache. Welcome to Apache Hadoop ［EB/OL］. ［2013-10-31］. http://hadoop.apache.org.
［9］SONG Y, LIU S, ZHU Y, et al.Application of cloud computing technology in the transmission line condition monitoring system ［J］. Mathematics in Practice and Theory, 2013,43(5):111-117.(宋亚奇,刘树仁,朱永利,等.云计算技术在输电线路状态监测系统中的应用［J］.数学的实践与认识,2013,43(5):111-117.)
［10］OGC. Open geospatial consortium ［EB/OL］. ［2009-12-31］. http://www.opengis.org/techno.
［11］YIN H, WEI Z, WANG X, et al.Real-time monitoring system for transmission line based on WebGIS ［J］. Modern Electric Power, 2009,26(5):70-74.(尹海燕,魏振华,王晓辉,等.基于WebGIS的输电线路实时监测系统［J］.现代电力,2009,26(5):70-74.)
［12］ZHANG S, HAN J, LIU Z, et al.Research on implementing spatial queries based on MapReduce ［J］. Chinese High Technology Letters, 2009,20(7):70-74.(张书彬,韩冀中,刘志勇,等.基于MapReduce实现空间查询的研究［J］.高科技通讯,2010,20(7):719-726.)
［13］WANG D, XIAO K, XIAO L. Data warehouse of electric power equipment condition information based on Hive ［J］. Power System Protection and Control, 2013,41(9):130-135.(王德文,肖凯,肖磊.基于Hive的电力设备状态信息数据仓库［J］.电力系统保护与控制,2013,41(9):130-135.)
［14］ARRES B, KABBACHI N, BOUSSAID O. Building OLAP cubes on a cloud computing environment with MapReduce ［C］// Proceedings of the 2013 ACS International Conference on Computer Systems and Applications. Piscataway: IEEE Press, 2013:1-5.
［15］YOU J, YANG Z, HU J, et al.Multi-dimensional visualized analysis framework for supporting large scale data ［J］. Computer Engineering, 2011,37(19):26-28.(游进国,杨卓荦,胡建华,等.一种支持大规模数据的多维可视化分析框架［J］.计算机工程,2011,37(19):26-28.)

[1]	. Pavement defect detection algorithm with enhanced morphological perception [J]. Journal of Computer Applications, 0, (): 0-0.
[2]	YANG Jianfeng, CHEN Bin, LI Yuxuan. Self-supervised point cloud anomaly detection method based on point cloud reconstruction [J]. Journal of Computer Applications, 0, (): 0-0.
[3]	XUE Zhenhua, LI Qiang, HUANG Chao. Vision foundation model-driven pixel-level image anomaly detection model [J]. Journal of Computer Applications, 0, (): 0-0.
[4]	. Binocular vision object location algorithm for robot arm grasping [J]. Journal of Computer Applications, 0, (): 0-0.
[5]	. Multi-attention contrastive learning for infrared small target detection [J]. Journal of Computer Applications, 0, (): 0-0.
[6]	LI Zhonghua, ZHONG Gengxin, FAN Ping, ZHU Hengliang. Camouflaged object detection by boundary mining and background guidance [J]. Journal of Computer Applications, 0, (): 0-0.
[7]	. Multi-view difficult airway recognition based on discriminant region guidance [J]. Journal of Computer Applications, 0, (): 0-0.
[8]	. Privacy preserving localization of surveillance images based on large vision models [J]. Journal of Computer Applications, 0, (): 0-0.
[9]	. Gravity data denoising method based on multilevel wavelet residual network#br# #br# [J]. Journal of Computer Applications, 0, (): 0-0.
[10]	. Data augmentation method for abnormal passenger behavior in elevators based on dynamic graph convolutional network [J]. Journal of Computer Applications, 0, (): 0-0.
[11]	. Text-based person retrieval method based on multi-granularity shared semantic center association [J]. Journal of Computer Applications, 0, (): 0-0.
[12]	ZHANG Qing, YANG Fan, FANG Yuhan. Chinese Spelling Correction Algorithm Based on Multi-Modal Information Fusion [J]. Journal of Computer Applications, 0, (): 0-0.
[13]	WANG Hao, WANG Jinwei, CHENG Xin, ZHANG Jiawei, WU Hao, LUO Xiangyang, MA Bin. A Review of JPEG Recompression Forensics for Color Images [J]. Journal of Computer Applications, 0, (): 0-0.
[14]	. Distribution adaptation and dynamic curriculum pseudo-labeling framework for semi-supervised fire detection [J]. Journal of Computer Applications, 0, (): 0-0.
[15]	. Multimodal harmful content detection method based on weakly supervised modality semantic enhancement [J]. Journal of Computer Applications, 0, (): 0-0.