面向城市基础设施智慧管养的大数据智能融合方法

doi:10.11772/j.issn.1001-9081.2017.10.2983

计算机应用 ›› 2017, Vol. 37 ›› Issue (10): 2983-2990.DOI: 10.11772/j.issn.1001-9081.2017.10.2983

• 应用前沿、交叉与综合 • 上一篇下一篇

面向城市基础设施智慧管养的大数据智能融合方法

刘佳俊^1,2, 喻钢^1,2, 胡珉^1,2

1. 上海大学悉尼工商学院, 上海 201800;
2. 上海大学-上海城建建筑产业化研究中心, 上海 200072

收稿日期:2017-04-11 修回日期:2017-06-24 出版日期:2017-10-10 发布日期:2017-10-16
通讯作者: 喻钢(1977-),男,江西南昌人,博士,讲师,主要研究方向:地下工程信息化施工管理,E-mail:yugang509@163.com
作者简介:刘佳俊(1994-),男,湖北荆州人,硕士研究生,主要研究方向:数据仓库、ETL建模;喻钢(1977-),男,江西南昌人,博士,讲师,主要研究方向:地下工程信息化施工管理;胡珉(1970-),女,上海人,副教授,博士,主要研究方向:建筑信息化.
基金资助:
上海市城乡建设和交通委员会建管项目（2014-009-002）；上海市科委重点项目（13511504803）；上海市国资委重大科研项目（2014008）。

Intelligent integration approach of big data for urban infrastructure management and maintenance

LIU Jiajun^1,2, YU Gang^1,2, HU Min^1,2

1. SHU-UTS SILC Business School, Shanghai 201800, China;
2. Shanghai University-Shanghai Urban Construction Group Research Center of Building Industrialization, Shanghai 200072, China

Received:2017-04-11 Revised:2017-06-24 Online:2017-10-10 Published:2017-10-16
Supported by:
This work is partially supported by the Construction Project of Shanghai Urban-Rural Construction and Transportation Committee (2014-009-002); the Key Project of Shanghai Committee of Science and Technology (13511504803); the Major Project of Shanghai SASAC (2014008).

摘要/Abstract

摘要： 针对运维大数据维度高、形式多样化和变化迅速等特性，为提高数据融合效率以及平台的数据统计和决策分析性能，降低抽取-转换-加载（ETL）执行时间开销和数据中心负担，面向智慧管养需求提出一种多层次任务调度（MTS）ETL框架（MTS-ETL）。首先，将数据仓库分为数据临时区、数据仓储区、数据分类区和数据分析区，并根据所分区域将完整的ETL过程划分为4个层次的ETL任务调度环节，同时设计了多频率ETL运行调度以及顺序和非顺序两种ETL工作模式；接着，基于MTS-ETL框架的非顺序工作模式进行数据融合的概念建模、逻辑建模和物理建模；最后，利用Pentaho Data Integration设计ETL转换模块和工作模块以实现数据融合方法。在交通流量数据融合实验中，该方法融合136754条数据的时间仅为28.4s；在千量级的数据融合实验中比传统ETL方法的总平均执行时间降低了6.51%；报表分析结果表明其在融合400万条数据时依然能保证ETL过程的可靠性。所提方法能够有效融合运维大数据，提高平台统计分析性能，并维持ETL执行时间开销在较低水平。

关键词: 大数据, 抽取-转换-加载, 数据融合, 数据仓库, 城市基础设施管养

Abstract: In order to improve the efficiency of data integration, enhance both statistical and decisional analysis performance of the platform and reduce Extract-Transform-Load (ETL) execution time and the burden of data center, according to the operation and maintenance big data with characteristics of high dimension, diversity and variability, a Multilevel Task Scheduling (MTS) ETL framework (MTS-ETL) was proposed for intelligent maintenance requirements. Firstly, the data warehouse was divided into a series of parts, including data temporary area, data storage area, data classification area and data analysis area. In the light of the sub-region, the integral ETL process was divided into four levels of ETL task scheduling. Moreover, the multi-frequency ETL operation scheduling and sequential and non-sequential ETL working modes were designed at the same time. Secondly, the conceptual modelling, logical modelling and physical modelling of data integration were implemented based on the non-sequential mode of MTS-ETL framework. Finally, the ETL transformation module and job module were designed by using Pentaho Data Integration to realize this data integration method. In the traffic flow data integration experiment, the method integrated 136754 data for only 28.4 seconds, and reduced the total average execution time by 6.51% compared to the traditional ETL method in a thousand-scale data integration experiment. The reliability of ETL process was proved by the report analysis results of integrating 4 million data. The proposed method can effectively integrate the operation and maintenance of big data, improve the statistical analysis performance of platform and maintain ETL execution time at a low level.

Key words: big data, Extract-Transform-Load (ETL), data integration, data warehouse, urban infrastructure management and maintenance

中图分类号:

TP311.13

刘佳俊, 喻钢, 胡珉. 面向城市基础设施智慧管养的大数据智能融合方法[J]. 计算机应用, 2017, 37(10): 2983-2990.

LIU Jiajun, YU Gang, HU Min. Intelligent integration approach of big data for urban infrastructure management and maintenance[J]. Journal of Computer Applications, 2017, 37(10): 2983-2990.

参考文献

[1] ZHAO J, DENG W. Fuzzy multiobjective decision support model for urban rail transit projects in China[J]. Transport, 2013, 28(3): 224-235.
[2] SILVA M S, TIMES V C, KWAKYE M. A framework for ETL systems development[J]. Journal of Information & Data Management, 2012, 3(3): 300-315.
[3] STUMPTNER R, FREUDENTHALER B, KRENN M. BIAccelerator-a template-based approach for rapid ETL development[C]//ISMIS 2012: Proceedings of the 20th International Symposium on Methodologies for Intelligent Systems. Berlin: Springer, 2012: 435-444.
[4] RAHMAN N, MARZ J, AKHTER S. An ETL metadata model for data warehousing[J]. Journal of Computing & Information Technology, 2012, 20(2).
[5] RADHAKRISHNA V, SRAVANKIRAN V, RAVIKIRAN K. Automating ETL process with scripting technology[C]//Proceedings of the 2012 Nirma University International Conference on Engineering. Piscataway, NJ: IEEE, 2013: 1-4.
[6] SADIQ A, FAZZIKI A E, SADGAL M. An Agent based ETL system: towards an automatic code generation[J]. World Applied Sciences Journal, 2014, 31(5): 979-987.
[7] THEODOROU V, ABELLÓ A, THIELE M, et al. A framework for user-centered declarative ETL[C]//DOLAP 2014: Proceedings of the 17th International Workshop on Data Warehousing and OLAP. New York: ACM, 2014: 67-70.
[8] GILL R, SINGH J. An open source ETL tool-medium and small scale enterprise ETL (MaSSEETL) [J]. International Journal of Computer Applications, 2014, 108(4): 15-22.
[9] NOVAK M, RABUZIN K. Prototype of a Web ETL tool[J]. International Journal of Advanced Computer Science & Applications, 2014, 5(6): 97-103.
[10] ASTRIANI W, TRISMININGSIH R. Extraction, Transformation, and Loading (ETL) module for hotspot spatial data warehouse using Geokettle[J]. Procedia Environmental Sciences, 2016, 33: 626-634.
[11] 王可欣. ETL技术在电信数据运营中的应用[J]. 电脑知识与技术, 2016, 12(24): 256-257. (WANG K X. Application of ETL technology in telecommunication data operation [J]. Computer Knowledge and Technology, 2016, 12(24): 256-257.)
[12] 张建兴. 中国邮政速递数据仓库系统ETL的设计与实现[D]. 北京: 北京交通大学, 2014. (ZHANG J X. Design and implementation of ETL for China post data warehouse system [D]. Beijing: Beijing Jiaotong University, 2014.)

[1]	周翔, 翟俊海, 黄雅婕, 申瑞彩, 侯璎真. 基于随机森林和投票机制的大数据样例选择算法[J]. 计算机应用, 2021, 41(1): 74-80.
[2]	屈景怡, 曹磊, 陈敏, 董樑, 曹烨琇. 基于团簇随机连接的CliqueNet航班延误预测模型[J]. 计算机应用, 2020, 40(8): 2420-2427.
[3]	曹策俊, 刘桔. 灾害运作管理中应急组织决策建模方法综述[J]. 计算机应用, 2020, 40(7): 2142-2149.
[4]	朱小杰, 赵子豪, 杜一. 模型驱动的大数据流水线框架PiFlow[J]. 计算机应用, 2020, 40(6): 1638-1647.
[5]	吴文莉, 刘国华, 张君宝. 大数据上函数查询解答的复杂度分析[J]. 计算机应用, 2020, 40(2): 416-419.
[6]	李孜颖, 石振国. 面向大数据任务的调度方法[J]. 计算机应用, 2020, 40(10): 2923-2928.
[7]	郝秦霞. 基于R2指标的高维多目标差分进化推荐式课程系统[J]. 计算机应用, 2020, 40(10): 2951-2959.
[8]	林政, 吕霞付. 基于改进模糊算法的水面无人艇自主避障[J]. 计算机应用, 2019, 39(9): 2523-2528.
[9]	章永来, 周耀鉴. 聚类算法综述[J]. 计算机应用, 2019, 39(7): 1869-1882.
[10]	马建刚, 马应龙. 语义驱动的司法文档学习分类方法[J]. 计算机应用, 2019, 39(6): 1696-1700.
[11]	纪丽娜, 陈凯, 于彦伟, 宋鹏, 王淑莹, 王成锐. 基于城市交通大数据的车辆类别挖掘及应用分析[J]. 计算机应用, 2019, 39(5): 1343-1350.
[12]	张译天, 于炯, 鲁亮, 李梓杨. 大数据流式计算框架Heron环境下的流分类任务调度策略[J]. 计算机应用, 2019, 39(4): 1106-1116.
[13]	黄丰雨, 吴业福, 陈镜任, 吴冰. 基于区域道路实况数据的交通行为谱分析方法[J]. 计算机应用, 2019, 39(3): 907-912.
[14]	丁景全, 马博, 李晓. 基于融合时空数据的车辆加油行为多视图深度异常检测框架[J]. 计算机应用, 2019, 39(11): 3370-3375.
[15]	常征, 吕勇. 基于正则表达式的海量数据清洗系统[J]. 计算机应用, 2019, 39(10): 2942-2947.

面向城市基础设施智慧管养的大数据智能融合方法

Intelligent integration approach of big data for urban infrastructure management and maintenance

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics