Intelligent integration approach of big data for urban infrastructure management and maintenance

doi:10.11772/j.issn.1001-9081.2017.10.2983

Journal of Computer Applications ›› 2017, Vol. 37 ›› Issue (10): 2983-2990.DOI: 10.11772/j.issn.1001-9081.2017.10.2983

Previous Articles Next Articles

Intelligent integration approach of big data for urban infrastructure management and maintenance

LIU Jiajun^1,2, YU Gang^1,2, HU Min^1,2

1. SHU-UTS SILC Business School, Shanghai 201800, China;
2. Shanghai University-Shanghai Urban Construction Group Research Center of Building Industrialization, Shanghai 200072, China

Received:2017-04-11 Revised:2017-06-24 Online:2017-10-16 Published:2017-10-10
Supported by:
This work is partially supported by the Construction Project of Shanghai Urban-Rural Construction and Transportation Committee (2014-009-002); the Key Project of Shanghai Committee of Science and Technology (13511504803); the Major Project of Shanghai SASAC (2014008).

面向城市基础设施智慧管养的大数据智能融合方法

刘佳俊^1,2, 喻钢^1,2, 胡珉^1,2

1. 上海大学悉尼工商学院, 上海 201800;
2. 上海大学-上海城建建筑产业化研究中心, 上海 200072

通讯作者: 喻钢(1977-),男,江西南昌人,博士,讲师,主要研究方向:地下工程信息化施工管理,E-mail:yugang509@163.com
作者简介:刘佳俊(1994-),男,湖北荆州人,硕士研究生,主要研究方向:数据仓库、ETL建模;喻钢(1977-),男,江西南昌人,博士,讲师,主要研究方向:地下工程信息化施工管理;胡珉(1970-),女,上海人,副教授,博士,主要研究方向:建筑信息化.
基金资助:
上海市城乡建设和交通委员会建管项目（2014-009-002）；上海市科委重点项目（13511504803）；上海市国资委重大科研项目（2014008）。

Abstract

Abstract: In order to improve the efficiency of data integration, enhance both statistical and decisional analysis performance of the platform and reduce Extract-Transform-Load (ETL) execution time and the burden of data center, according to the operation and maintenance big data with characteristics of high dimension, diversity and variability, a Multilevel Task Scheduling (MTS) ETL framework (MTS-ETL) was proposed for intelligent maintenance requirements. Firstly, the data warehouse was divided into a series of parts, including data temporary area, data storage area, data classification area and data analysis area. In the light of the sub-region, the integral ETL process was divided into four levels of ETL task scheduling. Moreover, the multi-frequency ETL operation scheduling and sequential and non-sequential ETL working modes were designed at the same time. Secondly, the conceptual modelling, logical modelling and physical modelling of data integration were implemented based on the non-sequential mode of MTS-ETL framework. Finally, the ETL transformation module and job module were designed by using Pentaho Data Integration to realize this data integration method. In the traffic flow data integration experiment, the method integrated 136754 data for only 28.4 seconds, and reduced the total average execution time by 6.51% compared to the traditional ETL method in a thousand-scale data integration experiment. The reliability of ETL process was proved by the report analysis results of integrating 4 million data. The proposed method can effectively integrate the operation and maintenance of big data, improve the statistical analysis performance of platform and maintain ETL execution time at a low level.

Key words: big data, Extract-Transform-Load (ETL), data integration, data warehouse, urban infrastructure management and maintenance

摘要： 针对运维大数据维度高、形式多样化和变化迅速等特性，为提高数据融合效率以及平台的数据统计和决策分析性能，降低抽取-转换-加载（ETL）执行时间开销和数据中心负担，面向智慧管养需求提出一种多层次任务调度（MTS）ETL框架（MTS-ETL）。首先，将数据仓库分为数据临时区、数据仓储区、数据分类区和数据分析区，并根据所分区域将完整的ETL过程划分为4个层次的ETL任务调度环节，同时设计了多频率ETL运行调度以及顺序和非顺序两种ETL工作模式；接着，基于MTS-ETL框架的非顺序工作模式进行数据融合的概念建模、逻辑建模和物理建模；最后，利用Pentaho Data Integration设计ETL转换模块和工作模块以实现数据融合方法。在交通流量数据融合实验中，该方法融合136754条数据的时间仅为28.4s；在千量级的数据融合实验中比传统ETL方法的总平均执行时间降低了6.51%；报表分析结果表明其在融合400万条数据时依然能保证ETL过程的可靠性。所提方法能够有效融合运维大数据，提高平台统计分析性能，并维持ETL执行时间开销在较低水平。

关键词: 大数据, 抽取-转换-加载, 数据融合, 数据仓库, 城市基础设施管养

CLC Number:

TP311.13

LIU Jiajun, YU Gang, HU Min. Intelligent integration approach of big data for urban infrastructure management and maintenance[J]. Journal of Computer Applications, 2017, 37(10): 2983-2990.

刘佳俊, 喻钢, 胡珉. 面向城市基础设施智慧管养的大数据智能融合方法[J]. 计算机应用, 2017, 37(10): 2983-2990.

References

[1] ZHAO J, DENG W. Fuzzy multiobjective decision support model for urban rail transit projects in China[J]. Transport, 2013, 28(3): 224-235.
[2] SILVA M S, TIMES V C, KWAKYE M. A framework for ETL systems development[J]. Journal of Information & Data Management, 2012, 3(3): 300-315.
[3] STUMPTNER R, FREUDENTHALER B, KRENN M. BIAccelerator-a template-based approach for rapid ETL development[C]//ISMIS 2012: Proceedings of the 20th International Symposium on Methodologies for Intelligent Systems. Berlin: Springer, 2012: 435-444.
[4] RAHMAN N, MARZ J, AKHTER S. An ETL metadata model for data warehousing[J]. Journal of Computing & Information Technology, 2012, 20(2).
[5] RADHAKRISHNA V, SRAVANKIRAN V, RAVIKIRAN K. Automating ETL process with scripting technology[C]//Proceedings of the 2012 Nirma University International Conference on Engineering. Piscataway, NJ: IEEE, 2013: 1-4.
[6] SADIQ A, FAZZIKI A E, SADGAL M. An Agent based ETL system: towards an automatic code generation[J]. World Applied Sciences Journal, 2014, 31(5): 979-987.
[7] THEODOROU V, ABELLÓ A, THIELE M, et al. A framework for user-centered declarative ETL[C]//DOLAP 2014: Proceedings of the 17th International Workshop on Data Warehousing and OLAP. New York: ACM, 2014: 67-70.
[8] GILL R, SINGH J. An open source ETL tool-medium and small scale enterprise ETL (MaSSEETL) [J]. International Journal of Computer Applications, 2014, 108(4): 15-22.
[9] NOVAK M, RABUZIN K. Prototype of a Web ETL tool[J]. International Journal of Advanced Computer Science & Applications, 2014, 5(6): 97-103.
[10] ASTRIANI W, TRISMININGSIH R. Extraction, Transformation, and Loading (ETL) module for hotspot spatial data warehouse using Geokettle[J]. Procedia Environmental Sciences, 2016, 33: 626-634.
[11] 王可欣. ETL技术在电信数据运营中的应用[J]. 电脑知识与技术, 2016, 12(24): 256-257. (WANG K X. Application of ETL technology in telecommunication data operation [J]. Computer Knowledge and Technology, 2016, 12(24): 256-257.)
[12] 张建兴. 中国邮政速递数据仓库系统ETL的设计与实现[D]. 北京: 北京交通大学, 2014. (ZHANG J X. Design and implementation of ETL for China post data warehouse system [D]. Beijing: Beijing Jiaotong University, 2014.)

[1]	Xu LI, Yulin HE, Laizhong CUI, Zhexue HUANG, Fournier‑Viger PHILIPPE. Distributed observation point classifier for big data with random sample partition [J]. Journal of Computer Applications, 2024, 44(6): 1727-1733.
[2]	Meng CAO, Sunjie YU, Hui ZENG, Hongzhou SHI. Hierarchical access control and sharing system of medical data based on blockchain [J]. Journal of Computer Applications, 2023, 43(5): 1518-1526.
[3]	Li YANG, Jianting CHEN, Yang XIANG. Performance optimization strategy of distributed storage for industrial time series big data based on HBase [J]. Journal of Computer Applications, 2023, 43(3): 759-766.
[4]	Yu LING, Zhilong SHAN. Knowledge concept recommendation system based on interest enhancement [J]. Journal of Computer Applications, 2023, 43(12): 3697-3702.
[5]	ZHOU Xiang, ZHAI Junhai, HUANG Yajie, SHEN Ruicai, HOU Yingzhen. Instance selection algorithm for big data based on random forest and voting mechanism [J]. Journal of Computer Applications, 2021, 41(1): 74-80.
[6]	CAO Cejun, LIU Ju. Overview of modeling method of emergency organization decision in disaster operations management [J]. Journal of Computer Applications, 2020, 40(7): 2142-2149.
[7]	ZHU Xiaojie, ZHAO Zihao, DU Yi. PiFlow: model driven big data pipeline framework [J]. Journal of Computer Applications, 2020, 40(6): 1638-1647.
[8]	Wenli WU, Guohua LIU, Junbao ZHANG. Complexity analysis of functional query answering on big data [J]. Journal of Computer Applications, 2020, 40(2): 416-419.
[9]	LI Ziying, SHI Zhenguo. Scheduling method for big data tasks [J]. Journal of Computer Applications, 2020, 40(10): 2923-2928.
[10]	HAO Qinxia. Course recommendation system based on R2 index and multi-objective differential evolution [J]. Journal of Computer Applications, 2020, 40(10): 2951-2959.
[11]	ZHANG Yonglai, ZHOU Yaojian. Review of clustering algorithms [J]. Journal of Computer Applications, 2019, 39(7): 1869-1882.
[12]	MA Jiangang, MA Yinglong. Semantic-driven learning and classification method of judicial documents [J]. Journal of Computer Applications, 2019, 39(6): 1696-1700.
[13]	JI Lina, CHEN Kai, YU Yanwei, SONG Peng, WANG Shuying, WANG Chenrui. Vehicle type mining and application analysis based on urban traffic big data [J]. Journal of Computer Applications, 2019, 39(5): 1343-1350.
[14]	ZHAO Wei, LIN Yuming, HUANG Taoyi, LI You. User opinion extraction based on adaptive crowd labeling with cost constrain [J]. Journal of Computer Applications, 2019, 39(5): 1351-1356.
[15]	ZHANG Yitian, YU Jiong, LU Liang, LI Ziyang. Task scheduling strategy based on data stream classification in Heron [J]. Journal of Computer Applications, 2019, 39(4): 1106-1116.

Intelligent integration approach of big data for urban infrastructure management and maintenance

面向城市基础设施智慧管养的大数据智能融合方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics