基于学习的容器环境Spark性能监控与分析

doi:10.11772/j.issn.1001-9081.2017.12.3586

计算机应用 ›› 2017, Vol. 37 ›› Issue (12): 3586-3591.DOI: 10.11772/j.issn.1001-9081.2017.12.3586

基于学习的容器环境Spark性能监控与分析

皮艾迪^1,2, 喻剑^1,2, 周笑波^1,2

1. 同济大学计算机科学与技术系, 上海 201804;
2. 嵌入式系统与服务计算教育部重点实验室(同济大学), 上海 201804

收稿日期:2017-05-16 修回日期:2017-07-14 出版日期:2017-12-10 发布日期:2017-12-18
通讯作者: 周笑波
作者简介:皮艾迪(1993-),男,上海人,硕士研究生,主要研究方向:大数据处理、云计算;喻剑(1975-),男,浙江义乌人,讲师,博士,主要研究方向:物联网、大数据处理;周笑波(1973-),男,浙江台州人,教授,博士生导师,博士,主要研究方向:云计算、大数据并行处理、分布式系统、数据中心。

Learning-based performance monitoring and analysis for Spark in container environments

PI Aidi^1,2, YU Jian^1,2, ZHOU Xiaobo^1,2

1. Department of Computer Science and Technology, Tongji University, Shanghai 201804, China;
2. Key Laboratory of Embedded System and Service Computing, Ministry of Education(Tongji University), Shanghai 201804, China

Received:2017-05-16 Revised:2017-07-14 Online:2017-12-10 Published:2017-12-18

摘要/Abstract

摘要： Spark计算框架被越来越多的企业用作大数据分析的框架，由于通常部署在分布式和云环境中因此增加了该系统的复杂性，对Spark框架的性能进行监控并查找导致性能下降的作业向来是非常困难的问题。针对此问题，提出并编写了一种针对分布式容器环境中Spark性能的实时监控与分析方法。首先，通过在Spark中植入代码和监控Docker容器中的API文件获取并整合了作业运行时资源消耗信息；然后，基于Spark作业历史信息，训练了高斯混合模型（GMM）；最后，使用训练后的模型对Spark作业的运行时资源消耗信息进行分类并找出导致性能下降的作业。实验结果表明，所提方法能检测出90.2%的异常作业，且其对Spark作业性能的影响仅有4.7%。该方法能减轻查错的工作量，帮助用户更快地发现Spark的异常作业。

关键词: Spark, 容器, 分布式监控系统, 高斯混合模型, 机器学习

Abstract: The Spark computing framework has been adopted as the framework for big data analysis by an increasing number of enterprises. However, the complexity of the system is increased due to the characteristic that it is typically deployed in distributed and cloud environments. Therefore, it is always considered to be difficult to monitor the performance of the Spark framework and finding jobs that lead to performance degradation. In order to solve this problem, a real-time monitoring and analysis method for Spark performance in distributed container environment was proposed and compiled. Firstly, the resource consumption information of jobs at runtime was acquired and integrated through the implantation of code in Spark and monitoring of Application Program Interface (API) files in Docker containers. Then, the Gaussian Mixture Model (GMM) was trained based on job history information of Spark. Finally, the trained model was used to classify the resource consumption information of Spark jobs at runtime and find jobs that led to performance degradation. The experimental results show that, the proposed method can detect 90.2% of the abnormal jobs and it only introduces 4.7% degradation to the performance of Spark jobs. The proposde method can lighten the burden of error checking and help users find the abnormal jobs of Spark in a shorter time.

Key words: Spark, container, distributed monitoring system, Gaussian Mixture Model (GMM), machine learning

中图分类号:

皮艾迪, 喻剑, 周笑波. 基于学习的容器环境Spark性能监控与分析[J]. 计算机应用, 2017, 37(12): 3586-3591.

PI Aidi, YU Jian, ZHOU Xiaobo. Learning-based performance monitoring and analysis for Spark in container environments[J]. Journal of Computer Applications, 2017, 37(12): 3586-3591.

参考文献

[1] SAMBASIVAN R R, SHAFER I, SIGELMAN B H, et al. Principled workflow-centric tracing of distributed systems[C]//SoCC 2016:Proceeding of the 2016 Seventh ACM symposium on Cloud Computing. New York:ACM, 2016:401-414.
[2] KAVULYA S P, DANIELS S, JOSHI K, et al. Draco:statistical diagnosis of chronic problems in large distributed systems[C]//DSN 2012:Proceedings of the 201242nd Annual IEEE/IFIP International Conference on Dependable System and Networks. Washington, DC:IEEE Computer Society, 2012:1-12.
[3] SAMBASIVAN R R, ZHENG A X, DE ROSA M, et al. Diagnosing performance changes by comparing request flows[C]//NSDI'11:Proceeding of the 20118th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX Association, 2011:43-56.
[4] NAGARAJ K, KILLIAN C, NEVILLE J. Structured comparative analysis of systems logs to diagnose performance problems[C]//NSDI 2012:Proceedings of the 20129th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX Association, 2012:353-366.
[5] OLINER A J, KULKARNI A V, AIKEN A. Using correlated surprise to infer shared influence[C]//DSN 2010:Proceedings of the 2010 IEEE/IFIP International Conference on Dependable Systems and Networks. Piscataway, NJ:IEEE, 2010:191-200.
[6] XU W, HUANG L, FOX A, et al. Detecting large-scale system problems by mining console logs[C]//SOSP'09:Proceedings of the 2009 ACM SIGOPS 22nd Symposium on Operating Systems. New York:ACM, 2009:117-132.
[7] ZHAO X, ZHANG Y, LION D, et al. Iprof:a non-intrusive request flow profiler for distributed systems[C]//OSDI 2014:Proceedings of the 201411th USENIX Conference on Operating Systems Design and Implementation. Berkeley, CA:USENIX Association, 2014:629-644.
[8] ZHAO X, RODRIGUES K, LUO Y, et al. Non-intrusive performance profiling for entire software stacks based on the flow reconstruction principle[C]//OSDI 2016:Proceedings of the 201612th USENIX Symposium on Operating System Design and Implementation. Berkeley, CA:USENIX Association, 2016:603-618.
[9] 刘海宝,蔡皖东,许俊杰,等.分布式网络行为监控系统设计与实现[J].微电子学与计算机,2006,23(3):76-79. (LIU H B, CAI W D, XU J J, et al. Design and implement of distributed network behavior monitoring system[J]. Microelectronics & Computer, 2006, 23(3):76-79.)
[10] CHANDA A, COX A L, ZWAENEPOEL W. Whodunit:transactional profiling for multi-tier applications[C]//EuroSys 2007:Proceedings of the 20072nd ACM SIGOPS/EuroSys European Conference on Computer Systems. New York:ACM, 2007:17-30.
[11] BARHAM P, DONNELLY A, ISAACS R, et al. Using magpie for request extraction and workload modelling[C]//OSDI 2004:Proceedings of the 20046th USENIX Symposium on Operating Systems Design and Implementation. Berkeley, CA:USENIX Association, 2004:259-272.
[12] CHEN M Y, ACCARDI A, KICIMAN E, et al. Path-based failure and evolution management[C]//NSDI 2004:Proceedings of the 1st USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX Association, 2004:23-36.
[13] REYNOLDS P, KILLIAN C E, WIENER J L, et al. Pip:detecting the unexpected in distributed systems[C]//NSDI 2006:Proceedings of the 20063rd USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX Association, 2006:115-128.
[14] THERESKA E, SALMON B, STRUNK J, et al. Stardust:tracking activity in a distributed storage system[C]//Proceedings of the 2006 Joint International Conference on Measurement and Modeling of Computer Systems. New York:ACM, 2006:3-14.
[15] FONSECA R, PORTER G, KATZ R H, et al. X-trace:a pervasive network tracing framework[C]//NSDI 2007:Proceedings of the 20074th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX Association, 2007:20-33.
[16] MACE J, BODIK P, FONSECA R, et al. Retro:targeted resource management in multi-tenant distributed systems[C]//NSDI 2015:Proceedings of the 201512th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX Association, 2015:589-603.
[17] SIGELMAN B H, BARROSO L A, BURROWS M, et al. Dapper, a large-scale distributed systems tracing infrastructure, GoogleTechnical Report dapper-2010-1[R]. Mountain View:Google, 2010:29.
[18] KASIKCI B, SCHUBERT B, PEREIRA C, et al. Failure sketching:a technique for automated root cause diagnosis of in-production failures[C]//SOSP 2015:Proceeding of the 25th ACM Symposium on Operating Systems Principles. New York:ACM, 2015:344-360.
[19] 楼桦.服务器监控系统的实现[D].郑州:郑州大学,2004:25-28.(LOU H. Implementation of server's monitoring system[D]. Zhengzhou:Zhengzhou University, 2004:25-28.)
[20] 和荣,肖海力.基于Nagios的监控平台的设计与实现[J].科研信息化技术与应用,2014,5(5):77-85.(HE R, XIAO H L. A monitor platform based on Nagios[J]. E-Science Technology & Application, 2014, 5(5):77-85.)
[21] CANTRILL B M, SHAPIRO M W, LEVENTHAL A H. Dynamic instrumentation of production systems[C]//USENIX 2004:Proceedings of the 2004 USENIX Annual Technical Conference. Berkeley, CA:USENIX Association, 2004:15-28.
[22] ERLINGSSON U, PEINADO M, PETER S, et al. Fay:extensible distributed tracing from kernels to clusters[J]. ACM Transactions on Computer Systems, 2012, 30(4):Article No. 13.
[23] MACE J, ROELKE R, FONSECA R. Pivot tracing:dynamic causal monitoring for distributed systems[C]//SOSP 2015:Proceedings of the 201525th Symposium on Operating Systems Principles. New York:ACM, 2015:378-393.
[24] Microsoft. Microsoft azure:cloud computing platform & services[EB/OL].[2017-04-15]. https://azure.microsoft.com/en-us/?v=17.14.
[25] Amazon Web Service, Inc. Elastic Compute Cloud (EC2)-cloud server & hosting-AWS[EB/OL].[2017-04-15]. https://aws.amazon.com/ec2/.
[26] Apache. Apache Spark:lightning-fast cluster computing[EB/OL].[2017-04-15]. https://spark.apache.org/.
[27] Docker, Inc. Docker-Build, ship, and run[EB/OL].[2017-04-15]. https://www.docker.com/.
[28] Ganglia. Ganglia monitoring system[EB/OL].[2017-04-15]. http://ganglia.sourceforge.net/.
[29] YAN Y, GAO Y, CHEN Y, et al. TR-Spark:transient computing for big data analytics[C]//SoCC 2016:Proceeding of the 2016 Seventh ACM Symposium on Cloud Computing. New York:ACM, 2016:484-496.
[30] Graphite. Graphite documentation[DB/OL].[2017-03-14]. https://graphite.readthedocs.io/.
[31] Apache. Apache thrift-home[EB/OL].[2017-02-17]. https://thrift.apache.org/.
[32] GitHub, Inc. Intel-Hadoop/Hibench[EB/OL].[2017-03-30]. https://github.com/intel-hadoop/HiBench/.

基于学习的容器环境Spark性能监控与分析

Learning-based performance monitoring and analysis for Spark in container environments

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	毛铭泽, 曹芮浩, 闫春钢. 基于权值多样性的半监督分类算法[J]. 计算机应用, 2021, 41(9): 2473-2480.
[2]	郭棉, 张锦友. 移动边缘计算环境中面向机器学习的计算迁移策略[J]. 计算机应用, 2021, 41(9): 2639-2645.
[3]	秦斌斌, 彭良康, 卢向明, 钱江波. 司机分心驾驶检测研究进展[J]. 计算机应用, 2021, 41(8): 2330-2337.
[4]	尹飞, 龙玲莉, 孔峥, 邵涵, 李鑫, 钱柱中. 面向动态负载的集群容器部署方法[J]. 计算机应用, 2021, 41(6): 1581-1588.
[5]	秦静, 左长青, 汪祖民, 季长清, 王宝凤. 基于堆叠分类器的心电异常监测模型设计[J]. 计算机应用, 2021, 41(3): 887-890.
[6]	姜倩玉, 王凤英, 贾立鹏. 基于感知哈希算法和特征融合的恶意代码检测方法[J]. 计算机应用, 2021, 41(3): 780-785.
[7]	孟祥瑞, 杨文忠, 王婷. 基于图文融合的情感分析研究综述[J]. 计算机应用, 2021, 41(2): 307-317.
[8]	刘晓龙, 王士同. 渐进式分离的开放集模糊域自适应算法[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3127-3131.
[9]	楼豪杰, 郑元林, 廖开阳, 雷浩, 李佳. 基于Siamese-YOLOv4的印刷品缺陷目标检测[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3206-3212.
[10]	王雅辉, 钱宇华, 刘郭庆. 基于模糊优势互补互信息的有序决策树算法[J]. 计算机应用, 2021, 41(10): 2785-2792.
[11]	蒋阳升, 王胜男, 涂家祺, 李莎, 王红军. 面向高铁站的热舒适度和能耗综合预测[J]. 计算机应用, 2021, 41(1): 249-257.
[12]	朱琳, 于海涛, 雷新宇, 刘静, 王若凡. 基于MRI图像的阿尔茨海默症患者脑网络特征识别算法[J]. 计算机应用, 2020, 40(8): 2455-2459.
[13]	梁登高, 周安民, 郑荣锋, 刘亮, 丁建伟. 基于大小突发块划分的微信支付行为识别模型[J]. 计算机应用, 2020, 40(7): 1970-1976.
[14]	徐周波, 杨健, 刘华东, 黄文文. 基于XGBoost与拓扑结构信息的蛋白质复合物识别算法[J]. 计算机应用, 2020, 40(5): 1510-1514.
[15]	张俊升, 徐晶晶, 余伟. 面部美化图像质量无参考评价方法[J]. 计算机应用, 2020, 40(4): 1184-1190.