面向高性能计算的分布式故障定位框架

doi:10.11772/j.issn.1001-9081.2017071948

计算机应用 ›› 2018, Vol. 38 ›› Issue (1): 44-49.DOI: 10.11772/j.issn.1001-9081.2017071948

• 2017年全国开放式分布与并行计算学术年会(DPCS 2017)论文 • 上一篇下一篇

面向高性能计算的分布式故障定位框架

高剑, 于康, 卿鹏, 尉红梅

江南计算技术研究所, 江苏无锡 214083

收稿日期:2017-08-08 修回日期:2017-08-24 出版日期:2018-01-10 发布日期:2018-01-22
通讯作者: 高剑
作者简介:高剑(1992-),男,云南富宁人,硕士研究生,主要研究方向:并行计算、运行时系统;于康(1987-),男,江西景德镇人,助理工程师,博士,主要研究方向:并行计算;卿鹏(1979-),男,四川资阳人,高级工程师,硕士,主要研究方向:并行编译、运行时系统;尉红梅(1968-),女,江苏无锡人,高级工程师,博士,主要研究方向:并行计算、并行编译。
基金资助:
国家重点研发计划项目（2016YFB0200502）。

Distributed fault localization framework for high performance computing

GAO Jian, YU Kang, QING Peng, WEI Hongmei

Jiangnan Institute of Computing Technology, Wuxi Jiangsu 214083, China

Received:2017-08-08 Revised:2017-08-24 Online:2018-01-10 Published:2018-01-22
Supported by:
This work is partially supported by the National Key Research and Development Program of China (2016YFB0200502).

摘要/Abstract

摘要： 针对高性能计算系统中故障定位难度高且实时性差的问题，提出了一种基于消息传递的故障定位框架（MPFL），包括基于树形拓扑的故障检测（TFD）和故障分析（TFA）算法。首先，在并行作业初始化时，将所有参与计算的节点进行逻辑上的树形划分，生成故障定位树（FLT），并将故障定位任务分布到节点上；然后，当消息库、操作系统等组件检测到节点异常状态时，基于TFD算法分析作业的FLT结构，根据负载平衡、性能开销等因素选择接收异常状态的节点；最后，节点利用TFA算法对接收到的异常状态进行推理得出故障，TFA算法使用基于规则的事件关联，并基于消息传递设计轻量级的主动探测，将两种方式相结合，提高了故障分析的准确性。实验以模拟节点停机故障为定位目标，并以NPB-FT与NPB-IS为基准测试，在集群上对MPFL框架进行了评估。实验结果表明，MPFL框架在故障定位能力与开销节省方面表现突出。

关键词: 高性能计算, 消息传递, 故障定位, 事件关联, 主动探测

Abstract: To solve the problem of high difficulty and poor real-time in fault localization for high performance computing system, a Message-Passing based Fault Localization (MPFL) framework was proposed, which included Tree-based Fault Detection (TFD) and Tree-based Fault Analysis (TFA) algorithms. Firstly, when the parallel application was initialized, the Fault Localization Tree (FLT) was obtained by logically dividing all the nodes participating in the computing, and the fault localization tasks were distributed to different nodes. Secondly, if the abnormal state of a node was detected by system components such as message-passing library and operating system, the TFD algorithm was used to analyze the FLT structure, and the node responsible for receiving the abnormal state was selected according to factors such as load balancing and performance cost. Finally, the fault was derived from the received abnormal state, which was reasoned by the node that used TFA algorithm. The rule-based event correlation and the lightweight active probing based on message-passing were used in TFA algorithm, and the accuracy of fault analysis was improved by combining these two approaches. The experimental evaluation was performed on a typical cluster, which demonstrated the capability of MPFL by locating the shutdown simulation nodes. The experimental results on the NPB-FT and NPB-IS benchmarks show that the MPFL framework has excellent performance on fault localization capability and cost saving.

Key words: high performance computing, message-passing, fault localization, event correlation, active probing

中图分类号:

TP302

高剑, 于康, 卿鹏, 尉红梅. 面向高性能计算的分布式故障定位框架[J]. 计算机应用, 2018, 38(1): 44-49.

GAO Jian, YU Kang, QING Peng, WEI Hongmei. Distributed fault localization framework for high performance computing[J]. Journal of Computer Applications, 2018, 38(1): 44-49.

参考文献

[1] ZHENG Z, LI Y, LAN Z. Anomaly localization in large-scale clusters[C]//Proceedings of the 2007 IEEE International Conference on Cluster Computing. Piscataway, NJ:IEEE, 2007:322-330.
[2] AVIZIENIS A, LAPRIE J C, RANDELL B. Fundamental concepts of dependability[R]. Newcastle:LAAS-CNRS, 2001:4.
[3] JORDAAN J F, PATEROK M. Event correlation in heterogeneous networks using the OSI management framework[C]//Proceedings of the IFIP TC6/WG6.6 Third International Symposium on Integrated Network Management with Participation of the IEEE Communications Society CNOM and with Support from the Institute for Educational Services. Amsterdam:North-Holland Publishing Co., 1993:683-695.
[4] NATU M, SETHI A S. Active probing approach for fault localization in computer networks[C]//Proceedings of the 20064th IEEE/IFIP Workshop on End-to-End Monitoring Techniques and Services. Piscataway, NJ:IEEE, 2006:25-33.
[5] LGORZATA STEINDER M, SETHI A S. A survey of fault localization techniques in computer networks[J]. Science of Computer Programming, 2004, 53(2):165-194.
[6] KATKER S, PATEROK M. Fault isolation and event correlation for integrated fault management[C]//Proceedings of the 5th IFIP/IEEE International Symposium on Integrated Network Management. Berlin:Springer, 1997:583-596.
[7] CHENG L, QIU X, MENG L, et al. Efficient active probing for fault diagnosis in large scale and noisy networks[C]//Proceedings of the 29th IEEE International Conference on Computer Communications. Washington, DC:IEEE Computer Society, 2010:1-9.
[8] PATIL B M, PATHAK V K. Survey of probe set and probe station selection algorithms for fault detection and localization in computer networks[J]. IEEE Transactions on Networks and Communications, 2015, 3(4):57.
[9] 孟洛明,黄婷,成璐,等.支持多故障定位的探测站点部署方法[J].北京邮电大学学报,2009,32(5):1-5.(MENG L M, HUANG T, CHENG L, et al. Probe station placement for multiple faults localization[J]. Journal of Beijing University of Posts and Telecommunications, 2009, 32(5):1-5.)
[10] HUKERIKAR S, DINIZ P C, LUCAS R F, et al. Opportunistic application-level fault detection through adaptive redundant multithreading[C]//Proceedings of the 2014 International Conference on High Performance Computing & Simulation. Piscataway, NJ:IEEE, 2014:243-250.
[11] GARDNER R D, HARLE D A. Network fault detection:a simplified approach to alarm correlation[C]//Proceedings of the 16th IEEE Global Telecommunications Conference. Washington, DC:IEEE Computer Society, 1997:44-51.
[12] SCHROEDER B, GIBSON G. A large-scale study of failures in high-performance computing systems[J]. IEEE Transactions on Dependable and Secure Computing, 2010, 7(4):337-350.
[13] JAKOBSON G, WEISSMAN M. Real-time telecommunication network management:extending event correlation with temporal constraints[C]//Proceedings of the Fourth International Symposium on Integrated Network Management IV. London:Chapman & Hall, 1995:290-301.
[14] LEMARINIER P, BOUTEILLER A, KRAWEZIK G, et al. Coordinated checkpoint versus message log for fault tolerant MPI[J]. International Journal of High Performance Computing and Networking, 2004, 2(2/3/4):146-155.
[15] SCHROEDER B, GIBSON G A. Understanding failures in petascale computers[C]//Proceedings of the 6th Scientific Discovery through Advanced Computing Conference. Bristol:IOP Publishing Ltd, 2007:2022-2032.
[16] 武林平,孟丹,梁毅,等.LUNF——基于节点失效特征的机群作业调度策略[J].计算机研究与发展,2005,42(6):1000-1005.(WU L P, MENG D, LIANG Y, et al. LUNF-a cluster job schedule strategy using characterization of nodes' failure[J]. Journal of Computer Research and Development, 2005, 42(6):1000-1005.)
[17] BAILTY D, HARRIS T, SAPHIR W, et al. The NAS parallel benchmarks 2.0:NAS-95-020[R]. Washington:NASA Ames Research Center, 1995:12.

面向高性能计算的分布式故障定位框架

Distributed fault localization framework for high performance computing

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	宋祥帅, 杨伏长, 谢江, 张武. Graphlet Degree Vector方法的优化与并行[J]. 计算机应用, 2020, 40(2): 398-403.
[2]	龚鸣清, 叶煌, 张鉴, 卢兴敬, 陈伟. 基于ARMv8架构的面向机器翻译的单精度浮点通用矩阵乘法优化[J]. 计算机应用, 2019, 39(6): 1557-1562.
[3]	赵士操, 肖永浩, 段博文, 李于锋. HSWAP:适用于高性能计算环境的数值模拟工作流管理平台[J]. 计算机应用, 2019, 39(6): 1569-1576.
[4]	孙佳敏, 朱嘉富, 杨伏长, 谢江. 大规模生物网络马尔可夫聚类的并行化算法[J]. 计算机应用, 2019, 39(1): 66-71.
[5]	杨伏长, 朱嘉富, 孙佳敏, 谢江. 生物复杂网络motif发现的并行算法[J]. 计算机应用, 2019, 39(1): 72-77.
[6]	徐佳庆, 万文, 蔡东京, 唐付桥, 何杰, 张磊. 高维胖树系统中确定性路由容错策略实现[J]. 计算机应用, 2018, 38(5): 1393-1398.
[7]	王鹏, 周岩. 面向高性能应用的MPI大数据处理[J]. 计算机应用, 2018, 38(12): 3496-3499.
[8]	吕宏武, 谷雷, 王慧强, 邹世辰, 冯光升. 分层检查点的近似最优周期计算模型[J]. 计算机应用, 2017, 37(1): 103-107.
[9]	赵灿明, 李祝红, 陶磊, 张信明. 基于故障传播模型与监督学习的电力通信网络故障定位[J]. 计算机应用, 2016, 36(4): 905-908.
[10]	刘盼盼, 洪旭东, 郭剑毅, 余正涛, 文永华, 陈玮. 基于灰色关联分析的中文新闻事件关联性识别[J]. 计算机应用, 2016, 36(2): 408-413.
[11]	熊壬浩, 刘羽. A^*算法的改进及并行化[J]. 计算机应用, 2015, 35(7): 1843-1848.
[12]	吴洁璇, 陈振杰, 张云倩, 骈宇哲, 周琛. 多核CPU下的K-means遥感影像分类并行方法[J]. 计算机应用, 2015, 35(5): 1296-1301.
[13]	王克朝, 王甜甜, 任向民, 贾宗福. 失效上下文统计分析的软件故障定位方法[J]. 计算机应用, 2015, 35(3): 882-885.
[14]	付朝江, 陈洪均. 基于残余平滑预处理共轭梯度算法的有限元并行计算[J]. 计算机应用, 2015, 35(12): 3387-3391.
[15]	束俊辉张武薛倩斐谢江. 基于消息传递接口的大规模生物网络比对并行化算法[J]. 计算机应用, 2014, 34(11): 3117-3120.