关键词: 高性能计算, 消息传递, 故障定位, 事件关联, 主动探测

Abstract: In order to solve the problem of high difficulty and poor real-time in fault localization for high performance computing system, a message-passing based fault localization framework (MPFL) is proposed, which includes tree-based fault detection (TFD) and tree-based fault analysis (TFA) algorithms. Firstly, when the parallel application is initialized, the fault localization tree (FLT) is obtained by logically dividing the nodes, and the fault localization tasks are distributed to different nodes. Then, if the components such as message-passing library, operating system detect the abnormal state of a node, the TFD algorithm is used to analyze the FLT, and the node responsible for receiving the abnormal state is selected according to factors such as load balancing and performance cost. Finally, the node uses TFA algorithm to derive fault by reasoning the received abnormal state. The TFA algorithm uses rule-based event correlation and designs lightweight active probing based on message-passing, which improves the accuracy of fault analysis. The experimental evaluation is performed on a typical cluster, which demonstrates the capability of MPFL by locating the simulated node outages fault. Additionally, the results on the NPB-FT and NPB-IS benchmarks show that the MPFL service does not affect the performance of an application in practice.

Key words: Keywords: high performance computing, message-passing, fault localization, event correlation, active probing