Abstract: In recent years, neural network has been widely applied in many fields such as medical treatment, communication, and security, driving the growth of scale and the industrial application of deep learning. However, the inherent ‘black-box’ nature and opaque learning process of neural network models limit people's deep understanding of its internal logic and behavior, making it difficult to fully trust the model results. To address these issues, based on the Layer-wise Relevance Propagation (LRP), this paper delves into the interpretability of image classification models such as VGG16 and ResNet50. An interpretability method named Selective Softmax Gradient for Layer-wise Relevance Propagation (SSGLRP) was proposed. The proposed methos effectively addresses the problems that the heatmaps of the LRP interpretation results contain noise and lack class discrimination by introducing the activation values of the positive gradients of output neurons and modifying the initial relevance values of non-target classes. Additionally, by adjusting the initial relevance values of non-target classes, the SSGLRP method can eliminate non-target class objects in the heatmap and generate heatmaps with class discrimination. The effectiveness of this method is quantitatively evaluated through experiments such as maximum patch masking and pointing game. The results of the maximum patch masking experiment show that the average change in model prediction values by perturbing input pixels according to the SSGLRP method is 77.0%, 62.6%, and 33.5% higher than those obtained using LRP, SLRP, and SGLRP, respectively. The experimental results demonstrate that the SSGLRP possesses a higher class discrimination ability and less noise. It exhibits superior performance in interpreting the VGG16 and ResNet50 model.

Key words: neural network, interpretation method, class-discrimination, Layer-wise Relevance Propagation (LRP), image classification

摘要： 近年来，神经网络模型被广泛运用于医疗、通信、安全等领域，推动深度学习向规模化与产业化方向发展。然而，神经网络模型固有的“黑盒”性质导致其内部逻辑与行为决策不透明，限制了其在关键领域的应用。针对该问题，基于分层相关传播(LRP)方法，对VGG16、ResNet50等图像分类模型的可解释性展开研究，提出了选择性Softmax梯度的分层相关传播(SSGLRP)解释方法。该方法通过在LRP中引入输出神经元正梯度的激活值与修正非目标类的初始相关性值，解决LRP方法解释结果的热力图中含有噪声且缺乏类别区分性问题。以经典LRP、选择性分层相关传播(SLRP)、Softmax梯度分层相关传播(SGLRP)为基线方法，采用最大补丁遮掩和定点游戏实验对SSGLRP方法进行定量评估。SSGLRP方法的平均模型预测变化量比LRP、SLRP和SGLRP分别高77.0%、62.6%和33.5%。实验结果表明，SSGLRP方法具有更高的类别区别能力和更少的噪声，解释VGG16、ResNet50网络模型的效果更好。

关键词: 神经网络, 解释方法, 类别区分性, 分层相关传播, 图像分类