Interpretation method based on selective Softmax gradient for layer-wise relevance propagation

doi:10.11772/j.issn.1001-9081.2025070906

Journal of Computer Applications ›› 2026, Vol. 46 ›› Issue (7): 2111-2118.DOI: 10.11772/j.issn.1001-9081.2025070906

• Artificial intelligence • Previous Articles

Interpretation method based on selective Softmax gradient for layer-wise relevance propagation

Chong CHEN¹, Hongyang ZOU¹, Jing CAO¹, Wenjing JIANG¹, Jie CHEN¹, Fumin GAO²()

^1.College of Artificial Intelligence，China University of Petroleum （Beijing），Beijing 102249，China
^2.College of Safety and Ocean Engineering，China University of Petroleum （Beijing），Beijing 102249，China

Received:2025-08-11 Revised:2025-11-06 Accepted:2025-11-10 Online:2025-12-22 Published:2026-07-10
Contact: Fumin GAO
About author:CHEN Chong， born in 1987， Ph. D.， associate professor. His research interests include machine learning and its interpretability， numerical modeling.
ZOU Hongyang， born in 2001， M. S. candidate. His research interests include interpretability of neural network models.
CAO Jing， born in 2002， M. S. candidate. His research interests include interpretability of neural network models.
JIANG Wenjing， born in 2002， M. S. Her research interests include data assimilation.
CHEN Jie， born in 1997， M. S. Her research interests include interpretability of deep learning models.
Supported by:
National Key Research and Development Program of China(2022YFC2803700);Oil&Gas Major Project(2024ZD1403305)

基于选择性Softmax梯度的分层相关性传播解释方法

陈冲¹, 邹宏杨¹, 曹靖¹, 蒋文静¹, 陈杰¹, 高富民²()

^1.中国石油大学（北京）人工智能学院，北京 102249
^2.中国石油大学（北京）安全与海洋工程学院，北京 102249

通讯作者: 高富民
作者简介:陈冲（1987—），男，河北衡水人，副教授，博士，CCF会员，主要研究方向：机器学习及其可解释性、数值模拟
邹宏杨（2001—），男，四川成都人，硕士研究生，主要研究方向：神经网络模型的可解释性
曹靖（2002—），男，北京人，硕士研究生，主要研究方向：神经网络模型的可解释性
蒋文静（2002—），女，湖南常德人，硕士，主要研究方向：数据同化
陈杰（1997—），女，山东菏泽人，硕士，主要研究方向：深度学习可解释性
基金资助:
国家重点研发计划项目(2022YFC2803700);油气重大专项(2024ZD1403305)

Abstract

Abstract:

The inherent “black-box” nature of neural network models leads to opaque internal logic and behavioral decision， restricting their application in critical areas. To address this issue， based on the Layer-wise Relevance Propagation （LRP） method， the interpretability of image classification models such as VGG16 and ResNet50 was researched， and an interpretation method for image classification models named SSGLRP （Selective Softmax Gradient for Layer-wise Relevance Propagation） was proposed. By introducing activation values for the positive gradients of output neurons and modifying the initial relevance values for non-target classes， the problem of LRP heatmaps containing noise and lacking class discrimination was effectively solved. The SSGLRP method was quantitatively evaluated using maximum patch masking and fixed-point game experiments， with classic LRP， Selective Layer-wise Relevance Propagation （SLRP）， and Softmax Gradient Layer-wise Relevance Propagation （SGLRP） as baseline methods. The results of the maximum patch masking experiments show that on VGG16， the prediction change obtained by SSGLRP is on average 91.8%， 60.3%， and 34.2% higher than those obtained by LRP， SLRP， and SGLRP， respectively； on ResNet50， the improvements are 61.9%， 64.9%， and 33.2%， respectively. The SSGLRP method has higher class discrimination ability and less noise with superior performance in interpreting VGG16 and ResNet50 models.

Key words: neural network, interpretation method, class discrimination, Layer-wise Relevance Propagation (LRP), image classification

摘要：

神经网络模型固有的“黑盒”性质导致它们的内部逻辑与行为决策不透明，限制了它们在关键领域的应用。针对该问题，基于分层相关传播（LRP）方法，对VGG16和ResNet50等图像分类模型的可解释性展开研究，并提出一种针对图像分类模型的可解释性方法SSGLRP （Selective Softmax Gradient for Layer-wise Relevance Propagation）。该方法在LRP中引入输出神经元正梯度的激活值与修正非目标类的初始相关性值，解决了LRP方法解释结果的热力图中含有噪声且缺乏类别区分性问题。以经典LRP、选择性分层相关传播（SLRP）、Softmax梯度分层相关传播（SGLRP）为基线方法，采用最大补丁遮掩和定点游戏实验对SSGLRP方法进行定量评估。最大补丁遮掩的实验结果表明，在VGG16上，SSGLRP的预测值变化量比LRP、SLRP和SGLRP分别平均高出91.8%、60.3%和34.2%；在ResNet50上则分别高出61.9%、64.9%和33.2%。SSGLRP方法具有更高的类别区别能力和更少的噪声，且解释VGG16和ResNet50模型的效果更好。

关键词: 神经网络, 解释方法, 类别区分性, 分层相关传播, 图像分类

CLC Number:

TP181

Chong CHEN, Hongyang ZOU, Jing CAO, Wenjing JIANG, Jie CHEN, Fumin GAO. Interpretation method based on selective Softmax gradient for layer-wise relevance propagation[J]. Journal of Computer Applications, 2026, 46(7): 2111-2118.

陈冲, 邹宏杨, 曹靖, 蒋文静, 陈杰, 高富民. 基于选择性Softmax梯度的分层相关性传播解释方法[J]. 《计算机应用》唯一官方网站, 2026, 46(7): 2111-2118.

Figures/Tables 15

Fig. 1 Process of SLRP

Fig. 2 Process of SGLRP

Fig. 3 Illustration of ImageNet dataset

Fig. 4 Illustration of CUB-200-2011 dataset

Fig. 5 Evaluation of maximum patch masking

Fig. 6 Concept of “energy” in fix-point game evaluation

Fig. 7 Interpretation of results of VGG16 model

Fig. 8 Class discrimination of interpretation results

Fig. 9 Interpretation of results of ResNet50 model

Fig. 10 Prediction changes of VGG16 model

Fig. 11 Prediction changes of ResNet50 model

Tab. 1 Prediction changes of VGG16 model using different interpretation methods

补丁大小	LRP	SLRP	SGLRP	SSGLRP
1	0.000 05	0.000 39	0.000 64	0.001 61
3	0.000 60	0.003 47	0.004 10	0.005 01
5	0.000 93	0.003 86	0.004 91	0.013 66
7	0.001 58	0.005 25	0.015 69	0.017 99
9	0.002 14	0.009 70	0.017 19	0.020 38

Tab. 2 Prediction changes of ResNet50 model using different interpretation methods

补丁大小	LRP	SLRP	SGLRP	SSGLRP
1	0.000 02	0.000 05	0.000 09	0.000 34
3	0.000 02	0.000 13	0.002 58	0.003 89
5	0.002 18	0.001 92	0.004 30	0.006 38
7	0.006 34	0.005 31	0.007 62	0.009 68
9	0.008 66	0.007 47	0.009 76	0.010 27

Fig. 12 Pointing accuracies of VGG16 model at different energies

Fig. 13 Pointing accuracies of ResNet50 model at different energies

References 22

[1]	刘广，李海旭，侯腾，等.基于检索任务和改进ResNet的机场跑道异物目标识别方法［J］.计算机应用， 2024， 44（）： 309-313.
	Liu Guang， Li Haixu， Hou Teng， et al. Foreign object debris target recognition method for airport runways based on retrieval task and improved ResNet ［J］. Journal of Computer Applications， 2024， 44（）： 309-313.
[2]	王璐，刘东，刘卫光.可变形卷积网络的解释性研究及其在蝴蝶物种识别模型中的应用［J］.计算机应用， 2025， 45（1）： 261-274.
	Wang Lu， Liu Dong， Liu Weiguang. Interpretability study on deformable convolutional network and its application in butterfly species recognition models ［J］. Journal of Computer Applications， 2025， 45（1）： 261-274.
[3]	Yang M， Jiao L， Liu F， et al. Transferred deep learning-based change detection in remote sensing images ［J］. IEEE Transactions on Geoscience and Remote Sensing， 2019， 57（9）： 6960-6973.
[4]	Wang Z， Liu J. A review of object detection based on convolutional neural network ［C］// CCC 2017. Piscataway： IEEE， 2017： 11104-11109.
[5]	周志华，陈世福.神经网络规则抽取［J］.计算机研究与发展， 2002， 39（4）： 398-405.
	Zhou Zhihua， Chen Shifu. Rule extraction from neural networks ［J］. Journal of Computer Research and Development， 2002， 39（4）： 398-405.
[6]	陈冲，陈杰，张慧，等.深度学习可解释性综述［J］.计算机科学， 2023， 50（5）： 52-63.
	Chen Chong， Chen Jie， Zhang Hui， et al. Review on interpretability of deep learning ［J］. Computer Science， 2023， 50（5）： 52-63.
[7]	Bach S， Binder A， Montavon G， et al. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation ［J］. PLoS ONE， 2015， 10（7）： No.e0130140.
[8]	Jung Y J， Han S H， Choi H J. Explaining CNN and RNN using selective layer-wise relevance propagation ［J］. IEEE Access， 2021， 9： 18670-18681.
[9]	Iwana B K， Kuroki R， Uchida S. Explaining convolutional neural networks using softmax gradient layer-wise relevance propagation ［C］// ICCV Workshops 2019. Piscataway： IEEE， 2019： 4176-4185.
[10]	陈珂锐，孟小峰.机器学习的可解释性［J］.计算机研究与发展， 2020， 57（9）： 1971-1986.
	Chen Kerui， Meng Xiaofeng. Interpretation and understanding in machine learning ［J］. Journal of Computer Research and Development， 2020， 57（9）： 1971-1986.
[11]	Barredo Arrieta A， Díaz-Rodríguez N， Del Ser J， et al. Explainable Artificial Intelligence （XAI）： concepts， taxonomies， opportunities and challenges toward responsible AI ［J］. Information Fusion， 2020， 58： 82-115.
[12]	Simonyan K， Vedaldi A， Zisserman A. Deep inside convolutional networks： visualising image classification models and saliency maps ［PP/OL］. V2. arXiv （2014-04-19）［2025-06-20］. .
[13]	Shrikumar A， Greenside P， Kundaje A. Learning important features through propagating activation differences ［C］// ICML 2017. New York： JMLR.org， 2017： 3145-3153.
[14]	Zhou B， Khosla A， Lapedriza A， et al. Learning deep features for discriminative localization ［C］// CVPR 2016. Piscataway： IEEE， 2016： 2921-2929.
[15]	Selvaraju R R， Cogswell M， Das A， et al. Grad-CAM： visual explanations from deep networks via gradient-based localization ［C］// ICCV 2017. Piscataway： IEEE， 2017： 618-626.
[16]	Chattopadhay A， Sarkar A， Howlader P， et al. Grad-CAM++： generalized gradient-based visual explanations for deep convolutional networks ［C］// WACV 2018. Piscataway： IEEE， 2018： 839-847.
[17]	Montavon G， Lapuschkin S， Binder A， et al. Explaining nonlinear classification decisions with deep Taylor decomposition ［J］. Pattern Recognition， 2017， 65： 211-222.
[18]	Montavon G， Binder A， Lapuschkin S， et al. Layer-wise relevance propagation： an overview ［M］// Samek W， Montavon G， Vedaldi A， et al.Explainable AI： interpreting， explaining and visualizing deep learning， LNCS 11700. Cham： Springer， 2019： 119-134.
[19]	He K， Zhang X， Ren S， et al. Identity mappings in deep residual networks ［C］// ECCV 2016. Cham： Springer， 2016： 630-645.
[20]	Wah C， Branson S， Welinder P， et al. Caltech-UCSD birds-200-2011 dataset ［DS/OL］. ［2025-05-11］. .
[21]	Gu J， Yang Y， Tresp V. Understanding individual decisions of CNNs via contrastive backpropagation ［C］// ACCV 2018. Cham： Springer， 2019： 119-134.
[22]	Zhang J， Bargal S A， Lin Z， et al. Top-down neural attention by excitation backprop ［J］. International Journal of Computer Vision， 2018， 126（10）： 1084-1102.

Interpretation method based on selective Softmax gradient for layer-wise relevance propagation

基于选择性Softmax梯度的分层相关性传播解释方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 22

Related Articles 15

Recommended Articles

Metrics

[1]	Yue ZHANG, Ying WANG. Scalable dynamic graph representation learning with neighborhood memory [J]. Journal of Computer Applications, 2026, 46(7): 2104-2110.
[2]	Min LU, Hui ZHOU. Airport gate assignment algorithm based on node prediction in conflict graph of assigned activities [J]. Journal of Computer Applications, 2026, 46(6): 1913-1921.
[3]	Yu CHEN, Shuaikang QI, Liwei XU, Haotian ZHU. Social bot detection framework fusing multi-scale wavelet enhancement and self-supervised learning [J]. Journal of Computer Applications, 2026, 46(6): 1756-1766.
[4]	Bingqing LI, Binhao HUANG, Yubei TANG, Baili ZHANG. Quality of service prediction model for data sparsity and cold start problems [J]. Journal of Computer Applications, 2026, 46(6): 1829-1835.
[5]	Hang QI, Tingting DONG, Yongqiang NAI, Xian MO. Contrastive collaborative filtering method based on graph diffusion generation and adaptive sampling [J]. Journal of Computer Applications, 2026, 46(6): 1818-1828.
[6]	Kun FU, Haoyu WEI, Weijing LIU, Xing DANG, Zezheng LIU, Jianwei LI. Graph neural network framework for topology semantic dual-domain collaboration [J]. Journal of Computer Applications, 2026, 46(5): 1378-1387.
[7]	Xiaobo QI, Jing ZHANG, Ying SHI, Hui QI, Hangyuan DU. Multiple active learning method based on concept drift detection [J]. Journal of Computer Applications, 2026, 46(5): 1388-1396.
[8]	Xinyao LIU, Jun LIANG, Jiahao LONG, Renliang YAN. Fine-grained Chinese herbal medicine image classification based on feature fusion and channel information compensation [J]. Journal of Computer Applications, 2026, 46(5): 1677-1683.
[9]	Fengwei CHENG, Bingqi ZHANG, Guohua XU, Wenjian WANG. Competitive loss-driven generative imbalanced node classification [J]. Journal of Computer Applications, 2026, 46(5): 1475-1481.
[10]	Chi ZHANG, Xianjing MENG, Changhao DOU, Qian WANG, Leilei GENG, Xiaoming XI. MD-FVR： cascaded finger vein recognition network based on multi-domain feature fusion [J]. Journal of Computer Applications, 2026, 46(5): 1658-1666.
[11]	Xumeng DOU, Bin XIE, Zhaohui ZHANG, Zhengang ZHAO, Hanyu DUAN, Aolei GUO. Drug-target interaction prediction based on structure-network collaborative features and grid-attention enhanced Kolmogorov-Arnold network [J]. Journal of Computer Applications, 2026, 46(4): 1344-1353.
[12]	Shengwei XU, Jianbo WANG, Jijie HAN, Yijie BAI. Face forgery detection method based on tri-branch feature extraction [J]. Journal of Computer Applications, 2026, 46(4): 1292-1299.
[13]	Rilong WANG, Zhenping LI, Xiaosong LI, Qiang GAO, Ya HE, Yong ZHONG, Yingxiao ZHAO. Multi-Agent collaborative knowledge reasoning framework [J]. Journal of Computer Applications, 2026, 46(3): 708-714.
[14]	Kaiguang MA, Xuebin CHEN, Yinlong JIAN, Liu WANG, Yuan GAO. Network intrusion detection based on hybrid sequence model and federated class balance algorithm [J]. Journal of Computer Applications, 2026, 46(3): 857-866.
[15]	Yongwei JIANG, Xiaoqing CHEN, Linjie FU. Elastic medical image registration model with high-frequency preservation based on spectrum decomposition [J]. Journal of Computer Applications, 2026, 46(3): 924-932.