Interpretability study on deformable convolutional network and its application in butterfly species recognition models

doi:10.11772/j.issn.1001-9081.2023121776

Abstract

Abstract:

In recent years， Deformable Convolutional Network （DCN） has been widely applied in fields such as image recognition and classification. However， research on the interpretability of this model is relatively limited， and its applicability lacks sufficient theoretical support. To address these issues， this paper proposed an interpretability study of DCN and its application in butterfly species recognition model. Firstly， deformable convolution was introduced to improve the VGG16， ResNet50， and DenseNet121 （Dense Convolutional Network121） classification models. Secondly， visualization methods such as deconvolution and Class Activation Mapping （CAM） were used to compare the feature extraction capabilities of deformable convolution and standard convolution. The results of ablation experiments show that deformable convolution performs better when used in the lower layers of the neural network and not continuously. Thirdly， the Saliency Removal （SR） method was proposed to uniformly evaluate the performance of CAM and the importance of activation features. By setting different removal thresholds and other perspectives， the objectivity of the evaluation is improved. Finally， based on the evaluation results， the FullGrad （Full Gradient-weighted） explanation model was used as the basis for the recognition judgment. Experimental results show that on the Archive_80 dataset， the accuracy of the proposed D_v2-DenseNet121 reaches 97.03%， which is 2.82 percentage points higher than that of DenseNet121 classification model. It can be seen that the introduction of deformable convolution endows the neural network model with the ability to extract invariant features and improves the accuracy of the classification model.

Key words: Deformable Convolutional Network (DCN), interpretability, butterfly species recognition, Class Activation Mapping (CAM), Saliency Removal (SR)

摘要：

近年来，可变形卷积网络（DCN）广泛运用于图像识别和分类等领域，然而对该模型的可解释性研究较为有限，它的适用性缺乏充分理论支持。针对上述问题，提出DCN的解释性研究及其在蝴蝶物种识别模型中的应用。首先，引入可变形卷积对VGG16、ResNet50和DenseNet121 （Dense Convolutional Network121）分类模型进行改进；其次，采用反卷积和类激活映射（CAM）等可视化手段来对比可变形卷积和标准卷积在特征提取能力上的差异，且通过消融实验结果表明可变形卷积在神经网络的较低层且不连续使用时效果更佳；再次，提出显著性移除（SR）并对CAM的性能和激活特征重要性进行统一评价，同时通过设置不同的移除阈值等多个角度，提高评价的客观性；最后，基于评价结果更高的FullGrad （Full Gradient-weighted）解释模型识别的判断依据。实验结果显示，在Archive_80数据集上，所提出的D_v2-DenseNet121的准确率达到97.03%，相较于DenseNet121分类模型提高了2.82个百分点。可见，可变形卷积的引入赋予了神经网络模型不变性特征提取能力，并提高了分类模型的准确率。

关键词: 可变形卷积网络, 可解释性, 蝴蝶物种识别, 类激活映射, 显著性移除

CLC Number:

TP183

Lu WANG, Dong LIU, Weiguang LIU. Interpretability study on deformable convolutional network and its application in butterfly species recognition models[J]. Journal of Computer Applications, 2025, 45(1): 261-274.

王璐, 刘东, 刘卫光. 可变形卷积网络的解释性研究及其在蝴蝶物种识别模型中的应用[J]. 《计算机应用》唯一官方网站, 2025, 45(1): 261-274.

Figures/Tables 22

References 47

1	DAI J， QI H， XIONG Y， et al. Deformable convolutional networks ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 764-773.
2	ZHU X， HAN H， LIN S， et al. Deformable ConvNets V2： more deformable， better results ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 9300-9308.
3	刘卫光，刘东，王璐.可变形卷积网络研究综述［J］.计算机科学与探索， 2023， 17（7）： 1549-1564.
	LIU W G， LIU D， WANG L. Survey of deformable convolutional networks ［J］. Journal Frontiers of Computer Science and Technology， 2023， 17（7）： 1549-1564.
4	LAI S C， TAN H K， LEE P Y. 3D deformable convolution for action classification in videos ［C］// Proceedings of the SPIE 11766， 2021 International Workshop on Advanced Imaging Technology. Bellingham， WA： SPIE， 2021： No.117660R.
5	谢娟英，鲁银圆，孔维轩，等.基于改进RetinaNet的自然环境中蝴蝶种类识别［J］.计算机研究与发展， 2021， 58（8）： 1686-1704.
	XIE J Y， LU Y Y， KONG W X， et al. Butterfly species identification from natural environment based on improved RetinaNet ［J］. Journal of Computer Research and Development， 2021， 58（8）： 1686-1704.
6	赵云霄，钱宇华，王克琪.基于可变形卷积的多人人体姿态估计［J］.模式识别与人工智能， 2020， 33（10）： 944-950.
	ZHAO Y X， QIAN Y H， WANG K Q. Multi-person human pose estimation based on deformable convolution ［J］. Pattern Recognition and Artificial Intelligence， 2020， 33（10）： 944-950.
7	CHEN T， PU T， WU H， et al. Cross-domain facial expression recognition： a unified evaluation benchmark and adversarial graph learning ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 44（12）： 9887-9903.
8	YASMIN R， DAS A， ROZARIO L J， et al. Butterfly detection and classification techniques： a review ［J］. Intelligent Systems with Applications， 2023， 18： No.200214.
9	KANG S H， SONG S H， LEE S H. Identification of butterfly species with a single neural network system ［J］. Journal of Asia-Pacific Entomology， 2012， 15（3）： 431-435.
10	KANG S H， CHO J H， LEE S H. Identification of butterfly based on their shapes when viewed from different angles using an artificial neural network ［J］. Journal of Asia-Pacific Entomology， 2014， 17（2）： 143-149.
11	KAYA Y， KAYCI L. Application of artificial neural network for automatic detection of butterfly species using color and texture features ［J］. The Visual Computer， 2014， 30（1）： 71-79.
12	KAYA Y， KAYCI L， TEKIN R， et al. Evaluation of texture features for automatic detecting butterfly species using extreme learning machine ［J］. Journal of Experimental and Theoretical Artificial Intelligence， 2014， 26（2）： 267-281.
13	KAYA Y， FARUK ERTUĞRUL Ö， TEKIN R. Two novel local binary pattern descriptors for texture analysis ［J］. Applied Soft Computing， 2015， 34： 728-735.
14	LIN Z， JIA J， GAO W， et al. Increasingly specialized perception network for fine-grained visual categorization of butterfly specimens ［J］. IEEE Access， 2019， 7： 123367-123392.
15	LIN Z， JIA J， GAO W， et al. Fine-grained visual categorization of butterfly specimens at sub-species level via a convolutional neural network with skip-connections ［J］. Neurocomputing， 2020， 384： 295-313.
16	CARVAJAL J A， ROMERO D G， SAPPA A D. Fine-tuning based deep convolutional networks for lepidopterous genus recognition ［C］// Proceedings of the 2016 Iberoamerican Congress on Pattern Recognition. Cham： Springer， 2017： 467-475.
17	LIU S， DENG W. Very deep convolutional neural network based image classification using small training sample size ［C］// Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition. Piscataway： IEEE， 2015： 730-734.
18	KAMARON ARZAR N N， SABRI N， MOHD JOHARI N F， et al. Butterfly species identification using Convolutional Neural Network （CNN）［C］// Proceedings of the 2019 IEEE International Conference on Automatic Control and Intelligent Systems. Piscataway： IEEE， 2019： 221-224.
19	SZEGEDY C， LIU W， JIA Y， et al. Going deeper with convolutions ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 1-9.
20	ALMRYAD A， KUTUCU H. Automatic identification for field butterflies by convolutional neural networks ［J］. Engineering Science and Technology， an International Journal， 2020， 23（1）： 189-195.
21	HE K， ZHANG X， REN S， et al. Deep residual learning for image recognition ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778.
22	THEIVAPRAKASHAM H. Identification of Indian butterflies using deep convolutional neural network ［J］. Journal of Asia-Pacific Entomology， 2021， 24（1）： 329-340.
23	HUANG G， LIU Z， VAN DER MAATEN L， et al. Densely connected convolutional networks ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2261-2269.
24	FATHIMATHUL R P P， ORBAN R， VADIVEL K S， et al. A novel method for the classification of butterfly species using pre-trained CNN models ［J］. Electronics， 2022， 11（13）： No.2016.
25	谢娟英，侯琦，史颖欢，等.蝴蝶种类自动识别研究［J］.计算机研究与发展， 2018， 55（8）： 1609-1618.
	XIE J Y， HOU Q， SHI Y H， et al. The automatic identification of butterfly species ［J］. Journal of Computer Research and Development， 2018， 55（8）： 1609-1618.
26	XIE J， LU Y， WU Z， et al. Investigations of butterfly species identification from images in natural environments ［J］. International Journal of Machine Learning and Cybernetics， 2021， 12（8）： 2431-2442.
27	LIANG B， WU S， XU K， et al. Butterfly detection and classification based on integrated YOLO algorithm ［C］// Proceedings of the 2019 International Conference on Genetic and Evolutionary Computing， AISC 1107. Singapore： Springer， 2020： 500-512.
28	SEMIH KAYHAN O， VAN GEMERT J C. On translation invariance in CNNs： convolutional layers can exploit absolute spatial location ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 14262-14273.
29	周飞燕，金林鹏，董军.卷积神经网络研究综述［J］.计算机学报， 2017， 40（6）： 1229-1251.
	ZHOU F Y， JIN L P， DONG J. Review of convolutional neural network ［J］. Chinese Journal of Computers， 2017， 40（6）： 1229-1251.
30	ZEILER M D， FERGUS R. Visualizing and understanding convolutional networks ［C］// Proceedings of the 2014 European Conference on Computer Vision， LNCS 8689. Cham： Springer， 2014： 813-833.
31	ZHOU B， KHOSLA A， LAPEDRIZA A， et al. Learning deep features for discriminative localization ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 2921-2929.
32	SELVARAJU R R， COGSWELL M， DAS A， et al. Grad-CAM： visual explanations from deep networks via gradient-based localization ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 618-626.
33	CHATTOPADHYAY A， SARKAR A， HOWLADER P， et al. Grad-CAM++： generalized gradient-based visual explanations for deep convolutional networks ［C］// Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2018： 839-847.
34	WANG H， WANG Z， DU M， et al. Score-CAM： score-weighted visual explanations for convolutional neural networks ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2020： 111-119.
35	JIANG P T， ZHANG C B， HOU Q， et al. LayerCAM： exploring hierarchical class activation maps for localization ［J］. IEEE Transactions on Image Processing， 2021， 30： 5875-5888.
36	BANY MUHAMMAD M， YEASIN M. Eigen-CAM： visual explanations for deep convolutional neural networks ［J］. SN Computer Science， 2021， 2： No.47.
37	SRINIVAS S， FLEURET F. Full-gradient representation for neural network visualization ［C］// Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2019： 4124-4133.
38	魏靖，王玉亭，袁会珠，等.基于深度学习与特征可视化方法的草地贪夜蛾及其近缘种成虫识别［J］.智慧农业（中英文）， 2020， 2（3）： 75-85.
	WEI J， WANG Y T， YUAN H Z， et al. Identification and morphological analysis of adult Spodoptera frugiperda and its close related species using deep learning ［J］. Smart Agriculture （Chinese and English）， 2020， 2（3）： 75-85.
39	MONTAVON G， SAMEK W， MÜLLER K R. Methods for interpreting and understanding deep neural networks ［J］. Digital Signal Processing， 2018， 73： 1-15.
40	MONTAVON G， LAPUSCHKIN S， BINDER A， et al. Explaining nonlinear classification decisions with deep Taylor decomposition ［J］. Pattern Recognition， 2017， 65： 211-222.
41	LUNDBERG S M， LEE S I. A unified approach to interpreting model predictions ［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2017： 4768-4777.
42	SUNDARARAJAN M， TALY A， YAN Q. Axiomatic attribution for deep networks ［C］// Proceedings of the 34th International Conference on Machine Learning. New York： JMLR.org， 2017： 3319-3328.
43	HOOKER S， ERHAN D， KINDERMANS P J， et al. A benchmark for interpretability methods in deep neural networks ［C］// Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2019： 9734-9745.
44	TOMSETT R， HARBORNE D， CHAKRABORTY S， et al. Sanity checks for saliency metrics ［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto： AAAI Press， 2019： 6021-6029.
45	YONA G， GREENFELD D. Revisiting sanity checks for saliency maps ［EB/OL］. ［2023-10-22］. .
46	RONG Y， LEEMANN T， BORISOV V， et al. A consistent and efficient evaluation strategy for attribution methods ［C］// Proceedings of the 39th International Conference on Machine Learning. New York： JMLR.org， 2022： 18770-18795.
47	PETSIUK V， DAS A， SAENKO K. RISE： randomized input sampling for explanation of black-box models ［EB/OL］. ［2023-11-22］. .

种类	数量最小值	数量最大值	图像数
合计	68	96	6 497
标本图	11	32	1 410
生态图	46	75	5 087

种类	数量最小值	数量最大值	图像数
合计	68	96	6 497
标本图	11	32	1 410
生态图	46	75	5 087

网络模型	可变形卷积数	可变形卷积位置	准确率/%
ResNet50			93.67
D_v2-ResNet50@L0	1	Conv1	95.79
D_v2-ResNet50@L1	1	layer1［0］.conv2	95.83
		layer1［1］.conv2	95.88
		layer1［2］.conv2	95.86
D_v2-ResNet50@L2	1	Layer2［0］.conv2	95.78
D_v2-ResNet50@L3	1	Layer3［5］.conv2	95.82
D_v2-ResNet50@L4	1	Layer4［0］.conv2	95.52
D_v2-ResNet50@L1	2	Layer1［0］.conv2；Layer1［1］.conv2	95.68
D_v2-ResNet50@L1	2	Layer1［0］.conv2；Layer1［2］.conv2	95.84
D_v2-ResNet50@L0~L1	2	Conv1；Layer1［1］.conv2	95.89
D_v2-ResNet50@L0~L1	2	Conv1；Layer1［2］.conv2	95.98
D_v2-ResNet50@L1~L3	3	Layer1［0］.conv2；Layer2［2］.conv2；Layer3［5］.conv2	95.99
D_v2-ResNet50@L0~L3	4	Conv1；Layer1［2］.conv2；Layer2［2］.conv2；Layer3［0］.conv2	95.77
D_v2-ResNet50@L0~L3	4	Conv1；Layer1［2］.conv2；Layer2［3］.conv2；Layer3［5］.conv2	95.97
D_v2-ResNet50@L0~L2	3	Conv1；Layer1［1］.conv2；Layer2［0］.conv2	96.07
		Layer1［0］.conv2；Layer2［2］.conv2；Layer3［3］.conv2	96.01
		Conv1；Layer1［1］.conv2；Layer2［3］.conv2	96.28

网络模型	可变形卷积数	可变形卷积位置	准确率/%
ResNet50			93.67
D_v2-ResNet50@L0	1	Conv1	95.79
D_v2-ResNet50@L1	1	layer1［0］.conv2	95.83
		layer1［1］.conv2	95.88
		layer1［2］.conv2	95.86
D_v2-ResNet50@L2	1	Layer2［0］.conv2	95.78
D_v2-ResNet50@L3	1	Layer3［5］.conv2	95.82
D_v2-ResNet50@L4	1	Layer4［0］.conv2	95.52
D_v2-ResNet50@L1	2	Layer1［0］.conv2；Layer1［1］.conv2	95.68
D_v2-ResNet50@L1	2	Layer1［0］.conv2；Layer1［2］.conv2	95.84
D_v2-ResNet50@L0~L1	2	Conv1；Layer1［1］.conv2	95.89
D_v2-ResNet50@L0~L1	2	Conv1；Layer1［2］.conv2	95.98
D_v2-ResNet50@L1~L3	3	Layer1［0］.conv2；Layer2［2］.conv2；Layer3［5］.conv2	95.99
D_v2-ResNet50@L0~L3	4	Conv1；Layer1［2］.conv2；Layer2［2］.conv2；Layer3［0］.conv2	95.77
D_v2-ResNet50@L0~L3	4	Conv1；Layer1［2］.conv2；Layer2［3］.conv2；Layer3［5］.conv2	95.97
D_v2-ResNet50@L0~L2	3	Conv1；Layer1［1］.conv2；Layer2［0］.conv2	96.07
		Layer1［0］.conv2；Layer2［2］.conv2；Layer3［3］.conv2	96.01
		Conv1；Layer1［1］.conv2；Layer2［3］.conv2	96.28

网络模型	准确率/%	参数量/10⁶	浮点运算量/GFLOPs
Inception V3	93.37	20.93	2.85
AlexNet	89.87	54.67	0.71
GoogLeNet	93.94	5.41	1.51
Xception	93.88	9.22	2.41
SqueezeNet V1.1	93.49	0.73	0.28
MobileNet V2	92.37	2.21	2.41
VGG16	92.03	128.35	15.50
ResNet50	93.67	22.57	4.12
DenseNet121	94.21	6.71	2.88