Few-shot object detection combining feature fusion and enhanced attention

doi:10.11772/j.issn.1001-9081.2023030315

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (3): 745-751.DOI: 10.11772/j.issn.1001-9081.2023030315

• Artificial intelligence • Previous Articles Next Articles

Few-shot object detection combining feature fusion and enhanced attention

Xinye LI¹^,², Yening HOU¹(), Yinghui KONG¹^,², Zhiqi YAN¹

^1.Department of Electronic & Communication Engineering，North China Electric Power University，Baoding Hebei 071003，China
^2.Hebei Key Laboratory of Power Internet of Things Technology （North China Electric Power University），Baoding Hebei 071003，China

Received:2023-03-29 Revised:2023-04-27 Accepted:2023-05-05 Online:2023-05-24 Published:2024-03-10
Contact: Yening HOU
About author:LI Xinye， born in 1969， Ph. D.， associate professor. Her research interests include computer vision， deep learning， few-shot learning.
KONG Yinghui， born in 1964， Ph. D.， professor. Her research interests include computer vision， object detection.
YAN Zhiqi， born in 1998， M. S. candidate. His research interests include few-shot object detection.
Supported by:
Science and Technology Program of Hebei Province(SZX2020034)

结合特征融合与增强注意力的少样本目标检测

李新叶¹^,², 侯晔凝¹(), 孔英会¹^,², 燕志旗¹

^1.华北电力大学电子与通信工程系，河北保定 071003
^2.河北省电力物联网技术重点实验室（华北电力大学），河北保定 071003

通讯作者: 侯晔凝
作者简介:李新叶（1969—），女，河北平山人，副教授，博士，主要研究方向：计算机视觉、深度学习、少样本学习
孔英会（1964—），女，河北衡水人，教授，博士，主要研究方向：计算机视觉、目标检测
燕志旗（1998—），男，山西吕梁人，硕士研究生，主要研究方向：少样本目标检测。
基金资助:
河北省科技计划项目(SZX2020034)

Abstract

Abstract:

In order to fully utilize the key information in support features and query features， a few-shot object detection method based on feature fusion and enhanced attention was proposed， namely FFA-FSOD （Feature Fusion and enhanced Attention Few-Shot Object Detection）. Firstly， the iterative Attention Feature Fusion （iAFF） module was introduced to effectively fuse the key features of the support image and the query image. Secondly， the feature enhancement operation was added after the iAFF module， which made full use of the support feature information to enhance the object features in the query image. To avoid the loss of part of the details of the query image after the above two operations， the Multi-Scale Channel Attention Module （MS-CAM） was improved in the iAFF module to capture more context information. Experimental results on MS COCO dataset under 2-way 10-shot condition show that compared with FSOD （Few-Shot Object Detection） method， after adding the iAFF module， feature enhancement operation and improving MS-CAM， FFA-FSOD has mean Average Precision （mAP） increased by 8.0%. Experimental results show that the proposed feature fusion enhancement method pays full attention to the details of features， thus achieving better detection effect of few-shot objects.

Key words: object detection, few-shot, feature fusion, feature enhancement, Faster R-CNN

摘要：

为了更充分地利用支持特征和查询特征中的关键信息，提出一种基于特征融合和增强注意力的少样本目标检测方法FFA-FSOD（Feature Fusion and enhanced Attention Few-Shot Object Detection）。首先引入迭代注意力特征融合（iAFF）模块，以有效融合支持图像和查询图像的关键特征；其次在iAFF模块后添加特征增强操作，充分利用支持特征信息对查询图像中的目标特征进行增强。为避免上述两次处理可能导致的查询图像特征部分细节信息的丢失，对iAFF模块中的多尺度通道注意力模块（MS-CAM）进行改进，以捕获更多的上下文信息。在MS COCO数据集上的实验结果表明，在2-way 10-shot条件下，与小样本目标检测（FSOD）方法相比，加入iAFF模块、特征增强操作并改进MS-CAM后，FFA-FSOD的平均精度均值（mAP）提升了8.0%。实验结果验证了所提特征融合增强方法充分关注到了特征中的细节信息，从而实现了更好的少样本目标检测效果。

关键词: 目标检测, 少样本, 特征融合, 特征增强, Faster R-CNN

CLC Number:

TN911.73

Xinye LI, Yening HOU, Yinghui KONG, Zhiqi YAN. Few-shot object detection combining feature fusion and enhanced attention[J]. Journal of Computer Applications, 2024, 44(3): 745-751.

李新叶, 侯晔凝, 孔英会, 燕志旗. 结合特征融合与增强注意力的少样本目标检测[J]. 《计算机应用》唯一官方网站, 2024, 44(3): 745-751.

Figures/Tables 10

References 22

1	曹家乐，李亚利，孙汉卿，等.基于深度学习的视觉目标检测技术综述［J］.中国图象图形学报，2022，27（6）：1697-1722. 10.11834/jig.220069
	CAO J L， LI Y L， SUN H Q， et al. A survey on deep learning based visual object detection ［J］. Journal of Image and Graphics， 2022，27（6）：1697-1722. 10.11834/jig.220069
2	赵永强，饶元，董世鹏，等. 深度学习目标检测方法综述［J］. 中国图象图形学报， 2020， 25（4）： 629-654. 10.11834/jig.190307
	ZHAO Y Q， RAO Y， DONG S P， et al. Survey on deep learning object detection ［J］. Journal of Image and Graphics， 2020， 25（4）： 629-654. 10.11834/jig.190307
3	张振伟，郝建国，黄健，等. 小样本图像目标检测研究综述［J］. 计算机工程与应用， 2022，58（5）：1-11. 10.3778/j.issn.1002-8331.2109-0405
	ZHANG Z W， HAO J G， HUANG J， et al. Review of few-shot object detection ［J］. Computer Engineering and Applications， 2022， 58（5）： 1-11. 10.3778/j.issn.1002-8331.2109-0405
4	WANG X， HUANG T E， DARRELL T， et al. Frustratingly simple few-shot object detection ［C］// Proceedings of the 37th International Conference on Machine Learning. New York： JMLR.org， 2020： 9919-9928.
5	ZHANG W， WANG Y-X. Hallucination improves few-shot object detection ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 13003-13012. 10.1109/cvpr46437.2021.01281
6	ZHANG G， LUO Z， CUI K， et al. Meta-DETR： few-shot object detection via unified image-level meta-learning ［EB/OL］. ［2022-05-22］. . 10.1109/tpami.2022.3195735
7	FAN Z， MA Y， LI Z， et al. Generalized few-shot object detection without forgetting ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 4525-4534. 10.1109/cvpr46437.2021.00450
8	HU H， BAI S， LI A， et al. Dense relation distillation with context-aware aggregation for few-shot object detection ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 10180-10189. 10.1109/cvpr46437.2021.01005
9	XIAO Y， LEPETIT V， MARLET R. Few-shot object detection and viewpoint estimation for objects in the wild ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2023， 45（3）： 3090-3106.
10	FAN Q， ZHUO W， TANG C-K， et al. Few-shot object detection with Attention-RPN and multi-relation detector ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 4012-4021. 10.1109/cvpr42600.2020.00407
11	LENG J， CHEN T， GAO X， et al. A comparative review of recent few-shot object detection algorithms ［EB/OL］. ［2023-01-22］. .
12	DAI Y， GIESEKE F， OEHMCKE S， et al. Attentional feature fusion ［C］// Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2021： 3559-3568. 10.1109/wacv48630.2021.00360
13	WANG Y-X， RAMANAN D， HEBERT M. Meta-learning to detect rare objects ［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 9924-9933. 10.1109/iccv.2019.01002
14	YAN X， CHEN Z， XU A， et al. Meta R-CNN： towards general solver for instance-level low-shot learning ［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 9576-9585. 10.1109/iccv.2019.00967
15	WU J， LIU S， HUANG D， et al. Multi-scale positive sample refinement for few-shot object detection ［C］// Proceedings of the 2020 European Conference on Computer Vision. Cham： Springer， 2020： 456-472. 10.1007/978-3-030-58517-4_27
16	ZHANG D， PU H， LI F， et al. Few-shot object detection based on the Transformer and high-resolution network ［J］. Computers， Materials & Continua， 2023， 74（2）： 3439-3454. 10.32604/cmc.2023.027267
17	LU Y， CHEN X， WU Z， et al. Decoupled metric network for single-stage few-shot object detection ［J］. IEEE Transactions on Cybernetics， 2023， 53（1）： 514-525. 10.1109/tcyb.2022.3149825
18	WU A， HAN Y， ZHU L， et al. Universal-prototype enhancing for few-shot object detection ［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2021： 9547-9556. 10.1109/iccv48922.2021.00943
19	SUN B， LI B， CAI S， et al. FSCE： few-shot object detection via contrastive proposal encoding ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 7348-7358. 10.1109/cvpr46437.2021.00727
20	ZHU C， CHEN F， AHMED U， et al. Semantic relation reasoning for shot-stable few-shot object detection ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 8778-8787. 10.1109/cvpr46437.2021.00867
21	HAN G， HE Y， HUANG S， et al. Query adaptive few-shot object detection with heterogeneous graph convolutional networks ［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2021： 3243-3252. 10.1109/iccv48922.2021.00325
22	LI Y， ZHU H， CHENG Y， et al. Few-shot object detection via classification refinement and distractor retreatment ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 15390-15398. 10.1109/cvpr46437.2021.01514

k	mAP/%	mAP₅₀/%	mAP₇₅/%
3	11.23	22.32	10.18
5	11.99	23.79	10.79
7	11.62	22.68	10.65
9	10.92	21.67	9.95
11	10.95	21.83	9.82

k	mAP/%	mAP₅₀/%	mAP₇₅/%
3	11.23	22.32	10.18
5	11.99	23.79	10.79
7	11.62	22.68	10.65
9	10.92	21.67	9.95
11	10.95	21.83	9.82

方法	1-shot			2-shot			3-shot			5-shot			10-shot
方法	mAP	mAP₅₀	mAP₇₅	mAP	mAP₅₀	mAP₇₅	mAP	mAP₅₀	mAP₇₅	mAP	mAP₅₀	mAP₇₅	mAP	mAP₅₀	mAP₇₅
FSOD	4.26	9.55	3.56	4.45	9.73	3.74	6.01	13.08	5.04	7.16	16.08	6.29	11.10	20.40	10.60
FSOD+iAFF	4.30	9.85	3.62	4.66	10.06	3.89	6.06	13.17	5.19	7.82	16.68	6.47	11.28	22.39	10.33
FSOD+iAFF+特征增强	4.46	9.95	3.66	4.97	10.79	4.25	6.57	13.70	5.53	8.03	16.86	6.89	11.40	22.71	10.40
FFA-FSOD	4.55	9.97	3.67	5.44	11.89	4.58	6.74	14.54	5.74	8.16	17.03	7.02	11.99	23.79	10.79

方法	1-shot			2-shot			3-shot			5-shot			10-shot
方法	mAP	mAP₅₀	mAP₇₅	mAP	mAP₅₀	mAP₇₅	mAP	mAP₅₀	mAP₇₅	mAP	mAP₅₀	mAP₇₅	mAP	mAP₅₀	mAP₇₅
FSOD	4.26	9.55	3.56	4.45	9.73	3.74	6.01	13.08	5.04	7.16	16.08	6.29	11.10	20.40	10.60
FSOD+iAFF	4.30	9.85	3.62	4.66	10.06	3.89	6.06	13.17	5.19	7.82	16.68	6.47	11.28	22.39	10.33
FSOD+iAFF+特征增强	4.46	9.95	3.66	4.97	10.79	4.25	6.57	13.70	5.53	8.03	16.86	6.89	11.40	22.71	10.40
FFA-FSOD	4.55	9.97	3.67	5.44	11.89	4.58	6.74	14.54	5.74	8.16	17.03	7.02	11.99	23.79	10.79

方法	主干网络	mAP	mAP₅₀	mAP₇₅
Meta-Det^［13］	VGG16	7.10	14.60	6.10
Meta R-CNN^［14］	ResNet-101	8.70	19.10	6.60
MPSR^［15］	ResNet-101	9.80	17.90	9.70
THR^［16］	—	9.80	17.90	9.70
DMNET^［17］	ResNet-101	10.00	17.40	10.40
TFA^［4］	ResNet-101	10.00	—	9.30
Retentive R-CNN^［7］	ResNet-101	10.50	—	—
FSOD-UP^［18］	ResNet-101	11.00	—	10.70
FSCE^［19］	ResNet-101	11.10	—	9.80
SRR-FSD^［20］	ResNet-101	11.30	23.00	9.80
QA-FewDet^［21］	ResNet-101	11.60	23.90	9.80
CGDP+FSCN^［22］	ResNet-50	11.30	20.30	—
FSOD^［10］	ResNet-50	11.10	20.40	10.60
FFA-FSOD	ResNet-50	11.99	23.79	10.79

Few-shot object detection combining feature fusion and enhanced attention

结合特征融合与增强注意力的少样本目标检测

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 22

Related Articles 15

Recommended Articles

Metrics

类别划分	K	不同方法的mAP₅₀/%
类别划分	K	THR	Meta-Det	Meta R-CNN	MPSR	TFA	FSOD-UP	FSCE	CGDP+FSCN	FFA-FSOD
Set 1	1	16.2	18.9	19.9	41.7	39.8	43.8	32.9	40.7	43.9
	2	19.0	20.6	25.5	—	36.1	47.8	44.0	45.1	48.1
	3	29.4	30.2	35.0	51.4	44.7	50.3	46.8	46.5	50.3
	5	37.7	36.8	45.7	55.2	55.7	55.4	52.9	57.4	52.7
	10	49.9	49.6	51.5	61.8	56.0	61.7	59.7	62.4	58.4
Set 2	1	17.2	21.8	10.4	24.4	23.5	31.2	23.7	27.3	31.8
	2	18.9	23.1	19.4	—	26.9	30.5	30.6	31.4	34.5
	3	27.8	27.8	29.6	39.2	34.1	41.2	38.4	40.8	39.7
	5	36.8	31.7	34.8	39.9	35.1	42.2	43.0	42.7	43.3
	10	47.5	43.0	45.4	47.8	39.1	48.3	48.5	46.3	48.7
Set 3	1	20.0	20.6	14.3	35.6	30.8	35.5	22.6	31.2	39.8
	2	23.5	23.9	18.2	—	34.8	39.7	33.4	36.4	42.4
	3	28.9	29.4	27.5	42.3	42.8	43.9	39.5	43.7	45.7
	5	44.2	43.9	41.2	48.0	49.5	50.6	47.3	50.1	48.9
	10	49.2	44.1	48.1	49.7	49.8	53.5	54.0	55.6	51.5

[1]	Jie WANG, Hua MENG. Image classification algorithm based on overall topological structure of point cloud [J]. Journal of Computer Applications, 2024, 44(4): 1107-1113.
[2]	Zhanjun JIANG, Baijing WU, Long MA, Jing LIAN. Faster-RCNN water-floating garbage recognition based on multi-scale feature and polarized self-attention [J]. Journal of Computer Applications, 2024, 44(3): 938-944.
[3]	Ning WU, Yangyang LUO, Huajie XU. Semantic segmentation method for remote sensing images based on multi-scale feature fusion [J]. Journal of Computer Applications, 2024, 44(3): 737-744.
[4]	Wei WANG, Chunhui ZHAO, Xinyao TANG, Liugang XI. 3D vehicle detection with adaptive horizon line constraints [J]. Journal of Computer Applications, 2024, 44(3): 909-915.
[5]	Yuliang ZHENG, Yunhua CHEN, Weijie BAI, Pinghua CHEN. Vehicle target detection by fusing event data and image frames [J]. Journal of Computer Applications, 2024, 44(3): 931-937.
[6]	Zongze JIA, Pengfei GAO, Yinglong MA, Xiaofeng LIU, Haixin XIA. Multi-feature fusion attention-based hierarchical classification method for dialogue act [J]. Journal of Computer Applications, 2024, 44(3): 715-721.
[7]	Yuqiu LI, Liping HOU, Jian XUE, Ke LYU, Yong WANG. Remote sensing image recommendation method based on content interpretation [J]. Journal of Computer Applications, 2024, 44(3): 722-731.
[8]	Qiaoling HUANG, Bochuan ZHENG, Zicheng DING, Zedong WU. Improved image inpainting network incorporating supervised attention module and cross-stage feature fusion [J]. Journal of Computer Applications, 2024, 44(2): 572-579.
[9]	Ziqi HUANG, Jianpeng HU. Entity category enhanced nested named entity recognition in automotive domain [J]. Journal of Computer Applications, 2024, 44(2): 377-384.
[10]	Zhiping ZHU, Yan YANG, Jie WANG. Scene graph-aware cross-modal image captioning model [J]. Journal of Computer Applications, 2024, 44(1): 58-64.
[11]	Jia CHEN, Hong ZHANG. Image text retrieval method based on feature enhancement and semantic correlation matching [J]. Journal of Computer Applications, 2024, 44(1): 16-23.
[12]	Bihui YU, Xingye CAI, Jingxuan WEI. Few-shot text classification method based on prompt learning [J]. Journal of Computer Applications, 2023, 43(9): 2735-2740.
[13]	Hao YANG, Yi ZHANG. Feature pyramid network algorithm based on context information and multi-scale fusion importance awareness [J]. Journal of Computer Applications, 2023, 43(9): 2727-2734.
[14]	Junyu YANG, Yan DONG, Zhennan LONG, Xin YANG, Bin HAN. Rain detection algorithm based on event camera [J]. Journal of Computer Applications, 2023, 43(9): 2904-2909.
[15]	Xiaomin ZHOU, Fei TENG, Yi ZHANG. Automatic international classification of diseases coding model based on meta-network [J]. Journal of Computer Applications, 2023, 43(9): 2721-2726.