Weakly supervised salient object detection algorithm based on bounding box annotation

doi:10.11772/j.issn.1001-9081.2022050706

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (6): 1910-1918.DOI: 10.11772/j.issn.1001-9081.2022050706

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Weakly supervised salient object detection algorithm based on bounding box annotation

Qiang WANG¹^,², Xiaoming HUANG¹^,², Qiang TONG¹^,², Xiulei LIU¹^,²()

^1.Institute of Data Science and Information Analysis，Beijing Information Science and Technology University，Beijing 100101，China
^2.Beijing Advanced Innovation Center for Materials Genome Engineering （Beijing Information Science and Technology University），Beijing 100101，China

Received:2022-05-18 Revised:2023-01-04 Accepted:2023-01-10 Online:2023-06-08 Published:2023-06-10
Contact: Xiulei LIU
About author:WANG Qiang， born in 1996， M. S. candidate. His research interests include machine learning， image recognition.
TONG Qiang， born in 1985， Ph. D.， lecturer. His research interests include image recognition， computer vision， machine learning.
First author contact:HUNAG Xiaoming， born in 1977， Ph. D.， associate professor. His research interests include machine learning， object detection， semantic segmentation.
Supported by:
National Key Research and Development Program of China(2021YFB2600600);Fund of Beijing Information Science and Technology University(2121YJPY225);Innovation Capacity Building of Scientific Research Institutions,Beijing Municipal Education Commission Science and Technology Program(KM202011232014)

基于边界框标注的弱监督显著性目标检测算法

王强¹^,², 黄小明¹^,², 佟强¹^,², 刘秀磊¹^,²()

^1.北京信息科技大学数据科学与情报分析研究所, 北京 100101
^2.北京材料基因工程高精尖创新中心(北京信息科技大学), 北京 100101

通讯作者: 刘秀磊
作者简介:王强（1996—），男，安徽潜山人，硕士研究生，主要研究方向：机器学习、图像识别
黄小明（1977—），男，安徽潜山人，副教授，博士，CCF会员，主要研究方向：机器学习、目标检测、语义分割
佟强（1985—），男（锡伯族），辽宁沈阳人，讲师，博士，CCF会员，主要研究方向：图像识别、计算机视觉、机器学习
刘秀磊（1981—），男，河南濮阳人，教授，博士，CCF会员，主要研究方向：语义 Web、本体匹配、语义搜索、知识图谱、语义传感器Email：liuxiulei@bistu.edu.cn。
基金资助:
国家重点研发计划项目(2021YFB2600600);北京信息科技大学校级基金资助项目(2121YJPY225);科研机构创新能力建设;北京市教委科研计划项目(KM202011232014)

Abstract

Abstract:

Aiming at the inaccurate positioning problem of salient object in the previous weakly supervised salient object detection algorithms， a weakly supervised salient object detection algorithm based on bounding box annotation was proposed. In the proposed algorithm， the minimum bounding rectangle boxes， which are the bounding boxes of all objects in the image were adopted as supervision information. Firstly， the initial saliency map was generated based on the bounding box annotation and GrabCut algorithm. Then， a correction module for missing object was designed to obtain the optimized saliency map. Finally， by combining the advantages of the traditional methods and deep learning methods， the optimized saliency map was used as the pseudo ground-truth to learn a salient object detection model through neural network. Comparison of the proposed algorithm and six unsupervised and four weakly supervised saliency detection algorithms was carried on four public datasets. Experimental results show that the proposed algorithm significantly outperforms comparison algorithms in both Max F-measure value （Max-F） and Mean Absolute Error （MAE） on four datasets. Compared with SBB （Sales Bounding Boxes）， which is also a weakly supervised method based on boundary box annotation， the annotation method of the proposed algorithm is simpler. Experiments were conducted on four datasets， ECSSD， DUTS-TE， HKU-IS， DUT-OMRON， and the Max-F increased by 1.82%， 4.00%， 1.27% and 5.33% respectively， and the MAE decreased by 13.89%， 15.07%， 8.77% and 13.33%， respectively. It can be seen that the proposed algorithm is a weakly supervised salient object detection algorithm with good detection performance.

Key words: weakly supervised, bounding box annotation, saliency map, pseudo ground-truth, salient object detection

摘要：

针对以往的弱监督显著性目标检测算法存在的显著目标定位不准确问题，提出一种基于边界框标注的弱监督显著目标检测算法。所提算法利用图像中所有目标的最小外接矩形框，即边界框，作为监督信息。首先基于边界框标注和GrabCut算法生成初始显著图；然后在此基础上设计了一个缺失修正模块，以得到优化后的显著图；最后结合传统方法和深度学习方法各自的优势，将优化后的显著图作为伪真值，通过神经网络学习一个显著性目标检测模型。在4个公开数据集上与6种无监督、4种弱监督的显著性检测算法进行比较的实验结果显示，所提算法在所有数据集上的最大F度量值（Max-F）和平均绝对误差（MAE）均明显优于对比算法：与同样基于边界框标注的弱监督方法SBB（Saliency Bounding Boxes）相比，所提算法的标注方法更简单，在ECSSD、DUTS-TE、HKU-IS、DUT-OMRON等4个数据集上进行实验，Max-F分别提高了1.82%、4.00%、1.27%和5.33%，MAE分别降低了13.89%、15.07%、8.77%和13.33%。可见，所提算法是一种具有良好检测性能的弱监督显著目标检测算法。

关键词: 弱监督, 边界框标注, 显著图, 伪真值, 显著性目标检测

CLC Number:

TP391.41

Qiang WANG, Xiaoming HUANG, Qiang TONG, Xiulei LIU. Weakly supervised salient object detection algorithm based on bounding box annotation[J]. Journal of Computer Applications, 2023, 43(6): 1910-1918.

王强, 黄小明, 佟强, 刘秀磊. 基于边界框标注的弱监督显著性目标检测算法[J]. 《计算机应用》唯一官方网站, 2023, 43(6): 1910-1918.

Figures/Tables 15

References 30

1	张文达，许悦雷，倪嘉成，等. 基于多尺度分块卷积神经网络的图像目标识别算法［J］. 计算机应用， 2016， 36（4）： 1033-1038. 10.11772/j.issn.1001-9081.2016.04.1033
	ZHANG W D， XU Y L， NI J C， et al. Image target recognition method based on multi-scale block convolutional neural network［J］. Journal of Computer Applications， 2016， 36（4）： 1033-1038. 10.11772/j.issn.1001-9081.2016.04.1033
2	YANG X Y， QIAN X M， XUE Y. Scalable mobile image retrieval by exploring contextual saliency ［J］. IEEE Transactions on Image Processing， 2015， 24（6）： 1709-1721. 10.1109/tip.2015.2411433
3	SU Y Y， ZHAO Q J， ZHAO L J， et al. Abrupt motion tracking using a visual saliency embedded particle filter［J］. Pattern Recognition， 2014， 47（5）： 1826-1834. 10.1016/j.patcog.2013.11.028
4	ACHANTA R， HEMAMI S， ESTRADA F， et al. Frequency-tuned salient region detection［C］// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2009： 1597-1604. 10.1109/cvpr.2009.5206596
5	GOFERMAN S， ZELNIK-MANOR L， TAL A. Context-aware saliency detection ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2012， 34（10）： 1915-1926. 10.1109/tpami.2011.272
6	ZHANG J M， SCLAROFF S， LIN Z， et al. Minimum barrier salient object detection at 80 FPS［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 1404-1412. 10.1109/iccv.2015.165
7	ACHANTA R， ESTRADA F， WILS P， et al. Salient region detection and segmentation［C］// Proceedings of the 2008 International Conference on Computer Vision Systems， LNCS 5008. Berlin： Springer， 2008： 66-75.
8	VALENTI R， SEBE N， GEVERS T. Image saliency by isocentric curvedness and color［C］// Proceedings of the IEEE 12th International Conference on Computer Vision. Piscataway： IEEE， 2009： 2185-2192. 10.1109/iccv.2009.5459240
9	ZHANG J， ZHANG T， DAI Y C， et al. Deep unsupervised saliency detection： a multiple noisy labeling perspective［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 9029-9038. 10.1109/cvpr.2018.00941
10	CHENG M M， MITRA N J， HUANG X L， et al. Global contrast based salient region detection ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2015， 37（3）： 569-582. 10.1109/tpami.2014.2345401
11	李小雨，房体育，夏英杰，等. 基于图割精细化和可微分聚类的无监督显著性目标检测［J］. 计算机应用， 2021， 41（12）： 3571-3577. 10.11772/j.issn.1001-9081.2021061054
	LI X Y， FANG T Y， XIA Y J， et al. Unsupervised salient object detection based on graph cut refinement and differentiable clustering ［J］. Journal of Computer Applications， 2021， 41（12）： 3571-3577. 10.11772/j.issn.1001-9081.2021061054
12	YANG C， ZHANG L H， LU H C， et al. Saliency detection via graph-based manifold ranking ［C］// Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2013： 3166-3173. 10.1109/cvpr.2013.407
13	LI X H， LU H C， ZHANG L H， et al. Saliency detection via dense and sparse reconstruction［C］// Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2013： 2976-2983. 10.1109/iccv.2013.370
14	YAN Q， XU L， SHI J P， et al. Hierarchical saliency detection ［C］// Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2013： 1155-1162. 10.1109/cvpr.2013.153
15	QIN Y， LU H C， XU Y Q， et al. Saliency detection via cellular automata［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 110-119. 10.1109/cvpr.2015.7298606
16	TU W C， HE S F， YANG Q X， et al. Real-time salient object detection with a minimum spanning tree ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 2334-2342. 10.1109/cvpr.2016.256
17	赵恒，安维胜，付为刚. 深度导向显著性检测算法［J］. 计算机应用， 2019， 39（1）： 143-147. 10.11772/j.issn.1001-9081.2018061194
	ZHAO H， AN W S， FU W G. Saliency detection algorithm of deep guidance［J］. Journal of Computer Applications， 2019， 39（1）： 143-147. 10.11772/j.issn.1001-9081.2018061194
18	后云龙，朱磊，陈琴，等. 基于高斯差分特征网络的显著目标检测［J］. 计算机应用， 2021， 41（3）： 706-713. 10.11772/j.issn.1001-9081.2020060957
	HOU Y L， ZHU L， CHEN Q， et al. Salient object detection based on difference of Gaussian feature network ［J］. Journal of Computer Applications， 2021， 41（3）： 706-713. 10.11772/j.issn.1001-9081.2020060957
19	温静，宋建伟. 基于多级全局信息传递模型的视觉显著性检测［J］. 计算机应用， 2021， 41（1）： 208-214. 10.11772/j.issn.1001-9081.2020060968
	WEN J， SONG J W. Visual saliency detection based on multi-level global information propagation model ［J］. Journal of Computer Applications， 2021， 41（1）： 208-214. 10.11772/j.issn.1001-9081.2020060968
20	LI G B， XIE Y， LIN L. Weakly supervised salient object detection using image labels ［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2018： 7024-7031. 10.1609/aaai.v32i1.12308
21	谭台哲，轩康西，曾群生. 基于图像级标签及超像素块的弱监督显著性检测［J］. 计算机应用研究， 2020， 37（2）： 601-605. 10.19734/j.issn.1001-3695.2018.06.0576
	TAN T Z， XUAN K X， ZENG Q S. Supervised significant detection based on image level labels and superpixel blocks ［J］. Application Research of Computers， 2020， 37（2）： 601-605. 10.19734/j.issn.1001-3695.2018.06.0576
22	WANG L J， LU H C， WANG Y F， et al. Learning to detect salient objects with image-level supervision ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 3796-3805. 10.1109/cvpr.2017.404
23	ZENG Y， ZHUGE Y Z， LU H C， et al. Multi-source weak supervision for saliency detection［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 6067-6076. 10.1109/cvpr.2019.00623
24	LIU Y X， WANG P J， CAO Y， et al. Weakly-supervised salient object detection with saliency bounding boxes［J］. IEEE Transactions on Image Processing， 2021， 30： 4423-4435. 10.1109/tip.2021.3071691
25	DAI J F， HE K M， SUN J. BoxSup： exploiting bounding boxes to supervise convolutional networks for semantic segmentation ［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 1635-1643. 10.1109/iccv.2015.191
26	KHOREVA A， BENENSON R， HOSANG J， et al. Simple does it： weakly supervised instance and semantic segmentation［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 876-885. 10.1109/cvpr.2017.181
27	ROTHER C， KOLMOGOROV V， BLAKE A. “GrabCut”： interactive foreground extraction using iterated graph cuts［J］. ACM Transactions on Graphics， 2004， 23（3）： 309-314. 10.1145/1015706.1015720
28	LIU J J， HOU Q B， CHENG M M， et al. A simple pooling-based design for real-time salient object detection ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3912-3921. 10.1109/cvpr.2019.00404
29	BOYKOV Y， JOLLY M P. Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images ［C］// Proceedings 8th IEEE International Conference on Computer Vision. Piscataway： IEEE， 2001，1： 105-112.
30	LI G B， YU Y Z. Visual saliency based on multiscale deep features［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 5455-5463. 10.1109/cvpr.2015.7299184

算法	监督方式	ECSSD		DUTS-TE		HKU-IS		DUT-OMRON
算法	监督方式	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓
MR^［12］	无监督	0.690	0.186	0.510	0.189	0.655	0.174	0.577	0.194
DSR^［13］	无监督	0.676	0.179	0.506	0.163	0.677	0.149	0.536	0.145
HS^［14］	无监督	0.627	0.229	0.460	0.258	0.623	0.223	0.507	0.237
BSCA^［15］	无监督	0.707	0.185	0.500	0.197	0.654	0.175	0.509	0.190
MB+^［6］	无监督	0.697	0.174	0.528	0.179	0.678	0.151	0.531	0.167
MST^［16］	无监督	0.693	0.151	0.540	0.156	0.680	0.131	0.542	0.149
ASMO^［20］	弱监督（类别标注）	0.810	0.114	0.625	0.123	0.821	0.091	0.633	0.100
WSS^［22］	弱监督（类别标注）	0.828	0.105	0.657	0.106	0.821	0.081	0.611	0.111
MSW^［23］	弱监督（类别+标题标注）	0.846	0.096	0.704	0.097	0.823	0.086	0.619	0.109
SBB^［24］	弱监督（边界框标注）	0.878	0.072	0.775	0.073	0.869	0.057	0.751	0.075
本文算法	弱监督（边界框标注）	0.894	0.062	0.806	0.062	0.880	0.052	0.791	0.065

算法	监督方式	ECSSD		DUTS-TE		HKU-IS		DUT-OMRON
算法	监督方式	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓
MR^［12］	无监督	0.690	0.186	0.510	0.189	0.655	0.174	0.577	0.194
DSR^［13］	无监督	0.676	0.179	0.506	0.163	0.677	0.149	0.536	0.145
HS^［14］	无监督	0.627	0.229	0.460	0.258	0.623	0.223	0.507	0.237
BSCA^［15］	无监督	0.707	0.185	0.500	0.197	0.654	0.175	0.509	0.190
MB+^［6］	无监督	0.697	0.174	0.528	0.179	0.678	0.151	0.531	0.167
MST^［16］	无监督	0.693	0.151	0.540	0.156	0.680	0.131	0.542	0.149
ASMO^［20］	弱监督（类别标注）	0.810	0.114	0.625	0.123	0.821	0.091	0.633	0.100
WSS^［22］	弱监督（类别标注）	0.828	0.105	0.657	0.106	0.821	0.081	0.611	0.111
MSW^［23］	弱监督（类别+标题标注）	0.846	0.096	0.704	0.097	0.823	0.086	0.619	0.109
SBB^［24］	弱监督（边界框标注）	0.878	0.072	0.775	0.073	0.869	0.057	0.751	0.075
本文算法	弱监督（边界框标注）	0.894	0.062	0.806	0.062	0.880	0.052	0.791	0.065

方法	ECSSD		DUTS-TE		HKU-IS		DUT-OMRON
方法	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓
GrabCut生成的初始显著图	0.892 7	0.056 4	0.814 0	0.058 7	0.855 7	0.061 4	0.870 5	0.040 5
GrabCut +四周调整的显著图	0.893 5	0.054 6	0.817 0	0.057 4	0.857 0	0.059 9	0.871 7	0.039 7
GrabCut +四周中部调整的显著图	0.894 1	0.054 4	0.817 2	0.057 4	0.856 9	0.060 0	0.872 0	0.039 6
GrabCut +全部后处理的显著图	0.894 3	0.054 2	0.817 5	0.057 3	0.857 1	0.059 9	0.872 2	0.039 5

方法	ECSSD		DUTS-TE		HKU-IS		DUT-OMRON
方法	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓
GrabCut生成的初始显著图	0.892 7	0.056 4	0.814 0	0.058 7	0.855 7	0.061 4	0.870 5	0.040 5
GrabCut +四周调整的显著图	0.893 5	0.054 6	0.817 0	0.057 4	0.857 0	0.059 9	0.871 7	0.039 7
GrabCut +四周中部调整的显著图	0.894 1	0.054 4	0.817 2	0.057 4	0.856 9	0.060 0	0.872 0	0.039 6
GrabCut +全部后处理的显著图	0.894 3	0.054 2	0.817 5	0.057 3	0.857 1	0.059 9	0.872 2	0.039 5

方法	ECSSD		DUTS-TE		HKU-IS		DUT-OMRON
方法	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓	Max-F↑	MAE↓
GrabCut生成的初始显著图	0.886 5	0.068 9	0.803 9	0.064 3	0.874 9	0.054 7	0.781 7	0.067 2
GrabCut +四周调整的显著图	0.890 6	0.064 5	0.804 0	0.063 2	0.876 0	0.053 7	0.793 2	0.064 3
GrabCut +四周、中部调整的显著图	0.893 8	0.063 2	0.807 4	0.062 9	0.875 4	0.053 6	0.794 2	0.064 6
GrabCut +全部后处理的显著图	0.893 6	0.062 1	0.805 7	0.061 9	0.879 6	0.052 0	0.791 2	0.065 1

Weakly supervised salient object detection algorithm based on bounding box annotation

基于边界框标注的弱监督显著性目标检测算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 30

Related Articles 15

Recommended Articles

Metrics

[1]	Song XU, Wenbo ZHANG, Yifan WANG. Lightweight video salient object detection network based on spatiotemporal information [J]. Journal of Computer Applications, 2024, 44(7): 2192-2199.
[2]	Zimeng ZHU, Zhixin LI, Zhan HUAN, Ying CHEN, Jiuzhen LIANG. Weakly supervised video anomaly detection based on triplet-centered guidance [J]. Journal of Computer Applications, 2024, 44(5): 1452-1457.
[3]	Cong HU, Gang HUA. Weakly supervised action localization method based on attention mechanism [J]. Journal of Computer Applications, 2022, 42(3): 960-967.
[4]	Quan CHEN, Li LI, Yongle CHEN, Yuexing DUAN. Adversarial attack algorithm for deep learning interpretability [J]. Journal of Computer Applications, 2022, 42(2): 510-518.
[5]	Shujie QI, Chunyi CHEN, Xiaojuan HU, Haiyang YU. Visual‑saliency‑driven reuse algorithm of indirect lighting in 3D scene rendering [J]. Journal of Computer Applications, 2022, 42(11): 3551-3557.
[6]	Ping LUO, Ling DING, Xue YANG, Yang XIANG. Chinese event detection based on data augmentation and weakly supervised adversarial training [J]. Journal of Computer Applications, 2022, 42(10): 2990-2995.
[7]	Shuang DENG, Xiaohai HE, Linbo QING, Honggang CHEN, Qizhi TENG. Weakly supervised fine-grained classification method of Alzheimer’s disease based on improved visual geometry group network [J]. Journal of Computer Applications, 2022, 42(1): 302-309.
[8]	LU Xinwei, YU Pengfei, LI Haiyan, LI Hongsong, DING Wenqian. Weakly supervised fine-grained image classification algorithm based on attention-attention bilinear pooling [J]. Journal of Computer Applications, 2021, 41(5): 1319-1325.
[9]	HOU Yunlong, ZHU Lei, CHEN Qin, LYU Suidong. Salient object detection based on difference of Gaussian feature network [J]. Journal of Computer Applications, 2021, 41(3): 706-713.
[10]	Xiaoyu LI, Tiyu FANG, Yingjie XIA, Jinping LI. Unsupervised salient object detection based on graph cut refinement and differentiable clustering [J]. Journal of Computer Applications, 2021, 41(12): 3571-3577.
[11]	WANG Hongyu, ZHANG Yu, YANG Heng, MU Nan. Salient object detection in weak light images based on ant colony optimization algorithm [J]. Journal of Computer Applications, 2021, 41(10): 2970-2978.
[12]	BIAN Xiaoyong, JIANG Peiling, ZHAO Min, DING Sheng, ZHANG Xiaolong. Multi-branch neural network model based weakly supervised fine-grained image classification method [J]. Journal of Computer Applications, 2020, 40(5): 1295-1300.
[13]	SHI Xiangbin, ZHOU Jincheng, LIU Cuiwei. Weakly supervised action localization based on action template matching [J]. Journal of Computer Applications, 2019, 39(8): 2408-2413.
[14]	HU Xuemin, YI Chonghui, CHEN Qin, CHEN Xi, CHEN Long. Abnormal crowd behavior detection based on motion saliency map [J]. Journal of Computer Applications, 2018, 38(4): 1164-1169.
[15]	WANG Xin, ZHOU Yun, NING Chen, SHI Aiye. Image saliency detection via adaptive fusion of local and global sparse representation [J]. Journal of Computer Applications, 2018, 38(3): 866-872.