基于分组卷积进行特征融合的全景分割算法

doi:10.11772/j.issn.1001-9081.2020091523

计算机应用 ›› 2021, Vol. 41 ›› Issue (7): 2054-2061.DOI: 10.11772/j.issn.1001-9081.2020091523

所属专题：多媒体计算与计算机仿真

• 多媒体计算与计算机仿真 • 上一篇下一篇

基于分组卷积进行特征融合的全景分割算法

冯兴杰^1,2, 张天泽¹

1. 中国民航大学计算机科学与技术学院, 天津 300300;
2. 中国民航大学信息网络中心, 天津 300300

收稿日期:2020-09-30 修回日期:2020-11-25 出版日期:2021-07-10 发布日期:2020-12-14
通讯作者: 张天泽
作者简介:冯兴杰(1969-),男,河北邢台人,教授,博士,主要研究方向:数据库、数据仓库、智能信息处理;张天泽(1994-),男,天津人,硕士研究生,主要研究方向:计算机视觉、图像分割。
基金资助:
中国民用航空局安全能力建设项目（AADSA201909）；天津市教委科研计划项目（2019SK110）；中央高校基本科研业务费专项资金资助项目（3122019009）。

Panoptic segmentation algorithm based on grouped convolution for feature fusion

FENG Xingjie^1,2, ZHANG Tianze¹

1. College of Computer Science and Technology, Civil Aviation University of China, Tianjin 300300, China;
2. Information Network Center, Civil Aviation University of China, Tianjin 300300, China

Received:2020-09-30 Revised:2020-11-25 Online:2021-07-10 Published:2020-12-14
Supported by:
This work is partially supported by the CAAC Security Capacity Building Project (AADSA 201909), the Scientific Research Program of Tianjin Municipal Education Commission (2019sk110), the Fundamental Research Funds for the Central Universities (312219009).

摘要/Abstract

摘要： 针对图像全景分割任务对于实践应用中现有网络结构运算不够快速的问题，提出一种基于分组卷积进行特征融合的全景分割算法。首先，通过自底向上的方式选择经典残差网络结构（ResNet）进行特征提取，并采用不同扩张率的空洞卷积空间金字塔池化操作（ASPP）对提取到的特征进行语义分割与实例分割的多尺度特征融合；然后，通过提出一种单路分组卷积上采样方法，整合语义与实例特征进行上采样特征融合至指定大小；最后，通过对语义分支、实例分支以及实例中心点这三个分支进行损失函数运算以得到更加精细的全景分割输出结果。该模型在CityScapes数据集上与注意力引导的联合全景分割网络（AUNet）、全景特征金字塔网络（Panoptic FPN）、亲和金字塔单阶段实例分割算法（SSAP）、联合全景分割网络（UPSNet）、Panoptic-DeepLab等方法进行了实验对比。实验结果表明，与对比方法中表现最好的Panoptic-DeepLab模型相比，所提模型在极大减少了解码网络参数量的情况下，全景分割质量（PQ）值为0.565，仅下降了0.003，在建筑物、火车、自行车等物体的分割质量上有0.3~5.5的提升，平均精确率（AP）、目标IoU阈值超过50%的平均精确率（AP₅₀）分别提升了0.002与0.014，平均交并比（mIoU）值提升了0.06。可见该方法能提升图像全景分割速度，在PG、AP、mIoU三个指标上均有较好的精度，可以有效地完成全景分割任务。

关键词: 图像全景分割, 语义分割, 实例分割, 分组卷积, 空洞卷积, 空间金字塔池化

Abstract: Aiming at the problem that the computing of the image panoptic segmentation task is not fast enough for the existing network structures in practical applications, a panoptic segmentation algorithm based on grouped convolution for feature fusion was proposed. Firstly, through the bottom-up method, the classic Residual Network structure (ResNet) was selected for feature extraction, and the multi-scale feature fusion of semantic segmentation and instance segmentation was performed on the extracted features by using the Atrous convolutional Spatial Pyramid Pooling operation (ASPP) with different expansion rates. Secondly, a single-channel grouped convolution upsampling method was proposed to integrate the semantics and instance features for performing upsampling feature fusion to a specified size. Finally, a more refined panoptic segmentation output result was obtained by performing loss function on semantic branch, instance branch and instance center point respectively. The model was compared with Attention-guided Unified Network for panoptic segmentation (AUNet), Panoptic Feature Pyramid Network (Panoptic FPN), Single-shot instance Segmentation with Affinity Pyramid (SSAP), Unified Panoptic Segmentation Network (UPSNet), Panoptic-DeepLab and other methods on CityScapes dataset. Compared with the Panoptic-DeepLab model, which is the best-performing model in the comparison models, with the decoding network parameters reduced significantly, the proposed model has the Panoptic Quality (PQ) of 0.565, with a slight decrease of 0.003, and the segmentation qualities of objects such as buildings, trains, bicycles were improved by 0.3-5.5, the Average Precision (AP) and the Average Precision with target IoU (Intersection over Union) threshold over 50% (AP₅₀) were improved by 0.002 and 0.014 respectively, and the mean IoU (mIoU) value was increased by 0.06. It can be seen that the proposed method improves the speed of image panoptic segmentation, has good accuracy in the three indexes of PQ, AP and mIoU, and can effectively complete the panoptic segmentation tasks.

Key words: image panoptic segmentation, semantic segmentation, instance segmentation, grouped convolution, atrous convolution, spatial pyramid pooling

中图分类号:

TP391.4

冯兴杰, 张天泽. 基于分组卷积进行特征融合的全景分割算法[J]. 计算机应用, 2021, 41(7): 2054-2061.

FENG Xingjie, ZHANG Tianze. Panoptic segmentation algorithm based on grouped convolution for feature fusion[J]. Journal of Computer Applications, 2021, 41(7): 2054-2061.

参考文献

[1] KIRILLOV A,HE K,GIRSHICK R,et al. Panoptic segmentation[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2019:9396-9405.
[2] NEUHOLD G,OLLMANN T,BULÒ S R,et al. The Mapillary Vistas Dataset for semantic understanding of street scenes[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:5000-5009.
[3] XIONG Y,LIAO R,ZHAO H,et al. UPSNet:a unified panoptic segmentation network[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2019:8810-8818.
[4] KIRILLOV A,GIRSHICK R,HE K,et al. Panoptic feature pyramid networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2019:6392-6401.
[5] LI Y,CHEN X,ZHU Z,et al. Attention-guided unified network for panoptic segmentation[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2019:7019-7028.
[6] HE K,GKIOXARI G,DOLLÁR P,et al. Mask R-CNN[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:2980-2988.
[7] YANG T J,MAXWELL D C,ZHU Y K,et al. DeeperLab:singleshot image parser[EB/OL].[2020-05-12]. https://arxiv.org/pdf/1902.05093.pdf.
[8] GAO N,SHAN Y,WANG Y,et al. SSAP:single-shot instance segmentation with affinity pyramid[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE,2019:642-651.
[9] CHENG B W,COLLINS M D,ZHU Y,et al. Panoptic-DeepLab:a simple, strong, and fast baseline for bottom-up panoptic segmentation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2020:12472-12482.
[10] KRIZHEVSKY A,SUTSKEVER I,HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Red Hook, NY:Curran Associates Inc.,2012:1097-1105.
[11] BULÒ S R,PORZI L,KONTSCHIEDER P. In-place activated BatchNorm for memory-optimized training of DNNs[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:5639-5647.
[12] HE K,ZHANG X,REN S,et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:770-778.
[13] IOFFE S,SZEGEDY C. Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on Machine Learning. New York:JMLR. org,2015:448-456.
[14] CHENG B, WEI Y, SHI H, et al. Revisiting RCNN:on awakening the classification power of faster RCNN[C]//Proceedings of the 2018 European Conference on Computer Vision. LNCS 11219. Cham:Springer,2018:473-490.
[15] ZHAO H,SHI J,QI X,et al. Pyramid scene parsing network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:6230-6239.
[16] 冯兴杰, 孙少杰. 一种融合多级特征信息的图像语义分割方法[J]. 计算机应用研究,2020,37(11):3512-3515.(FENG X J, SUN S J. Semantic segmentation method integrating multilevel features[J]. Application Research of Computers,2020,37(11):3512-3515.)
[17] HOWARD A G,ZHU M,CHEN B,et al. MobileNets:efficient convolutional neural networks for mobile vision applications[EB/OL].[2020-04-17]. https://arxiv.org/pdf/1704.04861.pdf.
[18] WU Z,SHEN C,HENGEL A V D. Bridging category-level and instance-level semantic image segmentation[EB/OL].[2020-05-23]. https://arxiv.org/pdf/1605.06885.pdf.
[19] KINGMA D P, BA J L. Adam:a method for stochastic optimization[EB/OL].[2019-09-12]. https://arxiv.org/pdf/1412.6980.pdf.
[20] KIM J,LEE J K,LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1646-1654.
[21] 胡雪颖, 郭海儒, 朱蓉. 基于混合深度卷积网络的图像超分辨率重建[J]. 计算机应用,2020,40(7):2069-2076.(HU X Y, GUO H R,ZHU R. Image super-resolution reconstruction based on hybrid deep convolutional network[J]. Journal of Computer Applications,2020,40(7):2069-2076.)
[22] CORDTS M,OMRAN M,RAMOS S,et al. The Cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition,Piscataway:IEEE,2016:3213-3223.
[23] HARIHARAN B,ARBELÁEZ P,BOURDEV L,et al. Semantic contours from inverse detectors[C]//Proceedings of the 2011 International Conference on Computer Vision. Piscataway:IEEE, 2011:991-998.
[24] CHEN L C,PAPANDREOU G,KOKKINOS I,et al. DeepLab:semantic image segmentation with deep convolutional nets,atrous convolution,and fully connected CRFs[J]. IEEE Transaction on Pattern Analysis and Machine Intelligence, 2018, 40(4):834-848.

基于分组卷积进行特征融合的全景分割算法

Panoptic segmentation algorithm based on grouped convolution for feature fusion

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	马佳良, 陈斌, 孙晓飞. 基于改进的Faster R-CNN的通用目标检测框架[J]. 计算机应用, 2021, 41(9): 2712-2719.
[2]	樊玮, 李晨炫, 邢艳, 黄睿, 彭洪健. 航空发动机损伤图像的二分类到多分类递进式检测网络[J]. 计算机应用, 2021, 41(8): 2352-2357.
[3]	胡嵽, 冯子亮. 基于深度学习的轻量级道路图像语义分割算法[J]. 计算机应用, 2021, 41(5): 1326-1331.
[4]	胡屹杉, 秦品乐, 曾建潮, 柴锐, 王丽芳. 基于特征融合和动态多尺度空洞卷积的超声甲状腺分割网络[J]. 计算机应用, 2021, 41(3): 891-897.
[5]	佘玉龙, 张晓龙, 程若勤, 邓春华. 基于边缘关注模型的语义分割方法[J]. 计算机应用, 2021, 41(2): 343-349.
[6]	高世伟, 张长柱, 王祝萍. 基于可分离金字塔的轻量级实时语义分割算法[J]. 计算机应用, 2021, 41(10): 2937-2944.
[7]	黄巨挺, 高宏力, 戴志坤. 基于编码解码结构的移动端电力线语义分割方法[J]. 计算机应用, 2021, 41(10): 2952-2958.
[8]	韩建栋, 李晓宇. 基于多尺度特征融合的行人重识别方法[J]. 计算机应用, 2021, 41(10): 2991-2996.
[9]	田锦, 袁家政, 刘宏哲. 基于实例分割的车道线检测及自适应拟合算法[J]. 计算机应用, 2020, 40(7): 1932-1937.
[10]	胡雪影, 郭海儒, 朱蓉. 基于混合深度卷积网络的图像超分辨率重建[J]. 计算机应用, 2020, 40(7): 2069-2076.
[11]	付磊, 任德均, 吴华运, 郜明, 邱吕, 胡云起. 基于改进Fast-SCNN的塑瓶气泡缺陷实时分割算法[J]. 计算机应用, 2020, 40(6): 1824-1829.
[12]	张盼盼, 李其申, 杨词慧. 基于轻量级分组注意力模块的图像分类算法[J]. 计算机应用, 2020, 40(3): 645-650.
[13]	陈力, 丁世飞, 于文家. 基于跨通道交叉融合和跨模块连接的轻量级卷积神经网络[J]. 计算机应用, 2020, 40(12): 3451-3457.
[14]	冯涛, 陈斌, 张跃飞. 基于改进的Mask R-CNN的染色体图像分割框架[J]. 计算机应用, 2020, 40(11): 3332-3339.
[15]	邓天民, 方芳, 周臻浩. 基于改进空间金字塔池化卷积神经网络的交通标志识别[J]. 计算机应用, 2020, 40(10): 2872-2880.