Lightweight real-time semantic segmentation algorithm based on separable pyramid

doi:10.11772/j.issn.1001-9081.2020121939

Journal of Computer Applications ›› 2021, Vol. 41 ›› Issue (10): 2937-2944.DOI: 10.11772/j.issn.1001-9081.2020121939

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Lightweight real-time semantic segmentation algorithm based on separable pyramid

GAO Shiwei, ZHANG Changzhu, WANG Zhuping

College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China

Received:2020-12-11 Revised:2021-04-12 Online:2021-10-10 Published:2021-07-16
Supported by:
This work is partially supported by the Natural Science Foundation of Shanghai (19ZR1461400).

基于可分离金字塔的轻量级实时语义分割算法

高世伟, 张长柱, 王祝萍

同济大学电子与信息工程学院, 上海201804

通讯作者: 张长柱
作者简介:高世伟(1997-),男,安徽滁州人,硕士研究生,主要研究方向:深度学习、图像语义分割;张长柱(1984-),男,山东曲阜人,副教授,博士,主要研究方向:智能控制、深度学习;王祝萍(1973-),女,上海人,教授,博士,主要研究方向:智能控制、深度学习。
基金资助:
上海市自然科学基金资助项目（19ZR1461400）。

Abstract

Abstract: The existing semantic segmentation algorithms have too many parameters and huge memory usage, so that it is difficult to meet the requirements real-world applications such as automatic driving. In order to solve the problem, a novel, effective and lightweight real-time semantic segmentation algorithm based on Separable Pyramid Module (SPM) was proposed. Firstly, factorized convolution and dilated convolution were adopted in the form of a feature pyramid to construct the bottleneck structure, providing a simple but effective way to extract local and contextual information. Then, the Context Channel Attention (CCA) module based on computer vision attention was proposed to modify the channel weights of shallow feature maps by utilizing deep semantic features, thereby optimizing the segmentation results. Experimental results show that without pre-training or any additional processing, the proposed algorithm achieves mean Intersection-over-Union (mIoU) of 71.86% on Cityscapes test set at the speed of 91 Frames Per Second (FPS). Compared to Efficient Residual Factorized ConvNet (ERFNet), the proposed algorithm has the mIoU 3.86 percentage points higher, and the processing speed of 2.2 times. Compared with the latest Light-weighted Network with Efficient Reduced Non-local operation for real-time semantic segmentation (LRNNet), the proposed algorithm has the mIoU slightly lower by 0.34 percentage points, but the processing speed increased by 20 FPS. The experimental results show that the proposed algorithm has great value for completing tasks such as efficient and accurate street scene image segmentation required in automatic driving.

Key words: real-time semantic segmentation, Deep Convolutional Net (DeepLab), factorized convolution, dilated convolution, channel attention mechanism

摘要： 针对现有语义分割算法参数量过多、内存占用巨大导致其很难满足自动驾驶需要等现实应用的问题，提出一种基于可分离金字塔模块（SPM）的新颖、有效且轻量的实时语义分割算法。首先，利用特征金字塔形式的分解卷积和扩张卷积来构建瓶颈结构，从而以一种简单但有效的方式提取局部和上下文信息；然后，提出基于计算机视觉注意力的上下文通道注意力（CCA）模块，来利用深层语义修改浅层特征图通道权重优化分割效果。实验结果显示：所提出的算法在Cityscapes测试集上以每秒91帧的速度达到了71.86%的平均交并比（mIoU）。相较高效残差分解卷积网络（ERFNet），所提算法mIoU提高了3.86个百分点，处理速度是其2.2倍；与最新的非局部高效实时算法（LRNNet）相比，所提算法mIoU略低0.34个百分点，但处理速度每秒上升了20帧。实验结果表明，所提算法有助于完成如自动驾驶中要求的高效、准确的街道场景图像分割任务。

关键词: 实时语义分割, 深度卷积网络, 分解卷积, 扩张卷积, 通道注意力机制

CLC Number:

TP391.4

GAO Shiwei, ZHANG Changzhu, WANG Zhuping. Lightweight real-time semantic segmentation algorithm based on separable pyramid[J]. Journal of Computer Applications, 2021, 41(10): 2937-2944.

高世伟, 张长柱, 王祝萍. 基于可分离金字塔的轻量级实时语义分割算法[J]. 计算机应用, 2021, 41(10): 2937-2944.

References

[1] GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? the KITTI vision benchmark suite[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2012:3354-3361.
[2] FU J, LIU J, TIAN H J, et al. Dual attention network for scene segmentation[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2019:3141-3149.
[3] JIANG W H, XIE Z Z, LI Y Y, et al. LRNNet:a light-weighted network with efficient reduced non-local operation for real-time semantic segmentation[C]//Proceedings of the 2020 IEEE International Conference on Multimedia and Expo Workshops. Piscataway:IEEE, 2020:1-6.
[4] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2015:3431-3440.
[5] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12):2481-2495.
[6] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[EB/OL]. (2016-06-07)[2020-10-01]. https://arxiv.org/pdf/1412.7062.pdf.
[7] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab:semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4):834-848.
[8] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:770-778.
[9] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL]. (2017-12-05)[2020-10-24]. https://arxiv.org/pdf/1706.05587.pdf.
[10] CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoderdecoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the 2018 European Conference on Computer Vision, LNCS 11211. Cham:Springer, 2018:833-851.
[11] 祖朋达, 李晓敏, 陈更生, 等. DODNet:一种扩张卷积优化的图像语义分割模型[J]. 复旦学报(自然科学版), 2020, 59(5):585-598, 607.(ZU P D, LI X M, CHEN G S, et al. DODNet:an image semantic segmentation model with optimized extended convolution[J]. Journal of Fudan University(Natural Science), 2020, 59(5):585-598, 607.)
[12] YANG M K, YU K, ZHANG C, et al. DenseASPP for semantic segmentation in street scenes[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:3684-3692.
[13] 喻根, 崔炜, 徐照翔, 等. 基于DeepLabV3+的远距离目标语义分割模型[J]. 电光与控制, 2021, 28(1):66-70.(YU G, CUI W, XU Z X, et al. A semantic segmentation model of long-distance targets based on DeepLabV3+[J]. Electronics Optics and Control, 2021, 28(1):66-70.)
[14] ROMERA E, ÁLVAREZ J M, BERGASA L M, et al. ERFNet:efficient residual factorized convnet for real-time semantic segmentation[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 19(1):263-272.
[15] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:7132-7141.
[16] CORDTS M, OMRAN M, RAMOS S, et al. The Cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:3213-3223.
[17] BROSTOW G J, FAUQUEUR J, CIPOLLA R. Semantic object classes in video:a high-definition ground truth database[J]. Pattern Recognition Letters, 2009, 30(2):88-97.
[18] WANG P Q, CHEN P F, YUAN Y, et al. Understanding convolution for semantic segmentation[C]//Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision. Piscataway:IEEE, 2018:1451-1460.
[19] ZHANG Z P, ZHANG K P. FarSee-Net:real-time semantic segmentation by efficient multi-scale context aggregation and feature space super-resolution[C]//Proceedings of the 2020 IEEE International Conference on Robotics and Automation. Piscataway:IEEE, 2020:8411-8417.
[20] YU C Q, WANG J B, PENG C, et al. BiSeNet:bilateral segmentation network for real-time semantic segmentation[C]//Proceedings of the 2018 European Conference on Computer Vision, LNCS 11217. Cham:Springer, 2018:339-349.
[21] LI H C, XIONG P F, FAN H Q, et al. DFANet:deep feature aggregation for real-time semantic segmentation[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2019:9514-9523.
[22] ZHAO H S, QI X J, SHEN X Y, et al. ICNet for real-time semantic segmentation on high-resolution images[C]//Proceedings of the 2018 European Conference on Computer Vision, LNCS 11207. Cham:Springer, 2018:418-434.

Lightweight real-time semantic segmentation algorithm based on separable pyramid

基于可分离金字塔的轻量级实时语义分割算法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 9

Recommended Articles

Metrics

[1]	LIU Shize, QIN Yanjun, WANG Chenxing, GAO Cunyuan, LUO Haiyong, ZHAO Fang, WANG Baohui. Transportation mode recognition algorithm based on multi-scale feature extraction [J]. Journal of Computer Applications, 2021, 41(6): 1573-1580.
[2]	ZHANG Quanlong, WANG Huaibin. Intrusion detection model based on combination of dilated convolution and gated recurrent unit [J]. Journal of Computer Applications, 2021, 41(5): 1372-1377.
[3]	HU Yishan, QIN Pinle, ZENG Jianchao, CHAI Rui, WANG Lifang. Ultrasound thyroid segmentation network based on feature fusion and dynamic multi-scale dilated convolution [J]. Journal of Computer Applications, 2021, 41(3): 891-897.
[4]	HAN Jiandong, LI Xiaoyu. Pedestrian re-identification method based on multi-scale feature fusion [J]. Journal of Computer Applications, 2021, 41(10): 2991-2996.
[5]	HU Xueying, GUO Hairu, ZHU Rong. Image super-resolution reconstruction based on hybrid deep convolutional network [J]. Journal of Computer Applications, 2020, 40(7): 2069-2076.
[6]	FENG Lang, ZHANG Ling, ZHANG Xiaolong. Image inpainting based on dilated convolution [J]. Journal of Computer Applications, 2020, 40(3): 825-831.
[7]	YAO Lu, SONG Huihui, ZHANG Kaihua. Mixed-order channel attention network for single image super-resolution reconstruction [J]. Journal of Computer Applications, 2020, 40(10): 3048-3053.
[8]	XU Zihao, HUANG Weiquan, WANG Yin. Multi-class vehicle detection in surveillance video based on deep learning [J]. Journal of Computer Applications, 2019, 39(3): 700-705.
[9]	ZHANG Hao, DOU Qiwei, LUAN Guikai, YAO Shaowen, ZHOU Wei. Fast image background automatic replacement based on dilated convolution [J]. Journal of Computer Applications, 2018, 38(2): 405-409.