Semantic segmentation for 3D point clouds based on feature enhancement

doi:10.11772/j.issn.1001-9081.2022050688

Abstract

Abstract:

In order to mine and sense the geometric features of point clouds and further improve the semantic segmentation effect of point clouds by feature enhancement， a point clouds semantic segmentation network based on feature enhancement was proposed. Firstly， the Geometric Feature Sensing Of Point cloud （GFSOP） module was designed to make the network capable of sensing the local geometric structure of point clouds， semantic representations were enhanced by capturing spatial features between points， and multi-scale features were obtained by the idea of hierarchical extraction of features. At the same time， spatial attention and channel attention were fuseed to predict semantic labels of point clouds， and the segmentation performance was improved by strengthening spatial correlation and channel dependence. Experimental results on the indoor dataset S3DIS （Stanford large-scale 3D Indoor Spaces） show that compared with PointNet++， the proposed network improves the mean Intersection over Union （mIoU） by 5.7 percentage points and the Overall Accuracy （OA） by 3.1 percentage points， and has stronger generalization performance and more robust segmentation effect on point clouds with problems of noise， uneven point cloud density and unclear boundaries.

Key words: point cloud, semantic segmentation, feature enhancement, geometric feature, attention mechanism

摘要：

为挖掘感知点云几何特征并通过特征增强的方式进一步提高点云语义分割效果，提出了一种基于特征增强的点云语义分割网络。首先，通过设计点云的几何特征感知（GFSOP）模块赋予网络点云局部几何结构的感知能力，捕获点间的空间特征以强化语义表征，并利用分层提取特征思想获得多尺度特征。同时，使用空间注意力和通道注意力融合预测点云语义标签，并通过强化空间关联性和通道依赖性提升分割性能。在室内数据集S3DIS（Stanford large-scale 3D Indoor Spaces）上的实验结果显示，所提网络相较于PointNet++在平均交并比（mIoU）上提升了5.7个百分点，在总体准确度（OA）上提升了3.1个百分点，且在存在噪声、点云密度不均和边界不清晰等问题的点云上表现出更强的泛化性能和更加鲁棒的分割效果。

关键词: 点云, 语义分割, 特征增强, 几何特征, 注意力机制

CLC Number:

TP391.41

Bin LU, Jielin LIU. Semantic segmentation for 3D point clouds based on feature enhancement[J]. Journal of Computer Applications, 2023, 43(6): 1818-1825.

鲁斌, 柳杰林. 基于特征增强的三维点云语义分割[J]. 《计算机应用》唯一官方网站, 2023, 43(6): 1818-1825.

Figures/Tables 17

Fig. 1 Architecture of the proposed network

Fig.2 Schematic diagram of geometric relationships between point clouds

Fig. 3 Structure of GFSOP module

Fig. 4 Schematic diagram of S8N algorithm sampling

Fig. 5 Structure of single-layer combined feature extraction

Fig. 6 Structure of fusion prediction module with dual attention

Fig. 7 Performance of S8N algorithm in 3D space

Fig. 8 Schematic diagram of optimized feature extraction module

Tab. 1 Experimental results of semantic segmentation on S3DIS dataset

网络	OA	mAcc	mIoU
PointNet	78.6	57.8	47.7
PointNet++	81.0	67.1	52.8
RSNet	—	65.5	56.1
DGCNN	82.7	—	56.3
本文网络	84.1	71.0	58.5

Fig. 9 Comparison of segmentation accuracy for thirteen categories of objects

Fig. 10 Visualization of some experimental results of semantic segmentation

Fig. 11 Comparative results of semantic segmentation of bathroom scene

Fig. 12 Comparison results of semantic segmentation of lobby scene

Fig. 13 Comparative results of semantic segmentation of office scene

Fig. 14 Three-view drawings of density of point cloud stereo projection

Fig. 15 Sparsity experimental results

Tab. 2 Results of ablation experiments of different module combinations

方法	模块	mIoU	OA	mAcc
PointNet++	基准模型	56.7	84.6	71.1
+GFSOP	添加几何特征感知模块	58.6	86.4	72.6
+SAM	添加空间注意力模块	56.9	84.7	71.9
+CAM	添加通道注意力模块	57.6	85.0	72.1
+SCAM	添加双注意力融合模块	57.9	86.5	73.2
+GFSOP+SCAM	添加几何特征感知模块、双注意力融合模块	60.3	87.0	73.9

References 20

1	王明军，李乐，易芳，等. 模拟真实水体环境下目标激光点云数据的三维重建与分析［J］. 中国激光， 2022， 49（3）： No.0309001. 10.3788/CJL202249.0309001
	WANG M J， LI L， YI F， et al. Three-dimensional reconstruction and analysis of target laser point cloud data in simulated real water environment［J］. Chinese Journal of Lasers， 2022， 49（3）： No.0309001. 10.3788/CJL202249.0309001
2	缪永伟，刘家宗，孙瑜亮，等. 基于类别-实例分割的室内点云场景修复补全［J］. 计算机学报， 2021， 44（11）： 2189-2202. 10.11897/SP.J.1016.2021.02189
	MIAO Y W， LIU J Z， SUN Y L， et al. Point cloud completion of indoor scenes based on category-instance segmentation ［J］. Chinese Journal of Computers， 2021， 44（11）： 2189-2202. 10.11897/SP.J.1016.2021.02189
3	秦庭威，赵鹏程，秦品乐，等. 基于残差注意力机制的点云配准算法［J］. 计算机应用， 2022， 42（7）： 2184-2191.
	QIN T W， ZHAO P C， QIN P L， et al. Point cloud registration algorithm based on residual attention mechanism［J］. Journal of Computer Applications， 2022， 42（7）： 2184-2191.
4	MI X X， YANG B S， CHEN C， et al. Automatic road structure detection and vectorization using MLS point clouds［J］. The International Archives of Photogrammetry， Remote Sensing and Spatial Information Sciences， 2019， XLII-2/W13： 1069-1075. 10.5194/isprs-archives-xlii-2-w13-1069-2019
5	WU Y C， RIVENSON Y， ZHANG Y B， et al. Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery ［J］. Optica， 2018， 5（6）： 704-710. 10.1364/optica.5.000704
6	CHEN H， LIANG M， LIU W Q， et al. An approach to boundary detection for 3D point clouds based on DBSCAN clustering［J］. Pattern Recognition， 2022， 124： No.108431. 10.1016/j.patcog.2021.108431
7	GHAHREMANI M， WILLIAMS K， CORKE F， et al. Direct and accurate feature extraction from 3D point clouds of plants using RANSAC［J］. Computers and Electronics in Agriculture， 2021， 187： No.106240. 10.1016/j.compag.2021.106240
8	BUI M， CHANG L C， LIU H， et al. Comparative study of 3D point cloud compression methods［C］// Proceedings of the 2021 IEEE International Conference on Big Data. Piscataway： IEEE， 2021： 5859-5861. 10.1109/bigdata52589.2021.9671822
9	LI Y Y， BU R， SUN M C， et al. PointCNN： convolution on Χ-transformed points［C］// Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2018： 828-838.
10	ANGULO V， RODRIGUEZ J， GAONA E， et al. A supervoxel-based approach for leaves segmentation of potato plants from point clouds［C］// Proceedings of the 2020 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE， 2020： 4902-4905. 10.1109/igarss39084.2020.9324365
11	ZHOU H Y， LIU A A， NIE W Z， et al. Multi-view saliency guided deep neural network for 3-D object retrieval and classification［J］. IEEE Transactions on Multimedia， 2020， 22（6）： 1496-1506. 10.1109/tmm.2019.2943740
12	QI C R， SU H， MO K， et al. PointNet： deep learning on point sets for 3D classification and segmentation ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 77-85. 10.1109/cvpr.2017.16
13	QI C R， YI L， SU H， et al. PointNet++： deep hierarchical feature learning on point sets in a metric space［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 5105-5114.
14	WU W， QI Z， LI F X. PointConv： deep convolutional networks on 3D point clouds［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 9621-9630. 10.1109/cvpr.2019.00985
15	LU Q， CHEN C， XIE W J， et al. PointNGCNN： deep convolutional networks on 3D point clouds with neighborhood graph filters ［J］. Computers and Graphics， 2020， 86： 42-51. 10.1016/j.cag.2019.11.005
16	LI J X， CHEN B M， LEE G H. SO-Net： self-organizing network for point cloud analysis［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 9397-9406. 10.1109/cvpr.2018.00979
17	LIU T， YANG Z H， HU S J， et al. Neighbor reweighted local centroid for geometric feature identification［J］. IEEE Transactions on Visualization and Computer Graphics， 2023， 29（2）： 1545-1558. 10.1109/tvcg.2021.3124911
18	JIANG M Y， WU Y R， ZHAO T Q， et al. PointSIFT： a SIFT-like network module for 3D point cloud semantic segmentation［EB/OL］. （2018-11-24）［2022-05-02］.. 10.1145/3503161.3548251
19	FU J， LIU J， TIAN H J， et al. Dual attention network for scene segmentation ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3141-3149. 10.1109/cvpr.2019.00326
20	ARMENI I， SENER O， ZAMIR A R， et al. 3D semantic parsing of large-scale indoor spaces［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 1534-1543. 10.1109/cvpr.2016.170

[1]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[2]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[3]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[4]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.
[5]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[6]	Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594.
[7]	Shangbin MO, Wenjun WANG, Ling DONG, Shengxiang GAO, Zhengtao YU. Single-channel speech enhancement based on multi-channel information aggregation and collaborative decoding [J]. Journal of Computer Applications, 2024, 44(8): 2611-2617.
[8]	Wu XIONG, Congjun CAO, Xuefang SONG, Yunlong SHAO, Xusheng WANG. Handwriting identification method based on multi-scale mixed domain attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2225-2232.
[9]	Huanhuan LI, Tianqiang HUANG, Xuemei DING, Haifeng LUO, Liqing HUANG. Public traffic demand prediction based on multi-scale spatial-temporal graph convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2065-2072.
[10]	Dianhui MAO, Xuebo LI, Junling LIU, Denghui ZHANG, Wenjing YAN. Chinese entity and relation extraction model based on parallel heterogeneous graph and sequential attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2018-2025.
[11]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[12]	Song XU, Wenbo ZHANG, Yifan WANG. Lightweight video salient object detection network based on spatiotemporal information [J]. Journal of Computer Applications, 2024, 44(7): 2192-2199.
[13]	Dahai LI, Zhonghua WANG, Zhendong WANG. Dual-branch low-light image enhancement network combining spatial and frequency domain information [J]. Journal of Computer Applications, 2024, 44(7): 2175-2182.
[14]	Wenliang WEI, Yangping WANG, Biao YUE, Anzheng WANG, Zhe ZHANG. Deep learning model for infrared and visible image fusion based on illumination weight allocation and attention [J]. Journal of Computer Applications, 2024, 44(7): 2183-2191.
[15]	Zexin XU, Lei YANG, Kangshun LI. Shorter long-sequence time series forecasting model [J]. Journal of Computer Applications, 2024, 44(6): 1824-1831.