Point cloud semantic segmentation based on attention mechanism and global feature optimization

doi:10.11772/j.issn.1001-9081.2023050588

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (4): 1086-1092.DOI: 10.11772/j.issn.1001-9081.2023050588

• Artificial intelligence • Previous Articles

Point cloud semantic segmentation based on attention mechanism and global feature optimization

Pengfei ZHANG¹, Litao HAN¹^,²(), Hengjian FENG¹, Hongmei LI¹

^1.College of Geodesy and Geomatics，Shandong University of Science and Technology，Qingdao Shandong 266590，China
^2.Key Laboratory of Geomatics and Digital Technology of Shandong Province，Qingdao Shandong 266590，China

Received:2023-05-16 Revised:2023-06-12 Accepted:2023-06-21 Online:2023-08-01 Published:2024-04-10
Contact: Litao HAN
About author:ZHANG Pengfei， born in 1998， M. S. candidate. His research interests include 3D point cloud data processing， point cloud semantic segmentation.
HAN Litao， born in 1978， Ph. D.， associate professor. His research interests include 3D GIS， indoor positioning and navigation， emergency evacuation planning.
FENG Hengjian， born in 1998， M. S. candidate. His research interests include video processing.
LI Hongmei， born in 1998， M. S. candidate. Her research interests include emergency evacuation planning.
Supported by:
National Natural Science Foundation of China(42271436);Shandong Provincial Natural Science Foundation(ZR2021MD030)

基于注意力机制和全局特征优化的点云语义分割

张鹏飞¹, 韩李涛¹^,²(), 冯恒健¹, 李洪梅¹

^1.山东科技大学测绘与空间信息学院，山东青岛 266590
^2.山东省基础地理信息与数字化技术重点实验室，山东青岛 266590

通讯作者: 韩李涛
作者简介:张鹏飞（1998—），男，河南驻马店人，硕士研究生，主要研究方向：三维点云数据处理、点云语义分割
韩李涛（1978—），男，山东菏泽人，副教授，博士，主要研究方向：三维GIS、室内定位与导航、应急疏散规划 hlt1978@163.com
冯恒健（1998—），男，河北邯郸人，硕士研究生，主要研究方向：视频处理
李洪梅（1998—），女，山东潍坊人，硕士研究生，主要研究方向：应急疏散规划。
基金资助:
国家自然科学基金资助项目(42271436);山东省自然科学基金资助项目(ZR2021MD030)

Abstract

Abstract:

In the 3D point cloud semantic segmentation algorithm based on deep learning， to enhance the fine-grained ability to extract local features and learn the long-range dependencies between different local neighborhoods， a neural network based on attention mechanism and global feature optimization was proposed. First， a Single-Channel Attention （SCA） module and a Point Attention （PA） module were designed in the form of additive attention. The former strengthened the resolution of local features by adaptively adjusting the features of each point in a single channel， and the latter adjusted the importance of the single-point feature vector to suppress useless features and reduce feature redundancy. Second， a Global Feature Aggregation （GFA） module was added to aggregate local neighborhood features to capture global context information， thereby improving semantic segmentation accuracy. The experimental results show that the proposed network improves the mean Intersection?over?Union （mIoU） by 1.8 percentage points compared with RandLA-Net （Random sampling and an effective Local feature Aggregator Network） on the point cloud dataset S3DIS， and has good segmentation performance and good adaptability.

Key words: 3D point cloud, deep learning, semantic segmentation, attention mechanism, feature aggregation

摘要：

在基于深度学习的三维点云语义分割算法中，为了加强提取局部特征细粒度能力和学习不同局部邻域之间的长程依赖性，提出一种基于注意力机制和全局特征优化的神经网络。首先，通过加性注意力的形式设计单通道注意力（SCA）模块和点注意力（PA）模块，前者通过自适应调节单通道中各点特征加强对局部特征的分辨能力，后者通过调节单点特征向量之间的重要程度抑制无用特征并减少特征冗余；其次，加入全局特征聚合（GFA）模块，聚合各局部邻域特征，以捕获全局上下文信息，从而提高语义分割精度。实验结果表明，在点云数据集S3DIS上，所提网络的平均交并比（mIoU）相较于RandLA-Net（Random sampling and an effective Local feature Aggregator Network）提升了1.8个百分点，分割性能良好，具有较好的适应性。

关键词: 三维点云, 深度学习, 语义分割, 注意力机制, 特征聚合

CLC Number:

P391

Pengfei ZHANG, Litao HAN, Hengjian FENG, Hongmei LI. Point cloud semantic segmentation based on attention mechanism and global feature optimization[J]. Journal of Computer Applications, 2024, 44(4): 1086-1092.

张鹏飞, 韩李涛, 冯恒健, 李洪梅. 基于注意力机制和全局特征优化的点云语义分割[J]. 《计算机应用》唯一官方网站, 2024, 44(4): 1086-1092.

Figures/Tables 12

References 29

1	LIN Y， VOSSELMAN G， CAO Y， et al. Local and global encoder network for semantic segmentation of airborne laser scanning point clouds［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2021， 176： 151-168. 10.1016/j.isprsjprs.2021.04.016
2	ZHANG J， LIN X， NING X. SVM-based classification of segmented airborne LiDAR point clouds in urban areas［J］. Remote Sensing， 2013， 5（8）： 3749-3775. 10.3390/rs5083749
3	CHEHATA N， GUO L， MALLET C. Airborne LIDAR feature selection for urban classification using random forests［C］// Laser Scanning 2009. Paris： ISPRS.org， 2009： 207-212. 10.1109/icip.2009.5414234
4	ZHUANG Y， LIU Y， HE G， et al. Contextual classification of 3D laser points with conditional random fields in urban environments［C］// Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2015： 3908-3913. 10.1109/iros.2015.7353927
5	LU Y， RASMUSSEN C. Simplified Markov random fields for efficient semantic labeling of 3D point clouds［C］// Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2012： 2690-2697. 10.1109/iros.2012.6386039
6	TONG G， PENG H， SHAO Y， et al. ASCNet： 3D object detection from point cloud based on adaptive spatial context features［J］. Neurocomputing， 2022， 475： 89-101. 10.1016/j.neucom.2021.12.061
7	LIU M， ZHOU Q， ZHAO H， et al. Prototype-voxel contrastive learning for LiDAR point cloud panoptic segmentation［C］// Proceedings of the 2022 International Conference on Robotics and Automation. Piscataway： IEEE， 2022： 9243-9250. 10.1109/icra46639.2022.9811638
8	郝雯，王红霄，汪洋.结合空间注意力与形状特征的三维点云语义分割［J］.激光与光电子学进展，2022，59（8）：0828004. 10.3788/LOP202259.0828004
	HAO W， WANG H X， WANG Y. Semantic segmentation of three-dimensional point cloud based on spatial attention and shape feature［J］. Laser & Optoelectronics Progress， 2022， 59（8）：0828004. 10.3788/LOP202259.0828004
9	SU H， MAJI S， KALOGERAKIS E， et al. Multi-view convolutional neural networks for 3D shape recognition［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 945-953. 10.1109/iccv.2015.114
10	MILIOTO A， VIZZO I， BEHLEY J， et al. RangeNet++： fast and accurate LiDAR semantic segmentation［C］// Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2019： 4213-4220. 10.1109/iros40897.2019.8967762
11	GUO Y， WANG H， HU Q， et al. Deep learning for 3D point clouds： a survey［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2020， 43（12）： 4338-4364. 10.1109/tpami.2020.3005434
12	HUANG J， YOU S. Point cloud labeling using 3D convolutional neural network［C］// Proceedings of the 2016 23rd International Conference on Pattern Recognition. Piscataway： IEEE， 2016： 2670-2675. 10.1109/icpr.2016.7900038
13	RIEGLER G， OSMAN ULUSOY A， GEIGER A. OctNet： learning deep 3D representations at high resolutions［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Computer Society， 2017： 6620-6629. 10.1109/cvpr.2017.701
14	文沛，程英蕾，余旺盛. 基于深度学习的点云分类方法综述［J］. 激光与光电子学进展， 2021， 58（16）：1600003. 10.3788/lop202158.1600003
	WEN P， CHENG Y L， YU W S. Point cloud classification methods based on deep learning： a review［J］. Laser & Optoelectronics Progress， 2021，58（16）：1600003. 10.3788/lop202158.1600003
15	TCHAPMI L， CHOY C， ARMENI I， et al. SEGCloud： semantic segmentation of 3D point clouds［C］// Proceedings of the 2017 International Conference on 3D Vision. Piscataway： IEEE， 2017： 537-547. 10.1109/3dv.2017.00067
16	QI C R， SU H， MO K， et al. PointNet： deep learning on point sets for 3D classification and segmentation［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Computer Society， 2017： 652-660. 10.1109/cvpr.2017.16
17	QI C R， YI L， SU H， et al. PointNet++： deep hierarchical feature learning on point sets in a metric space［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2017： 5105-5114.
18	HU Q， YANG B， XIE L， et al. RandLA-Net： efficient semantic segmentation of large-scale point clouds［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Computer Society， 2020： 11108-11117. 10.1109/cvpr42600.2020.01112
19	QIU S， ANWAR S， BARNES N. Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Computer Society， 2021： 1757-1767. 10.1109/cvpr46437.2021.00180
20	SHAO Y， TONG G， PENG H. Mining local geometric structure for large-scale 3D point clouds semantic segmentation［J］. Neurocomputing， 2022， 500： 191-202. 10.1016/j.neucom.2022.05.060
21	LI Y， LIN Q， ZHANG Z， et al. MFNet： multi-level feature extraction and fusion network for large-scale point cloud classification［J］. Remote Sensing， 2022， 14（22）： 5707. 10.3390/rs14225707
22	WANG Y， SUN Y， LIU Z， et al. Dynamic graph CNN for learning on point clouds［J］. ACM Transactions on Graphics， 2019， 38（5）： No. 146. 10.1145/3326362
23	LANDRIEU L， SIMONOVSKY M. Large-scale point cloud semantic segmentation with superpoint graphs［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Computer Society， 2018： 4558-4567. 10.1109/cvpr.2018.00479
24	JIANG L， ZHAO H， LIU S， et al. Hierarchical point-edge interaction network for point cloud semantic segmentation［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE Computer Society， 2019： 10433-10441. 10.1109/iccv.2019.01053
25	HUANG R， XU Y， STILLA U. GraNet： global relation-aware attentional network for semantic segmentation of ALS point clouds［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2021， 177： 1-20. 10.1016/j.isprsjprs.2021.04.017
26	FAN S， DONG Q， ZHU F， et al. SCF-Net： learning spatial contextual features for large-scale point cloud segmentation［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Computer Society， 2021： 14504-14513. 10.1109/cvpr46437.2021.01427
27	ARMENI I， SENER O， ZAMIR A R， et al. 3D semantic parsing of large-scale indoor spaces［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 1534-1543. 10.1109/cvpr.2016.170
28	HACKEL T， SAVINOV N， LADICKY L， et al. Semantic3D. net： a new large-scale point cloud classification benchmark［J］. ISPRS Annals of the Photogrammetry， Remote Sensing and Spatial Information Sciences， 2017， Ⅵ-1/W1： 91-98. 10.5194/isprs-annals-iv-1-w1-91-2017
29	郝雯，汪洋，魏海南.基于多特征融合的点云场景语义分割［J］. 计算机应用， 2023， 43（10）： 3202-3208.
	HAO W， WANG Y， WEI H N. Semantic segmentation of point cloud scenes based on multi-feature fusion［J］. Journal of Computer Applications， 2023， 43（10）： 3202-3208.

网络模型	OA	mAcc	mIoU
PointNet^［16］	79.3	49.0	41.1
DGCNN^［22］	84.3	45.8	47.6
SPG^［23］	86.4	66.5	58.0
HPEIN^［24］	87.2	68.3	61.9
RandLA-Net^［18］	87.2	71.5	62.4
BAAFNet^［19］	87.0	71.2	62.3
MFNet^［21］	88.5	—	62.9
MFF-Net^［29］	87.1	72.4	63.0
本文网络	87.6	72.3	64.2

网络模型	OA	mAcc	mIoU
PointNet^［16］	79.3	49.0	41.1
DGCNN^［22］	84.3	45.8	47.6
SPG^［23］	86.4	66.5	58.0
HPEIN^［24］	87.2	68.3	61.9
RandLA-Net^［18］	87.2	71.5	62.4
BAAFNet^［19］	87.0	71.2	62.3
MFNet^［21］	88.5	—	62.9
MFF-Net^［29］	87.1	72.4	63.0
本文网络	87.6	72.3	64.2

模型	交并比
模型	天花板	地板	墙壁	梁	柱子	窗户	门	桌子	椅子	沙发	书柜	板子	杂物
PointNet	88.8	97.3	69.8	0.1	3.9	46.3	10.8	59.0	52.6	5.9	40.3	26.4	33.2
DGCNN	92.7	97.5	74.8	0.0	11.7	50.7	23.7	66.3	69.5	8.5	48.7	31.5	42.6
SPG	89.4	96.9	78.1	0.0	42.8	48.9	61.6	84.7	75.4	69.8	52.6	2.1	52.2
HPEIN	91.5	98.2	81.4	0.0	23.3	65.3	40.0	75.5	87.7	58.5	67.8	65.6	49.4
RandLA-Net	91.1	95.6	80.2	0.0	24.7	62.3	47.7	76.2	83.7	60.2	71.1	65.7	53.8
MFNet	91.2	98.2	80.4	0.0	35.1	57.2	46.6	77.4	86.0	65.6	65.4	66.2	47.6
BAAFNet	92.0	97.9	80.9	0.0	30.9	58.2	47.6	78.1	87.2	52.4	67.0	67.3	51.0
MFF-Net	93.0	98.0	81.4	0.0	26.4	59.9	46.2	78.9	87.1	65.8	65.8	64.9	51.0
本文网络	91.8	97.4	80.9	0.0	23.6	61.1	49.5	78.6	86.6	73.1	71.1	68.6	52.8

模型	交并比
模型	天花板	地板	墙壁	梁	柱子	窗户	门	桌子	椅子	沙发	书柜	板子	杂物
PointNet	88.8	97.3	69.8	0.1	3.9	46.3	10.8	59.0	52.6	5.9	40.3	26.4	33.2
DGCNN	92.7	97.5	74.8	0.0	11.7	50.7	23.7	66.3	69.5	8.5	48.7	31.5	42.6
SPG	89.4	96.9	78.1	0.0	42.8	48.9	61.6	84.7	75.4	69.8	52.6	2.1	52.2
HPEIN	91.5	98.2	81.4	0.0	23.3	65.3	40.0	75.5	87.7	58.5	67.8	65.6	49.4
RandLA-Net	91.1	95.6	80.2	0.0	24.7	62.3	47.7	76.2	83.7	60.2	71.1	65.7	53.8
MFNet	91.2	98.2	80.4	0.0	35.1	57.2	46.6	77.4	86.0	65.6	65.4	66.2	47.6
BAAFNet	92.0	97.9	80.9	0.0	30.9	58.2	47.6	78.1	87.2	52.4	67.0	67.3	51.0
MFF-Net	93.0	98.0	81.4	0.0	26.4	59.9	46.2	78.9	87.1	65.8	65.8	64.9	51.0
本文网络	91.8	97.4	80.9	0.0	23.6	61.1	49.5	78.6	86.6	73.1	71.1	68.6	52.8

网络模型	OA	mIoU
RandLA-Net^［18］	90.0	70.6
本文网络	90.0	73.5

Point cloud semantic segmentation based on attention mechanism and global feature optimization

基于注意力机制和全局特征优化的点云语义分割

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 29

Related Articles 15

Recommended Articles

Metrics

网络模型	计算量/MB	总时间/s
RandLA-Net	26.3	4.06
RandLA-Net+PA	28.4	4.20
RandLA-Net+PA+SCA	32.7	4.76
RandLA-Net+PA+SCA+GFA	44.2	5.40

[1]	Bin XIAO, Yun GAN, Min WANG, Xingpeng ZHANG, Zhaoxing WANG. Network abnormal traffic detection based on port attention and convolutional block attention module [J]. Journal of Computer Applications, 2024, 44(4): 1027-1034.
[2]	Xianfeng YANG, Yilei TANG, Ziqiang LI. Aspect-level sentiment analysis model based on alternating‑attention mechanism and graph convolutional network [J]. Journal of Computer Applications, 2024, 44(4): 1058-1064.
[3]	Haihan WANG, Yan ZHU. Offensive speech detection with irony mechanism [J]. Journal of Computer Applications, 2024, 44(4): 1065-1071.
[4]	Quan YUAN, Changping CHEN, Ze CHEN, Linfeng ZHAN. Twice attention mechanism distantly supervised relation extraction based on BERT [J]. Journal of Computer Applications, 2024, 44(4): 1080-1085.
[5]	Yuliang ZHENG, Yunhua CHEN, Weijie BAI, Pinghua CHEN. Vehicle target detection by fusing event data and image frames [J]. Journal of Computer Applications, 2024, 44(3): 931-937.
[6]	Weina DONG, Jia LIU, Xiaozhong PAN, Lifeng CHEN, Wenquan SUN. High-capacity robust image steganography scheme based on encoding-decoding network [J]. Journal of Computer Applications, 2024, 44(3): 772-779.
[7]	Kui ZHAO, Huiqi QIU, Xu LI, Zhifei XU. Real-time pulmonary nodule detection algorithm combining attention and multipath fusion [J]. Journal of Computer Applications, 2024, 44(3): 945-952.
[8]	Aiguo SHANG, Xinjuan ZHU. Joint approach of intent detection and slot filling based on multi-task learning [J]. Journal of Computer Applications, 2024, 44(3): 690-695.
[9]	Ning WU, Yangyang LUO, Huajie XU. Semantic segmentation method for remote sensing images based on multi-scale feature fusion [J]. Journal of Computer Applications, 2024, 44(3): 737-744.
[10]	Zijie HUANG, Yang OU, Degang JIANG, Cailing GUO, Bailin LI. Lightweight deep learning algorithm for weld seam surface quality detection of traction seat [J]. Journal of Computer Applications, 2024, 44(3): 983-988.
[11]	Yongfeng DONG, Jiaming BAI, Liqin WANG, Xu WANG. Chinese named entity recognition combining prior knowledge and glyph features [J]. Journal of Computer Applications, 2024, 44(3): 702-708.
[12]	Dapeng XU, Xinmin HOU. Feature selection method for graph neural network based on network architecture design [J]. Journal of Computer Applications, 2024, 44(3): 663-670.
[13]	Yuqiu LI, Liping HOU, Jian XUE, Ke LYU, Yong WANG. Remote sensing image recommendation method based on content interpretation [J]. Journal of Computer Applications, 2024, 44(3): 722-731.
[14]	Rui JIANG, Wei LIU, Cheng CHEN, Tao LU. Asymmetric unsupervised end-to-end image deraining network [J]. Journal of Computer Applications, 2024, 44(3): 922-930.
[15]	Tao SUN, Zhangtian DUAN, Haonan ZHU, Peihao GUO, Heli SUN. Social event recommendation method based on unexpectedness metric [J]. Journal of Computer Applications, 2024, 44(3): 760-766.

网络	PA	SCA	SDPA	GFA	mIoU
1					62.4
2	√				63.1
3		√			63.0
4	√	√			64.0
5			√		61.5
6	√	√		√	64.2

网络	PA	SCA	SDPA	GFA	mIoU
1					62.4
2	√				63.1
3		√			63.0
4	√	√			64.0
5			√		61.5
6	√	√		√	64.2