Outdoor scene point cloud segmentation model based on graph model and attention mechanism

doi:10.11772/j.issn.1001-9081.2022111704

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (12): 3911-3917.DOI: 10.11772/j.issn.1001-9081.2022111704

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Outdoor scene point cloud segmentation model based on graph model and attention mechanism

Feiyu LIAN¹^,², Liang ZHANG¹^,²(), Jiedong WANG³, Yukang JIN¹^,², Yu CHAI¹^,²

^1.Faculty of Resources and Environmental Science，Hubei University，Wuhan Hubei 430062，China
^2.Hubei Key Laboratory of Regional Development and Environmental Response （Hubei University），Wuhan Hubei 430062，China
^3.The Second Institute of Surveying and Mapping of Zhejiang Province，Hangzhou Zhejiang 310012，China

Received:2022-11-15 Revised:2023-03-13 Accepted:2023-03-20 Online:2023-07-12 Published:2023-12-10
Contact: Liang ZHANG
About author:LIAN Feiyu， born in 1997， M. S. candidate. His research interests include three-dimensional point cloud processing， point cloud segmentation.
WANG Jiedong， born in 1986， M. S.，engineer. His research interests include photogrammetry， remote sensing applications.
JIN Yukang， born in 1997， M. S. candidate. His research interests include unmanned aerial vehicle point cloud filtering.
CHAI Yu， born in 1999， M. S. candidate. Her research interests include three-dimensional point cloud processing.
Supported by:
National Natural Science Foundation of China(41601504);Major Project of High?Resolution Earth Observation System(11?H37B02?900?19/22)

基于图模型与注意力机制的室外场景点云分割模型

廉飞宇¹^,², 张良¹^,²(), 王杰栋³, 靳于康¹^,², 柴玉¹^,²

^1.湖北大学资源环境学院, 武汉 430062
^2.区域开发与环境响应湖北省重点实验室(湖北大学), 武汉 430062
^3.浙江省第二测绘院, 杭州 310012

通讯作者: 张良
作者简介:廉飞宇（1997—），男，山东临沂人，硕士研究生，主要研究方向：三维点云处理、点云分割
王杰栋（1986—），男，宁夏银川人，工程师，硕士，主要研究方向：摄影测量、遥感应用
靳于康（1997—），男，河南南阳人，硕士研究生，主要研究方向：无人机点云滤波
柴玉（1999—），女，河南信阳人，硕士研究生，主要研究方向：三维点云处理。
基金资助:
国家自然科学基金资助项目(41601504);高分辨率对地观测系统重大专项(11?H37B02?9001?19/22)

Abstract

Abstract:

Aiming at the problem that it is difficult to distinguish similar land types in outdoor scenes with multiple objects and complex spatial topological relationships， an A-Edge-SPG （Attention-EdgeConv SuperPoint Graph） graph neural network combining graph model and attention mechanism module was proposed. Firstly， the superpoints were segmented by the combination of graph cut and geometric features. Secondly， the local adjacency graph was constructed inside the superpoint to capture the context information of the point cloud in the scene and use the attention mechanism module to highlight the key information. Finally， a SuperPoint Graph （SPG） model was constructed， and the features of hyperpoints and hyperedges were aggregated by Gated Recurrent Unit （GRU） to realize accurate segmentation among different land types of point cloud. On Semantic3D dataset， the semantic segmentation effect of A-Edge-SPG model and SPG-Net （SPG neural Network） model was compared and analyzed. Experimental results show that compared with the SPG model， A-Edge-SPG model improves the Overall segmentation Accuracy（OA）， mean Intersection over Union （mIoU） and mean Average Accuracy （mAA） by 1.8， 5.1 and 2.8 percentage points respectively， and significantly improves the segmentation accuracy of similar land types such as high vegetation and dwarf vegetation， improving the effect of distinguishing similar land types.

Key words: semantic segmentation, outdoor scene, local feature, attention mechanism module, local adjacency graph, graph model

摘要：

针对在多对象且空间拓扑关系复杂的室外场景环境中相似地类区分难的问题，提出一种结合图模型与注意力机制模块的A-Edge-SPG（Attention-EdgeConv SuperPoint Graph）图神经网络。首先，利用图割和几何特征结合的方法对超点进行分割；其次，在超点内部构造局部邻接图，从而在捕获场景中点云的上下文信息的同时利用注意力机制模块凸显关键信息；最后，构建超点图（SPG）模型，并采用门控循环单元（GRU）聚合超点和超边特征，实现对不同地类点云间的精确分割。在Semantic3D数据集上对A-Edge-SPG模型和SPG-Net（SPG neural Network）模型的语义分割效果进行比较分析。实验结果表明，相较于SPG模型，A-Edge-SPG模型在总体分割精度（OA）、平均交并比（mIoU）和平均精度均值（mAA）上分别提升了1.8、5.1和2.8个百分点，并且在高植被、矮植被等相似地类的分割精度上取得了明显的提升，改善了相似地类间语义分割的效果。

关键词: 语义分割, 室外场景, 局部特征, 注意力机制模块, 局部邻接图, 图模型

CLC Number:

TP183

Feiyu LIAN, Liang ZHANG, Jiedong WANG, Yukang JIN, Yu CHAI. Outdoor scene point cloud segmentation model based on graph model and attention mechanism[J]. Journal of Computer Applications, 2023, 43(12): 3911-3917.

廉飞宇, 张良, 王杰栋, 靳于康, 柴玉. 基于图模型与注意力机制的室外场景点云分割模型[J]. 《计算机应用》唯一官方网站, 2023, 43(12): 3911-3917.

Figures/Tables 11

Fig.1 Framework of A-Edge-SPG graph neural network

Fig.2 Framework of superpoint information extraction model

Fig.3 Schematic diagram of EdgeConv module

Fig.4 A-EdgeConv module

Tab.1 Hyperedge feature definition

特征名称	维度	定义
偏移量均值	3	$p ¯$
偏移量标准差	3	$ε p$
重心偏移量	3	$p ¯ S - p ¯ T$
长度比	1	$l o g λ 1, S / l o g λ 1, T$
面积比	1	$l o g λ 1, S λ 2, S / l o g λ 1, T λ 2, T$
体积比	1	$l o g λ 1, S λ 2, S λ 3, S / l o g λ 1, T λ 2, T λ 3, T$
点数比	1	$l o g S s u m / l o g T s u m$

Tab.1 Hyperedge feature definition

特征名称	维度	定义
偏移量均值	3	$p ¯$
偏移量标准差	3	$ε p$
重心偏移量	3	$p ¯ S - p ¯ T$
长度比	1	$l o g λ 1, S / l o g λ 1, T$
面积比	1	$l o g λ 1, S λ 2, S / l o g λ 1, T λ 2, T$
体积比	1	$l o g λ 1, S λ 2, S λ 3, S / l o g λ 1, T λ 2, T λ 3, T$
点数比	1	$l o g S s u m / l o g T s u m$

Fig.5 Schematic diagram of GRU

Tab. 2 Comparison of OA， mAA and mIoU between two models

模型	OA	mAA	mIoU
SPG-Net	94.4	77.8	66.6
A-Edge-SPG	96.2	80.6	71.7

Tab.3 mAA comparison of different types

类别	SPG-Net	A-Edge-SPG
人造地面	98.1	98.2
自然地面	96.4	97.5
高植被	77.1	84.7
矮植被	56.5	60.9
建筑	96.6	98.7
人造景观	36.2	37.8
移动物体	68.1	68.5
汽车	94.3	99.0

Fig.6 Results of bildstein_station5 data semantic segmentation

Tab. 4 Results of segmentation accuracy for different nearest neighbor numbers

邻近点数k	总体分割精度/%	平均分割精度/%
5	95.6	80.2
10	96.2	80.6
15	95.3	79.5
20	94.3	79.0

Fig. 7 Comparison of time spent by two methods

References 31

1	YANG B， WEI Z， LI Q， et al. Semiautomated building facade footprint extraction from mobile LiDAR point clouds［J］. IEEE Geoscience and Remote Sensing Letters， 2013， 10（4）： 766-770. 10.1109/lgrs.2012.2222342
2	YANG B， DONG Z， ZHAO G， et al. Hierarchical extraction of urban objects from mobile laser scanning data［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2015， 99： 45-57. 10.1016/j.isprsjprs.2014.10.005
3	ZHANG J， LIN X. Advances in fusion of optical imagery and LiDAR point cloud applied to photogrammetry and remote sensing［J］. International Journal of Image and Data Fusion， 2017， 8（1）： 1-31. 10.1080/19479832.2016.1160960
4	范士俊，张爱武，胡少兴，等. 基于随机森林的机载激光全波形点云数据分类方法［J］. 中国激光， 2013， 40（9）： No.0914001. 10.3788/cjl201340.0914001
	FAN S J， ZHANG A W， HU S X， et al. A method of classification for airborne full waveform LiDAR data based on random forest［J］. Chinese Journal of Lasers， 2013， 40（9）： No.0914001. 10.3788/cjl201340.0914001
5	杨必胜，梁福逊，黄荣刚. 三维激光扫描点云数据处理研究进展、挑战与趋势［J］. 测绘学报， 2017， 46（10）： 1509-1516. 10.11947/j.AGCS.2017.20170351
	YANG B S， LIANG F X， HUANG R G. Progress， challenges and perspectives of 3D LiDAR point cloud processing［J］. Acta Geodaetica et Cartographica Sinica， 2017， 46（10）： 1509-1516. 10.11947/j.AGCS.2017.20170351
6	郭波，黄先锋，张帆，等. 顾及空间上下文关系的JointBoost点云分类及特征降维［J］. 测绘学报， 2013， 42（5）： 715-721.
	GUO B， HUANG X F， ZHANG F， et al. Points cloud classification using JointBoost combined with contextual information for feature reduction［J］. Acta Geodaetica et Cartographica Sinica， 2013， 42（5）： 715-721.
7	WEINMANN M， JUTZI B， HINZ S， et al. Semantic point cloud interpretation based on optimal neighborhoods， relevant features and efficient classifiers ［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2015， 105： 286-304. 10.1016/j.isprsjprs.2015.01.016
8	LIU W， SUN J， LI W， et al. Deep learning on point clouds and its application： a survey ［J］. Sensors， 2019， 19（19）： No.4188. 10.3390/s19194188
9	BENGIO Y. Learning deep architectures for AI［J］. Foundations and Trends in Machine Learning， 2009， 2（1）： 1-127. 10.1561/2200000006
10	杨柳，刘启亮，袁浩涛. 城市激光点云语义分割典型方法对比研究［J］. 地理与地理信息科学， 2021， 37（1）：17-25. 10.3969/j.issn.1672-0504.2021.01.004
	YANG L， LIU Q L， YUAN H T. A comparative study on typical methods for semantic segmentation of laser point clouds in urban areas ［J］. Geography and Geo-Information Science， 2021， 37（1）：17-25. 10.3969/j.issn.1672-0504.2021.01.004
11	MATURANA D， SCHERER S. VoxNet： a 3D convolutional neural network for real-time object recognition ［C］// Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2015： 922-928. 10.1109/iros.2015.7353481
12	WU Z， SONG S， KHOSLA A， et al. 3D ShapeNets： a deep representation for volumetric shapes［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 1912-1920. 10.1109/cvpr.2015.7298801
13	YI L， KIM V G， CEYLAN D， et al. A scalable active framework for region annotation in 3D shape collections［J］. ACM Transactions on Graphics， 2016， 35（6）： No.210. 10.1145/2980179.2980238
14	QI C R， SU H， MO K， et al. PointNet： deep learning on point sets for 3D classification and segmentation ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 77-85. 10.1109/cvpr.2017.16
15	QI C R， YI L， SU H， et al. PointNet++： deep hierarchical feature learning on point sets in a metric space ［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017：5101-5114.
16	WANG Y， SUN Y， LIU Z， et al. Dynamic graph CNN for learning on point clouds［J］. ACM Transactions on Graphics， 2019， 38（5）： No.146. 10.1145/3326362
17	赵中阳，程英蕾，释小松，等. 基于多尺度特征和PointNet的LiDAR点云地物分类方法［J］. 激光与光电子学进展， 2019， 56（5）： No.052804. 10.3788/lop56.052804
	ZHAO Z Y， CHENG Y L， SHI X S， et al. Terrain classification of LiDAR point cloud method based on multi-scale features and PointNet ［J］. Laser and Optoelectronics Progress， 2019， 56（5）： No.052804. 10.3788/lop56.052804
18	马京晖，潘巍，王茹. 基于K-means聚类的三维点云分类［J］. 计算机工程与应用， 2020， 56（17）：181-186. 10.3778/j.issn.1002-8331.1909-0305
	MA J H， PAN W， WANG R. 3D point cloud classification based on K-means clustering［J］. Computer Engineering and Applications， 2020， 56（17）：181-186. 10.3778/j.issn.1002-8331.1909-0305
19	罗海峰，方莉娜，陈崇成，等. 基于DBN的车载激光点云路侧多目标提取［J］. 测绘学报， 2018， 47（2）： 234-246. 10.11947/j.AGCS.2018.20170524
	LUO H F， FANG L N， CHEN C C， et al. Roadside multiple objects extraction from mobile laser scanning point cloud based on DBN［J］. Acta Geodaetica et Cartographica Sinica， 2018， 47（2）： 234-246. 10.11947/j.AGCS.2018.20170524
20	WANG Z， ZHANG L， ZHANG L， et al. A Deep Neural Network with Spatial Pooling （DNNSP） for 3-D point cloud classification［J］. IEEE Transactions on Geoscience and Remote Sensing， 2018， 56（8）： 4594-4604. 10.1109/tgrs.2018.2829625
21	LANDRIEU L， SIMONOVSKY M. Large-scale point cloud semantic segmentation with superpoint graphs ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 4558-4567. 10.1109/cvpr.2018.00479
22	FILLIN S， PFEIFER N. Segmentation of airborne laser scanning data using a slope adaptive neighborhood ［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2006， 60（2）： 71-80. 10.1016/j.isprsjprs.2005.10.005
23	DEMANTKÉ J， MALLET C， DAVID N， et al. Dimensionality based scale selection in 3D lidar point clouds［J］. The International Archives of the Photogrammetry， Remote Sensing and Spatial Information Sciences， 2012， 38（5）： 97-102.
24	KOLMOGOROV V， ZABIN R. What energy functions can be minimized via graph cuts？［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2004， 26（2）： 147-159. 10.1109/tpami.2004.1262177
25	LANDRIEU L， OBOZINSKI G. Cut Pursuit： fast algorithms to learn piecewise constant functions on general weighted graphs ［J］. SIAM Journal on Imaging Sciences， 2017， 10（4）： 1724-1766. 10.1137/17m1113436
26	任欢，王旭光. 注意力机制综述［J］. 计算机应用， 2021， 41（S1）：1-6. 10.11772/j.issn.1001-9081.2020101634
	REN H， WANG X G. Review of attention mechanisms［J］. Journal of Computer Applications， 2021， 41（S1）： 1-6. 10.11772/j.issn.1001-9081.2020101634
27	朱张莉，饶元，吴渊，等.注意力机制在深度学习中的研究进展［J］. 中文信息学报， 2019， 33（6）： 1-11. 10.3969/j.issn.1003-0077.2019.06.001
	ZHU Z L， RAO Y， WU Y， et al. Research progress of attention mechanism in deep learning ［J］. Journal of Chinese Information Processing， 2019， 33（6）： 1-11. 10.3969/j.issn.1003-0077.2019.06.001
28	杨丽，吴雨茜，王俊丽，等.循环神经网络研究综述［J］. 计算机应用， 2018， 38（S2）： 1-6， 26.
	YANG L， WU Y X， WANG J L， et al. Research on recurrent neural network［J］. Journal of Computer Applications， 2018， 38（S2）： 1-6， 26.
29	马超. 融合卷积神经网络和循环神经网络的车轮目标检测［J］. 测绘通报， 2020（8）：139-143.
	MA C. Wheel detection integrating convolutional neural network and recurrent neural network［J］. Bulletin of Surveying and Mapping， 2020（8）：139-143.
30	HACKEL T， SAVINOV N， LADICKY L， et al. Semantic3D.net： a new large-scale point cloud classification benchmark ［J］. ISPRS Annals of the Photogrammetry， Remote Sensing and Spatial Information Sciences， 2017， IV-1/W1： 91-98. 10.5194/isprs-annals-iv-1-w1-91-2017
31	BENTLEY J L. Multidimensional binary search trees used for associative searching ［J］. Communications of the ACM， 1975， 18（9）： 509-517. 10.1145/361002.361007

[1]	Yuwei DING, Hongbo SHI, Jie LI, Min LIANG. Image denoising network based on local and global feature decoupling [J]. Journal of Computer Applications, 2024, 44(8): 2571-2579.
[2]	Boyue WANG, Yingxiang LI, Jiandan ZHONG. Segmentation network for day and night ground-based cloud images based on improved Res-UNet [J]. Journal of Computer Applications, 2024, 44(4): 1310-1316.
[3]	Wei LI, Ling CHEN, Xiuyuan XU, Min ZHU, Jixiang GUO, Kai ZHOU, Hao NIU, Yuchen ZHANG, Shanye YI, Yi ZHANG, Fengming LUO. Interstitial lung disease segmentation algorithm based on multi-task learning [J]. Journal of Computer Applications, 2024, 44(4): 1285-1293.
[4]	Pengfei ZHANG, Litao HAN, Hengjian FENG, Hongmei LI. Point cloud semantic segmentation based on attention mechanism and global feature optimization [J]. Journal of Computer Applications, 2024, 44(4): 1086-1092.
[5]	Ning WU, Yangyang LUO, Huajie XU. Semantic segmentation method for remote sensing images based on multi-scale feature fusion [J]. Journal of Computer Applications, 2024, 44(3): 737-744.
[6]	Ziyi LI, Tingting QU, Qianpeng CHONG, Jindong XU. Remote sensing image segmentation network based on fuzzy multiscale features [J]. Journal of Computer Applications, 2024, 44(11): 3581-3586.
[7]	Yongjiang LIU, Bin CHEN. Pixel-level unsupervised industrial anomaly detection based on multi-scale memory bank [J]. Journal of Computer Applications, 2024, 44(11): 3587-3594.
[8]	Qiumei ZHENG, Weiwei NIU, Fenghua WANG, Dan ZHAO. Dual-branch real-time semantic segmentation network based on detail enhancement [J]. Journal of Computer Applications, 2024, 44(10): 3058-3066.
[9]	Di ZHOU, Zili ZHANG, Jia CHEN, Xinrong HU, Ruhan HE, Jun ZHANG. Stomach cancer image segmentation method based on EfficientNetV2 and object-contextual representation [J]. Journal of Computer Applications, 2023, 43(9): 2955-2962.
[10]	Shuai ZHENG, Xiaolong ZHANG, He DENG, Hongwei REN. 3D liver image segmentation method based on multi-scale feature fusion and grid attention mechanism [J]. Journal of Computer Applications, 2023, 43(7): 2303-2310.
[11]	Yubin GUO, Xiang WEN, Pan LIU, Ximing LI. Cross-modal person re-identification relation network based on dual-stream structure [J]. Journal of Computer Applications, 2023, 43(6): 1803-1810.
[12]	Bin LU, Jielin LIU. Semantic segmentation for 3D point clouds based on feature enhancement [J]. Journal of Computer Applications, 2023, 43(6): 1818-1825.
[13]	Zhixiong ZHENG, Jianhua LIU, Shuihua SUN, Ge XU, Honghui LIN. Aspect-based sentiment analysis model fused with multi-window local information [J]. Journal of Computer Applications, 2023, 43(6): 1796-1802.
[14]	Quan YUAN, Yunpeng XU, Chengliang TANG. Document-level relation extraction method based on path labels [J]. Journal of Computer Applications, 2023, 43(4): 1029-1035.
[15]	Xuedong HE, Shibin XUAN, Kuan WANG, Mengnan CHEN. DeepLabV3+ image segmentation algorithm fusing cumulative distribution function and channel attention mechanism [J]. Journal of Computer Applications, 2023, 43(3): 936-942.

Outdoor scene point cloud segmentation model based on graph model and attention mechanism

基于图模型与注意力机制的室外场景点云分割模型

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 31

Related Articles 15

Recommended Articles

Metrics