Robust RGB-D SLAM system incorporating instance segmentation and clustering in dynamic environment

doi:10.11772/j.issn.1001-9081.2022020261

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (4): 1220-1225.DOI: 10.11772/j.issn.1001-9081.2022020261

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Robust RGB-D SLAM system incorporating instance segmentation and clustering in dynamic environment

Tianzouzi XIAO¹(), Xiaobo ZHOU², Xin LUO¹, Qipeng TANG¹

^1.School of Artificial Intelligence and Automation，Huazhong University of Science and Technology，Wuhan Hubei 430074，China
^2.COMAC Shanghai Aircraft Manufacturing Company Limited，Shanghai 201324，China

Received:2022-03-09 Revised:2022-05-25 Accepted:2022-05-31 Online:2022-08-16 Published:2023-04-10
Contact: Tianzouzi XIAO
About author:ZHOU Xiaobo， born in 1992， assistant engineer. His research interests include design of civil aircraft process equipment.
LUO Xin， born in 1986， Ph. D.， lecturer. His research interests include computer vision based fault detection.
TANG Qipeng， born in 1991， Ph. D.， lecturer. His research interests include digital integrated control.
Supported by:
National Natural Science Foundation of China(51907174)

动态环境下结合实例分割与聚类的鲁棒RGB-D SLAM系统

肖田邹子¹(), 周小博², 罗欣¹, 唐其鹏¹

^1.华中科技大学人工智能与自动化学院，武汉 430074
^2.中国商飞上海飞机制造有限公司，上海 201324

通讯作者: 肖田邹子
作者简介:周小博（1992—），男，江苏启东人，助理工程师，主要研究方向：民用航空器工艺装备设计；
罗欣（1986—），男，河南信阳人，讲师，博士，主要研究方向：基于计算机视觉的故障检测；
唐其鹏（1991—），男，江西南昌人，讲师，博士，主要研究方向：数字集成化控制。
基金资助:
国家自然科学基金资助项目(51907174)

Abstract

Abstract:

Visual Simultaneous Location And Mapping （VSLAM） technology is commonly used for indoor robot navigation and perception. However， the pose estimation method of VSLAM aims at static environment， and might lead to the location and mapping failure when moving objects exist in the scene. To solve this problem， an Instance Segmentation and Clustering SLAM （ISC-SLAM） system was proposed. In this system， the instance segmentation network was used to generate the possibility masks of dynamic objects in the scene， and the dynamic points in the scene were detected by using the multi-view geometry method during the segmentation. After matching the obtained possibility masks and the detected dynamic points， the accurate dynamic masks of moving objects were determined. The feature points of the dynamic objects were able to be deleted by using the dynamic masks and then the position of camera was estimated accurately by using the remained static feature points. To solve the under-segmentation problem of the instance segmentation network， the depth filling algorithm and clustering algorithm were applied to ensure the completed deletion elimination of dynamic feature points. Finally， the moving objects obscured background was reconstructed， and the static point cloud map was built with the correct camera pose. Experimental results on Technical University of Munich （TUM） dataset demonstrate that the proposed system can achieve robust positioning and mapping while ensuring real-time performance in dynamic environment.

Key words: Simultaneous Location And Mapping (SLAM), instance segmentation, clustering, dynamic environment, static dense point cloud map

摘要：

视觉同步定位与建图（VSLAM）技术常常用于室内机器人的导航与感知，然而VSLAM的位姿估算方法是针对静态环境的，当场景中存在运动对象时，可能会导致定位和建图失败。针对此问题，提出了一个结合实例分割与聚类的VSLAM系统。所提系统使用实例分割网络生成场景中动态对象的概率掩膜，同时利用多视图几何的方法检测场景中的动态点，并将检测到的动态点与获得的概率掩膜匹配之后确定动态物体的精确动态掩膜；利用动态掩膜删除动态物体的特征点，然后利用剩余的静态特征点准确估计摄像机的位置。为了解决实例分割网络欠分割的问题，采用深度填充算法和聚类算法保证动态特征点完全删除。最后，重建图片被动态物体遮挡的背景，在正确的相机位姿下建立静态稠密点云地图。在公开的TUM（Technical University of Munich）数据集上的实验结果表明，在动态环境中，所提系统在保证实时性的同时能实现鲁棒的定位与建图。

关键词: 同步定位与建图, 实例分割, 聚类, 动态环境, 静态稠密点云地图

CLC Number:

TP242.6

Tianzouzi XIAO, Xiaobo ZHOU, Xin LUO, Qipeng TANG. Robust RGB-D SLAM system incorporating instance segmentation and clustering in dynamic environment[J]. Journal of Computer Applications, 2023, 43(4): 1220-1225.

肖田邹子, 周小博, 罗欣, 唐其鹏. 动态环境下结合实例分割与聚类的鲁棒RGB-D SLAM系统[J]. 《计算机应用》唯一官方网站, 2023, 43(4): 1220-1225.

Figures/Tables 16

References 22

1	SCHLEICHER D， BERGASA L M， OCAÑA M， et al. Real-time hierarchical GPS aided visual SLAM on urban environments［C］// Proceedings of the 2009 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2009： 4381-4386. 10.1109/robot.2009.5152370
2	MUR-ARTAL R， TARDÓS J D. ORB-SLAM2： an open-source SLAM system for monocular， stereo and RGB-D cameras［J］. IEEE Transactions on Robotics， 2017， 33（5）： 1255-1262. 10.1109/tro.2017.2705103
3	NEWCOMBE R A， LOVEGROVE S J， DAVISON A J. DTAM： dense tracking and mapping in real-time［C］// Proceedings of the 2011 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2011： 2320-2327. 10.1109/iccv.2011.6126513
4	YANG Z F， SHEN S J. Monocular visual-inertial state estimation with online initialization and camera-IMU extrinsic calibration［J］. IEEE Transactions on Automation Science and Engineering， 2017， 14（1）： 39-51. 10.1109/tase.2016.2550621
5	FORSTER C， CARLONE L， DELLAERT F， et al. On-manifold preintegration for real-time visual-inertial odometry［J］. IEEE Transactions on Robotics， 2017， 33（1）： 1-21. 10.1109/tro.2016.2597321
6	ZOU D P， WU Y X， PEI L， et al. StructVIO： visual-inertial odometry with structural regularity of man-made environments［J］. IEEE Transactions on Robotics， 2019， 35（4）： 999-1013. 10.1109/tro.2019.2915140
7	QIN T， LI P L， SHEN S J. VINS-Mono： a robust and versatile monocular visual-inertial state estimator［J］. IEEE Transactions on Robotics， 2018， 34（4）： 1004-1020. 10.1109/tro.2018.2853729
8	MUR-ARTAL R， TARDÓS J D. Visual-inertial monocular SLAM with map reuse［J］. IEEE Robotics and Automation Letters， 2017， 2（2）： 796-803. 10.1109/lra.2017.2653359
9	YAO E L， ZHANG H X， XU H， et al. Robust RGB-D visual odometry based on edges and points［J］. Robotics and Autonomous Systems， 2018， 107： 209-220. 10.1016/j.robot.2018.06.009
10	LI S L， LEE D. RGB-D SLAM in dynamic environments using static point weighting［J］. IEEE Robotics and Automation Letters， 2017， 2（4）： 2263-2270. 10.1109/lra.2017.2724759
11	BESCOS B， FÁCIL J M， CIVERA J， et al. DynaSLAM： tracking， mapping， and inpainting in dynamic scenes［J］. IEEE Robotics and Automation Letters， 2018， 3（4）： 4076-4083. 10.1109/lra.2018.2860039
12	HE K M， GKIOXARI G， DOLLÁR P， et al. Mask R-CNN［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017：2980-2988. 10.1109/iccv.2017.322
13	ZHAO L L， LIU Z L， CHEN J W， et al. A compatible framework for RGB-D SLAM in dynamic scenes［J］. IEEE Access， 2019， 7： 75604-75614. 10.1109/access.2019.2922733
14	XIE W F， LIU P X， ZHENG M H. Moving object segmentation and detection for robust RGBD-SLAM in dynamic environments［J］. IEEE Transactions on Instrumentation and Measurement， 2021， 70： No.5001008. 10.1109/tim.2020.3026803
15	徐少杰，曹雏清，王永娟. 视觉SLAM在室内动态场景中的应用研究［J］. 计算机工程与应用， 2021， 57（8）：175-179. 10.3778/j.issn.1002-8331.2009-0021
	XU S J， CAO C Q， WANG Y J. Application research of visual SLAM in indoor dynamic scenes［J］. Computer Engineering and Applications， 2021， 57（8）： 175-179. 10.3778/j.issn.1002-8331.2009-0021
16	席志红，温家旭. 基于目标检测的室内动态场景定位与建图［J］. 计算机应用， 2022， 42（9）：2853-2857. 10.11772/j.issn.1001-9081.2021061077
	XI Z H， WEN J X. Indoor dynamic scene localization and mapping based on target detection［J］. Journal of Computer Applications， 2022， 42（9）：2853-2857. 10.11772/j.issn.1001-9081.2021061077
17	LEE Y， PARK J. CenterMask： real-time anchor-free instance segmentation［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 13903-13912. 10.1109/cvpr42600.2020.01392
18	高翔，张涛，刘毅，等. 视觉SLAM十四讲：从理论到实践［M］. 2版.北京：电子工业出版社， 2019：152-153.
	GAO X， ZHANG T， LIU Y， et al. 14 Lectures on Visual SLAM： From Theory to Practice［M］. 2nd ed. Beijing： Publishing House of Electronics Industry， 2019： 152-153.
19	LIN T Y， MAIRE M， BELONGIE S， et al. Microsoft COCO： common objects in context［J］// Proceedings of the 2014 European Conference on Computer Vision， LNCS 8693. Cham： Springer， 2014： 740-755.
20	STURM J， ENGELHARD N， ENDRES F， et al. A benchmark for the evaluation of RGB-D SLAM systems［C］// Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2012： 573-580. 10.1109/iros.2012.6385773
21	SUN Y X， LIU M， MENG M Q H. Improving RGB-D SLAM in dynamic environments： a motion removal approach［J］. Robotics and Autonomous Systems， 2017， 89： 110-122. 10.1016/j.robot.2016.11.012
22	GRUPP M. Evo： Python package for the evaluation of odometry and SLAM［EB/OL］. ［2022-01-20］..

环境	数据集	绝对轨迹误差/m
		ORB-SLAM2		ISC-SLAM
		RMSE	S.D.	RMSE	S.D.
静态环境	fr1_xyz	0.009 4	0.005 2	0.009 6	0.005 2
	fr1_rpy	0.022 9	0.013 7	0.022 7	0.013 8
	fr2_desk	0.009 6	0.003 1	0.008 7	0.003 1
	fr3_long_office	0.010 9	0.004 9	0.009 0	0.003 6
低动态环境	fr2_desk_person	0.006 2	0.002 9	0.005 8	0.002 8
	fr3_sitting_xyz	0.009 7	0.004 9	0.010 9	0.005 2
	fr3_sitting_halfsphere	0.040 0	0.018 0	0.020 5	0.009 8
	fr3_sitting_static	0.008 9	0.004 3	0.006 0	0.002 9
高动态环境	fr3_walking_rpy	0.769 0	0.371 0	0.035 8	0.020 7
	fr3_walking_xyz	0.727 0	0.376 0	0.014 8	0.007 7
	fr3_walking_static	0.303 0	0.126 0	0.007 3	0.003 3
	fr3_walking_halfsphere	0.507 0	0.286 0	0.030 3	0.014 6

环境	数据集	绝对轨迹误差/m
		ORB-SLAM2		ISC-SLAM
		RMSE	S.D.	RMSE	S.D.
静态环境	fr1_xyz	0.009 4	0.005 2	0.009 6	0.005 2
	fr1_rpy	0.022 9	0.013 7	0.022 7	0.013 8
	fr2_desk	0.009 6	0.003 1	0.008 7	0.003 1
	fr3_long_office	0.010 9	0.004 9	0.009 0	0.003 6
低动态环境	fr2_desk_person	0.006 2	0.002 9	0.005 8	0.002 8
	fr3_sitting_xyz	0.009 7	0.004 9	0.010 9	0.005 2
	fr3_sitting_halfsphere	0.040 0	0.018 0	0.020 5	0.009 8
	fr3_sitting_static	0.008 9	0.004 3	0.006 0	0.002 9
高动态环境	fr3_walking_rpy	0.769 0	0.371 0	0.035 8	0.020 7
	fr3_walking_xyz	0.727 0	0.376 0	0.014 8	0.007 7
	fr3_walking_static	0.303 0	0.126 0	0.007 3	0.003 3
	fr3_walking_halfsphere	0.507 0	0.286 0	0.030 3	0.014 6

环境	数据集	平移RMSE/m			旋转RMSE/（°）
环境	数据集	SPW-IAICP	EP-VO	ISC-SLAM	SPW-IAICP	EP-VO	ISC-SLAM
静态环境	fr2_desk	0.017 3	0.007 2	0.007 2	0.726 6	0.450 6	0.447 7
静态环境	fr3_long_office	0.016 8	0.008 4	0.008 1	0.801 2	0.462 7	0.457 8
低动态环境	fr2_desk_person	0.017 3	0.006 8	0.006 7	0.821 3	0.435 9	0.434 0
	fr3_sitting_xyz	0.021 9	0.010 9	0.014 1	0.846 6	0.471 7	0.506 0
	fr3_sitting_halfsphere	0.038 9	0.016 8	0.025 5	1.883 6	0.569 0	0.722 4
高动态环境	fr3_walking_rpy	0.225 2	0.056 1	0.048 1	5.690 2	1.021 0	0.990 0
	fr3_walking_xyz	0.065 1	0.029 2	0.019 5	1.644 2	0.584 7	0.608 3
	fr3_walking_static	0.032 7	0.010 1	0.009 3	0.808 5	0.257 1	0.249 2

环境	数据集	平移RMSE/m			旋转RMSE/（°）
环境	数据集	SPW-IAICP	EP-VO	ISC-SLAM	SPW-IAICP	EP-VO	ISC-SLAM
静态环境	fr2_desk	0.017 3	0.007 2	0.007 2	0.726 6	0.450 6	0.447 7
静态环境	fr3_long_office	0.016 8	0.008 4	0.008 1	0.801 2	0.462 7	0.457 8
低动态环境	fr2_desk_person	0.017 3	0.006 8	0.006 7	0.821 3	0.435 9	0.434 0
	fr3_sitting_xyz	0.021 9	0.010 9	0.014 1	0.846 6	0.471 7	0.506 0
	fr3_sitting_halfsphere	0.038 9	0.016 8	0.025 5	1.883 6	0.569 0	0.722 4
高动态环境	fr3_walking_rpy	0.225 2	0.056 1	0.048 1	5.690 2	1.021 0	0.990 0
	fr3_walking_xyz	0.065 1	0.029 2	0.019 5	1.644 2	0.584 7	0.608 3
	fr3_walking_static	0.032 7	0.010 1	0.009 3	0.808 5	0.257 1	0.249 2

环境	数据集	SPW-IAICP		EP-VO		SD-SLAM		DynaSLAM		ISC-SLAM
环境	数据集	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.	RMSE	S.D.
低动态环境	fr2_desk_person	0.048 4	0.023 7	0.006 0	0.002 7	0.007 0	―	0.072 0	0.017 2	0.005 8	0.002 8
	fr3_sitting_xyz	0.039 7	0.020 6	0.008 7	0.004 1	0.013 0	―	0.015 0	0.006 5	0.010 9	0.005 2
	fr3_sitting_halfsphere	0.043 2	0.024 6	0.014 8	0.007 4	0.019 0	―	0.017 0	0.011 3	0.020 5	0.009 8
高动态环境	fr3_walking_rpy	0.179 1	0.116 1	0.038 8	0.024 1	0.033 0	―	0.035 0	0.043 7	0.035 8	0.020 7
	fr3_walking_xyz	0.060 1	0.033 0	0.022 2	0.012 2	0.014 0	―	0.015 0	0.008 6	0.014 8	0.007 7
	fr3_walking_static	0.026 1	0.012 2	0.007 8	0.004 0	0.010 0	―	0.006 0	0.003 4	0.007 3	0.003 3
	fr3_walking_halfsphere	0.048 9	0.726 6	0.032 8	0.018 4	0.028 0	―	0.025 0	0.016 1	0.030 3	0.014 6

Robust RGB-D SLAM system incorporating instance segmentation and clustering in dynamic environment

动态环境下结合实例分割与聚类的鲁棒RGB-D SLAM系统

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 22

Related Articles 15

Recommended Articles

Metrics

系统	fr3_walking_xyz		fr3_walking_halfsphere		fr3_walking_static
系统	RMSE	Mean	RMSE	Mean	RMSE	Mean
YOLOv4+ORB-SLAM2	0.017 4	0.014 7	0.032 8	0.028 0	0.008 0	0.006 9
GMS+ORB-SLAM2	0.016 5	0.014 3	0.029 8	0.025 0	0.007 9	0.007 0
ISC-SLAM	0.014 8	0.012 7	0.030 3	0.028 3	0.007 3	0.007 0

系统	平均定位跟踪单帧耗时
ORB-SLAM2	40.278
DynaSLAM	1 640.290
SD-SLAM	1 100.000
GMS+ORB-SLAM2	163.972
ISC-SLAM	87.931

[1]	Shunyong LI, Shiyi LI, Rui XU, Xingwang ZHAO. Incomplete multi-view clustering algorithm based on self-attention fusion [J]. Journal of Computer Applications, 2024, 44(9): 2696-2703.
[2]	Xun SUN, Ruifeng FENG, Yanru CHEN. Monocular 3D object detection method integrating depth and instance segmentation [J]. Journal of Computer Applications, 2024, 44(7): 2208-2215.
[3]	Qing WANG, Jieyu ZHAO, Xulun YE, Nongxiao WANG. Enhanced deep subspace clustering method with unified framework [J]. Journal of Computer Applications, 2024, 44(7): 1995-2003.
[4]	Yao DONG, Yixue FU, Yongfeng DONG, Jin SHI, Chen CHEN. Survey of incomplete multi-view clustering [J]. Journal of Computer Applications, 2024, 44(6): 1673-1682.
[5]	Xiaoxia JIANG, Ruizhang HUANG, Ruina BAI, Lina REN, Yanping CHEN. Deep event clustering method based on event representation and contrastive learning [J]. Journal of Computer Applications, 2024, 44(6): 1734-1742.
[6]	Tianyu HUANG, Yuanxing LI, Hao CHEN, Zijia GUO, Mingjun WEI. User cluster partitioning method based on weighted fuzzy clustering in ground-air collaboration scenarios [J]. Journal of Computer Applications, 2024, 44(5): 1555-1561.
[7]	Lin GAO, Yu ZHOU, Tak Wu KWONG. Evolutionary bi-level adaptive local feature selection [J]. Journal of Computer Applications, 2024, 44(5): 1408-1414.
[8]	Tongtong XU, Bin XIE, Chunhao ZHANG, Ximei ZHANG. Multi-order nearest neighbor graph clustering algorithm by fusing transition probability matrix [J]. Journal of Computer Applications, 2024, 44(5): 1527-1538.
[9]	Yu DING, Hanlin ZHANG, Rong LUO, Hua MENG. Fuzzy clustering algorithm based on belief subcluster cutting [J]. Journal of Computer Applications, 2024, 44(4): 1128-1138.
[10]	Long CHEN, Xuanlin YU, Wen CHEN, Yi YAO, Wenjing ZHU, Ying JIA, Denghong LI, Zhi REN. Efficient clustered routing protocol for intelligent road cone ad-hoc networks based on non-random clustering [J]. Journal of Computer Applications, 2024, 44(3): 869-875.
[11]	Lin SUN, Menghan LIU. K-means clustering based on adaptive cuckoo optimization feature selection [J]. Journal of Computer Applications, 2024, 44(3): 831-841.
[12]	Cunyi LIAO, Yi ZHENG, Weijin LIU, Huan YU, Shouyin LIU. Decoupling-fusing algorithm for multiple tasks with autonomous driving environment perception [J]. Journal of Computer Applications, 2024, 44(2): 424-431.
[13]	Zhuo ZHANG, Huazhu CHEN. Deep subspace clustering based on multiscale self-representation learning with consistency and diversity [J]. Journal of Computer Applications, 2024, 44(2): 353-359.
[14]	Fuqin DENG, Huifeng GUAN, Chaoen TAN, Lanhui FU, Hongmin WANG, Tinlun LAM, Jianmin ZHANG. Multi-robot reinforcement learning path planning method based on request-response communication mechanism and local attention mechanism [J]. Journal of Computer Applications, 2024, 44(2): 432-438.
[15]	Chenghao YANG, Jie HU, Hongjun WANG, Bo PENG. Incomplete multi-view clustering algorithm based on attention mechanism [J]. Journal of Computer Applications, 2024, 44(12): 3784-3789.