Visual simultaneous localization and mapping based on semantic and optical flow constraints in dynamic scenes

doi:10.11772/j.issn.1001-9081.2021010003

Journal of Computer Applications ›› 2021, Vol. 41 ›› Issue (11): 3337-3344.DOI: 10.11772/j.issn.1001-9081.2021010003

• Multimedia computing and computer simulation • Previous Articles Next Articles

Visual simultaneous localization and mapping based on semantic and optical flow constraints in dynamic scenes

Hao FU, Hegen XU(), Zhiming ZHANG, Shaohua QI

College of Electronic and Information Engineering，Tongji University，Shanghai 201804，China

Received:2021-01-05 Revised:2021-03-12 Accepted:2021-03-19 Online:2021-11-29 Published:2021-11-10
Contact: Hegen XU
About author:FU Hao，born in 1996，M. S. candidate. His research interests include visual simultaneous localization and mapping，deep learning， machine vision
XU Hegen，born in 1972，Ph. D.，associate professor. His research interests include image processing，machine vision，pattern recognition， mobile robotics
ZHANG Zhiming，born in 1978，Ph. D. lecturer. His research interests include embedded system，deep learning
QI Shaohua，born in 1995，M. S. candidate. His research interests include visual simultaneous localization and mapping，deep learning， augmented reality.

动态场景下基于语义和光流约束的视觉同步定位与地图构建

付豪, 徐和根(), 张志明, 齐少华

同济大学电子与信息工程学院，上海 201804

通讯作者: 徐和根
作者简介:付豪（1996—），男，安徽合肥人，硕士研究生，主要研究方向：视觉同步定位与地图构建、深度学习、机器视觉
徐和根（1972—），男，安徽安庆人，副教授，博士，主要研究方向：图像处理、机器视觉、模式识别、移动机器人
张志明（1978—），男，陕西西安人，讲师，博士，主要研究方向：嵌入式系统、深度学习
齐少华（1995—），男，湖南常德人，硕士研究生，主要研究方向：视觉同步定位与地图构建、深度学习、增强现实。

Abstract

Abstract:

For the localization and static semantic mapping problems in dynamic scenes， a Simultaneous Localization And Mapping （SLAM） algorithm in dynamic scenes based on semantic and optical flow constraints was proposed to reduce the impact of moving objects on localization and mapping. Firstly， for each frame of the input， the masks of the objects in the frame were obtained by semantic segmentation， then the feature points that do not meet the epipolar constraint were filtered out by the geometric method. Secondly， the dynamic probability of each object was calculated by combining the object masks with the optical flow， the feature points were filtered by the dynamic probabilities to obtain the static feature points， and the static feature points were used for the subsequent camera pose estimation. Then， the static point cloud was created based on RGB-D images and object dynamic probabilities， and the semantic octree map was built by combining the semantic segmentation. Finally， the sparse semantic map was created based on the static point cloud and the semantic segmentation. Test results on the public TUM dataset show that， in highly dynamic scenes， the proposed algorithm improves the performance on both the absolute trajectory error and relative pose error by more than 95% compared with ORB-SLAM2， and reduces the absolute trajectory error by 41% and 11% compared with DS-SLAM and DynaSLAM respectively， which verifies that the proposed algorithm has better localization accuracy and robustness in highly dynamic scenes. The experimental results of mapping show that the proposed algorithm creates a static semantic map， and the storage space requirement of the sparse semantic map is reduced by 99% compared to that of the point cloud map.

Key words: Simultaneous Localization And Mapping (SLAM), dynamic scene, semantic map, semantic segmentation, dynamic feature point filtering

摘要：

针对动态场景下的定位与静态语义地图构建问题，提出了一种基于语义和光流约束的动态环境下的同步定位与地图构建（SLAM）算法，以降低动态物体对定位与建图的影响。首先，对于输入的每一帧，通过语义分割获得图像中物体的掩模，再通过几何方法过滤不符合极线约束的特征点；接着，结合物体掩模与光流计算出每个物体的动态概率，根据动态概率过滤特征点以得到静态特征点，再利用静态特征点进行后续的相机位姿估计；然后，基于RGB-D图片和物体动态概率建立静态点云，并结合语义分割建立语义八叉树地图。最后，基于静态点云与语义分割创建稀疏语义地图。公共TUM数据集上的测试结果表明，高动态场景下，所提算法与ORB-SLAM2相比，在绝对轨迹误差和相对位姿误差上能取得95%以上的性能提升，与DS-SLAM、DynaSLAM相比分别减小了41%和11%的绝对轨迹误差，验证了该算法在高动态场景中具有较好的定位精度和鲁棒性。地图构建的实验结果表明，所提算法创建了静态语义地图，与点云地图相比，稀疏语义地图的存储空间需求量降低了99%。

关键词: 同步定位与地图构建, 动态场景, 语义地图, 语义分割, 动态特征点过滤

CLC Number:

TP249

Hao FU, Hegen XU, Zhiming ZHANG, Shaohua QI. Visual simultaneous localization and mapping based on semantic and optical flow constraints in dynamic scenes[J]. Journal of Computer Applications, 2021, 41(11): 3337-3344.

付豪, 徐和根, 张志明, 齐少华. 动态场景下基于语义和光流约束的视觉同步定位与地图构建[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3337-3344.

Figures/Tables 21

Fig. 1 Overall framework of proposed algorithm

Fig. 2 Framework of dynamic feature point filtering algorithm

Fig. 3 Segmentation results of semantic segmentation network

Fig. 4 Schematic diagram of epipolar geometry

Fig. 5 Results of direct calculation of optical flow

Fig. 6 Results of optical flow calculation after correction

Fig. 7 Using moving object masks and optical flow to filter out dynamic points forconstructing static point cloud map

Fig. 8 Point cloud segmentation results based onsemantic segmentation

Fig. 9 Object bounding box

Fig. 10 Flow chart of semantic map updating

Tab. 1 Result comparison of absolute trajectory error

序列	绝对轨迹误差/m								本文算法的性能提升/%
	ORB-SLAM2				本文算法				本文算法的性能提升/%
	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE
W-hs	0.599 9	0.566 3	0.329 7	0.684 5	0.021 7	0.018 4	0.013 6	0.025 6	96.4	96.8	95.9	96.3
W-rpy	0.686 2	0.623 3	0.408 7	0.799 1	0.025 9	0.020 5	0.019 4	0.032 4	96.2	96.7	95.3	95.9
W-static	0.287 6	0.245 9	0.133 9	0.317 3	0.005 9	0.005 4	0.003 1	0.006 7	97.9	97.8	97.6	97.9
W-xyz	0.370 5	0.650 0	0.645 3	0.744 1	0.013 1	0.011 4	0.007 5	0.015 1	96.5	98.2	98.8	98.0
S-hs	0.014 1	0.011 4	0.011 8	0.018 4	0.013 7	0.011 3	0.010 6	0.017 3	2.6	0.7	$-$ 15.0	5.7
S-rpy	0.016 0	0.010 9	0.016 1	0.022 6	0.013 7	0.010 6	0.010 0	0.017 0	14.2	2.9	37.4	25.0

Tab. 1 Result comparison of absolute trajectory error

序列	绝对轨迹误差/m								本文算法的性能提升/%
	ORB-SLAM2				本文算法				本文算法的性能提升/%
	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE
W-hs	0.599 9	0.566 3	0.329 7	0.684 5	0.021 7	0.018 4	0.013 6	0.025 6	96.4	96.8	95.9	96.3
W-rpy	0.686 2	0.623 3	0.408 7	0.799 1	0.025 9	0.020 5	0.019 4	0.032 4	96.2	96.7	95.3	95.9
W-static	0.287 6	0.245 9	0.133 9	0.317 3	0.005 9	0.005 4	0.003 1	0.006 7	97.9	97.8	97.6	97.9
W-xyz	0.370 5	0.650 0	0.645 3	0.744 1	0.013 1	0.011 4	0.007 5	0.015 1	96.5	98.2	98.8	98.0
S-hs	0.014 1	0.011 4	0.011 8	0.018 4	0.013 7	0.011 3	0.010 6	0.017 3	2.6	0.7	$-$ 15.0	5.7
S-rpy	0.016 0	0.010 9	0.016 1	0.022 6	0.013 7	0.010 6	0.010 0	0.017 0	14.2	2.9	37.4	25.0

Tab. 2 Result comparison of translation error of relative pose error

序列	相对平移误差/m								本文算法的性能提升/%
	ORB-SLAM2				本文算法				本文算法的性能提升/%
	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE
W-hs	0.830 9	0.761 2	0.605 4	1.028 1	0.031 5	0.028 4	0.017 3	0.036 0	96.2	96.3	97.1	96.5
W-rpy	0.991 9	0.919 7	0.650 7	1.186 0	0.038 9	0.032 3	0.026 0	0.046 7	96.1	96.5	96.0	96.1
W-static	0.318 4	0.109 2	0.325 6	0.455 4	0.008 5	0.007 9	0.004 1	0.009 4	97.3	92.7	98.7	97.9
W-xyz	0.892 5	0.870 8	0.629 1	1.091 9	0.019 3	0.017 3	0.010 5	0.022 0	97.8	98.0	98.3	98.0
S-hs	0.020 7	0.016 5	0.017 6	0.027 2	0.020 4	0.017 1	0.015 2	0.025 4	1.6	$-$ 3.9	13.6	6.5
S-rpy	0.021 5	0.018 7	0.025 4	0.033 3	0.025 4	0.012 4	0.022 2	0.037 3	$-$ 18.4	33.6	12.8	$-$ 12.0

Tab. 2 Result comparison of translation error of relative pose error

序列	相对平移误差/m								本文算法的性能提升/%
	ORB-SLAM2				本文算法				本文算法的性能提升/%
	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE
W-hs	0.830 9	0.761 2	0.605 4	1.028 1	0.031 5	0.028 4	0.017 3	0.036 0	96.2	96.3	97.1	96.5
W-rpy	0.991 9	0.919 7	0.650 7	1.186 0	0.038 9	0.032 3	0.026 0	0.046 7	96.1	96.5	96.0	96.1
W-static	0.318 4	0.109 2	0.325 6	0.455 4	0.008 5	0.007 9	0.004 1	0.009 4	97.3	92.7	98.7	97.9
W-xyz	0.892 5	0.870 8	0.629 1	1.091 9	0.019 3	0.017 3	0.010 5	0.022 0	97.8	98.0	98.3	98.0
S-hs	0.020 7	0.016 5	0.017 6	0.027 2	0.020 4	0.017 1	0.015 2	0.025 4	1.6	$-$ 3.9	13.6	6.5
S-rpy	0.021 5	0.018 7	0.025 4	0.033 3	0.025 4	0.012 4	0.022 2	0.037 3	$-$ 18.4	33.6	12.8	$-$ 12.0

Tab. 3 Result comparison of rotation error of relative pose error

序列	相对旋转误差/（°）								本文算法的性能提升/%
	ORB-SLAM2				本文算法				本文算法的性能提升/%
	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE	平均数	中位数	SD	RMSE
W-hs	19.485 0	16.638 0	13.369 0	23.631 0	0.755 3	0.704 0	0.372 8	0.842 2	96.1	95.8	97.2	96.4
W-rpy	16.920 0	14.999 0	12.646 0	21.124 0	0.803 4	0.711 0	0.463 0	0.927 2	95.3	95.3	96.3	95.6
W-static	5.569 5	1.886 2	5.634 1	7.922 3	0.246 4	0.233 3	0.114 5	0.271 7	95.6	87.6	98.0	96.6
W-xyz	16.338 0	15.561 0	11.638 0	20.059 0	0.490 4	0.408 6	0.395 2	0.629 8	97.0	97.4	96.6	96.9
S-hs	0.661 3	0.624 3	0.307 8	0.729 4	0.633 5	0.596 4	0.294 4	0.698 5	4.2	4.5	4.3	4.2
S-rpy	0.783 1	0.666 3	0.512 1	0.935 7	0.751 0	0.641 3	0.510 1	0.921 7	4.1	3.8	0.4	1.5

Fig. 11 Photograph of laboratory environment

Tab. 4 Comparison of absolute trajectory error of different algorithms

序列	DS-SLAM	DynaSLAM	本文算法
W-hs	0.030 3	0.025	0.025 6
W-rpy	0.444 2	0.040	0.032 4
W-static	0.008 1	0.009	0.006 7
W-xyz	0.024 7	0.015	0.015 1

Fig. 12 Comparison of trajectories estimated by different algorithms in highly dynamic scenes

Fig. 13 Trajectories estimated by different algorithms in laboratory environment

Tab. 5 Running times of different modules

模块	运行时间/ms
ORB特征提取	9.0
语义分割	169.4
动态特征点过滤	51.3

Fig. 14 Dense point cloud map

Fig. 15 Semantic map

Tab. 6 Comparison of map file storage space

轨迹	地图文件存储空间/MB
轨迹	点云地图	八叉树地图	稀疏语义地图
W-xyz	13.3	0.37	0.004
W-static	15.5	0.18	0.004

References 16

1	权美香，朴松昊，李国.视觉SLAM综述［J］.智能系统学报，2016，11（6）：768-776. 10.11992/tis.201607026
	QUAN M X， PIAO S H， LI G. An overview of visual SLAM ［J］. CAAI Transactions on Intelligent Systems， 2016， 11（6）： 768-776. 10.11992/tis.201607026
2	BRAHMBHATT S， GU J W， KIM K， et al. Geometry-aware learning of maps for camera localization ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 2616-2625. 10.1109/cvpr.2018.00277
3	付梦印，吕宪伟，刘彤，等.基于RGB-D数据的实时SLAM算法［J］.机器人，2015，37（6）：683-692. 10.13973/j.cnki.robot.2015.0683
	FU M Y， LYU X W， LIU T， et al. Real-time SLAM algorithm based on RGB-D data ［J］. Robot， 2015， 37（6）： 683-692. 10.13973/j.cnki.robot.2015.0683
4	CADENA C， CARLONE L， CARRILLO H， et al. Past， present， and future of simultaneous localization and mapping： toward the robust-perception age ［J］. IEEE Transactions on Robotics， 2016， 32（6）： 1309-1332. 10.1109/TRO.2016.2624754
5	FANG Y Q， DAI B. An improved moving target detecting and tracking based on Optical Flow technique and Kalman filter ［C］// Proceedings of the 2009 4th International Conference on Computer Science and Education. Piscataway： IEEE， 2009： 1197-1202. 10.1109/iccse.2009.5228464
6	WANG Y B， HUANG S D. Towards dense moving object segmentation based robust dense RGB-D SLAM in dynamic scenarios ［C］// Proceedings of the 2014 13th International Conference on Control Automation Robotics and Vision. Piscataway： IEEE， 2014： 1841-1846. 10.1109/icarcv.2014.7064596
7	BAKKAY M C， ARAFA M， ZAGROUBA E. Dense 3D SLAM in dynamic scenes using Kinect ［C］// Proceedings of the 2015 7th Iberian Conference on Pattern Recognition and Image Analysis， LNCS9117. Cham： Springer， 2015： 121-129.
8	ZHAO L L， LIU Z L， CHEN J W， et al. A compatible framework for RGB-D SLAM in dynamic scenes ［J］. IEEE Access， 2019， 7： 75604-75614. 10.1109/access.2019.2922733
9	BADRINARAYANAN V， KENDALL A， CIPOLLA R. SegNet： a deep convolutional encoder-decoder architecture for image segmentation ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017， 39（12）： 2481-2495. 10.1109/tpami.2016.2644615
10	CHEN L C， PAPANDREOU G， KOKKINOS I， et al. DeepLab： semantic image segmentation with deep convolutional nets， atrous convolution， and fully connected CRFs ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018， 40（4）： 834-848. 10.1109/tpami.2017.2699184
11	HE K M， GKIOXARI G， DOLLÁR P， et al. Mask R-CNN ［C］// Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2017： 2980-2988. 10.1109/iccv.2017.322
12	ZHONG F W， WANG S， ZHANG Z Q， et al. Detect-SLAM： making object detection and SLAM mutually beneficial ［C］// Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2018： 1001-1010. 10.1109/wacv.2018.00115
13	BESCOS B， FÁCIL J M， CIVERA J， et al. DynaSLAM： tracking， mapping and inpainting in dynamic scenes ［J］. IEEE Robotics and Automation Letters， 2018， 3（4）： 4076-4083. 10.1109/lra.2018.2860039
14	YU C， LIU Z X ， LIU X J， et al. DS-SLAM： a semantic visual SLAM towards dynamic environments ［C］// Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2018： 1168-1174. 10.1109/iros.2018.8593691
15	HORNUNG A， WURM K M， BENNEWITZ M， et al. OctoMap： an efficient probabilistic 3D mapping framework based on octrees ［J］. Autonomous Robots， 2013， 34（3）： 189-206. 10.1007/s10514-012-9321-0
16	WEINZAEPFEL P， REVAUD J， HARCHAOUI Z， et al. DeepFlow： large displacement optical flow with deep matching ［C］// Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2013： 1385-1392. 10.1109/iccv.2013.175

[1]	FENG Xingjie, ZHANG Tianze. Panoptic segmentation algorithm based on grouped convolution for feature fusion [J]. Journal of Computer Applications, 2021, 41(7): 2054-2061.
[2]	HU Die, FENG Ziliang. Light-weight road image semantic segmentation algorithm based on deep learning [J]. Journal of Computer Applications, 2021, 41(5): 1326-1331.
[3]	SHE Yulong, ZHANG Xiaolong, CHENG Ruoqin, DENG Chunhua. Semantic segmentation method based on edge attention model [J]. Journal of Computer Applications, 2021, 41(2): 343-349.
[4]	HUANG Juting, GAO Hongli, DAI Zhikun. Semantic segmentation method of power line on mobile terminals based on encoder-decoder structure [J]. Journal of Computer Applications, 2021, 41(10): 2952-2958.
[5]	ZHENG Sicheng, KONG Linghua, YOU Tongfei, YI Dingrong. Semantic SLAM algorithm based on deep learning in dynamic environment [J]. Journal of Computer Applications, 2021, 41(10): 2945-2951.
[6]	GAO Shiwei, ZHANG Changzhu, WANG Zhuping. Lightweight real-time semantic segmentation algorithm based on separable pyramid [J]. Journal of Computer Applications, 2021, 41(10): 2937-2944.
[7]	FU Lei, REN Dejun, WU Huayun, GAO Ming, QIU Lyu, HU Yunqi. Real-time segmentation algorithm for bubble defects of plastic bottle based on improved Fast-SCNN [J]. Journal of Computer Applications, 2020, 40(6): 1824-1829.
[8]	WEI Wenle, JIN Guodong, TAN Lining, LU Libin, CHEN Danqi. Real-time SLAM algorithm with keyframes determined by inertial measurement unit [J]. Journal of Computer Applications, 2020, 40(4): 1157-1163.
[9]	ZHAO Hong, LIU Xiangdong, YANG Yongjuan. Indoor robot simultaneous localization and mapping based on RGB-D image [J]. Journal of Computer Applications, 2020, 40(12): 3637-3643.
[10]	XI Zhihong, WANG Hongxu, HAN Shuangquan. Fast mismatch elimination algorithm and map-building based on ORB-SLAM2 system [J]. Journal of Computer Applications, 2020, 40(11): 3289-3294.
[11]	SONG Xiaona, RUI Ting, WANG Xinqing. Semantic segmentation method of road environment combined semantic boundary information [J]. Journal of Computer Applications, 2019, 39(9): 2505-2510.
[12]	WANG Yulong, PU Jun, ZHAO Jianghua, LI Jianhui. Detection of new ground buildings based on generative adversarial network [J]. Journal of Computer Applications, 2019, 39(5): 1518-1522.
[13]	HUANG Shuai, FU Guangyuan, WU Ming, YUE Min. Multi-mode filtering object tracking algorithm based on monocular suboptimal parallax under unknown environment [J]. Journal of Computer Applications, 2019, 39(3): 864-868.
[14]	LIU Yalong, LI Jie, WANG Ying, WU Saifei, ZOU Pei. Segmentation model of neonatal punctate white matter lesion based on refined deep residual U-Net [J]. Journal of Computer Applications, 2019, 39(12): 3456-3461.
[15]	XI Zhihong, HAN Shuangquan, WANG Hongxu. Simultaneous localization and semantic mapping of indoor dynamic scene based on semantic segmentation [J]. Journal of Computer Applications, 2019, 39(10): 2847-2851.

Visual simultaneous localization and mapping based on semantic and optical flow constraints in dynamic scenes

动态场景下基于语义和光流约束的视觉同步定位与地图构建

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 21

References 16

Related Articles 15

Recommended Articles

Metrics