Real-time binocular foreground depth estimation algorithm based on sparse convolution

doi:10.11772/j.issn.1001-9081.2021010076

Abstract

Abstract:

To improve the computational efficiency of stereo matching on foreground disparity estimation tasks， aiming at the disadvantage that the general networks use the complete binocular image as input and the input information redundancy is large due to the small proportion of the foreground space in the scene， a real-time target stereo matching algorithm based on sparse convolution was proposed. In order to realize and improve the sparse foreground disparity estimation of the algorithm， firstly， the sparse foreground mask and scene semantic features were obtained by the segmentation algorithm at the same time. Secondly， the sparse convolution was used to extract the spatial features of the foreground sparse region， and scene semantic features were fused with them. Then， the fused features were input into the decoding module for disparity regression. Finally， the foreground truth graph was used as the loss to generate the disparity graph. The test results on ApolloScape dataset show that the accuracy and real-time performance of the proposed algorithm are better than those of the state-of-the-art algorithms PSMNet （Pyramid Stereo Matching Network） and GANet （Guided Aggregation Network）， and the single run time of the algorithm is as low as 60.5 ms. In addition， the proposed algorithm has certain robustness to the foreground occlusion， and can be used for the real-time depth estimation of targets.

Key words: stereo matching, sparse convolution, deep learning, semantic segmentation, attention mechanism

摘要：

为了提高立体匹配算法处理前景视差估计任务的计算效率，针对一般网络采用完全双目图像作为输入，场景内前景空间占比小而导致输入信息冗余度大的缺点，提出了一种基于稀疏卷积的目标实时立体匹配算法。为实现和改善算法对稀疏前景的视差估计，首先利用分割算法同时获得稀疏前景掩膜和场景语义特征；其次通过稀疏卷积提取稀疏前景区域的空间特征后与场景语义特征相融合，并将融合特征输入到解码模块进行视差回归；最后以前景真值图作为损失生成视差图。在ApolloScape数据集上的测试结果表明，所提算法的准确性和实时性均优于先进算法PSMNet和GANet，且算法的单次运行时间低至60.5 ms，对目标遮挡具有一定的鲁棒性，可用于目标实时深度估计。

关键词: 立体匹配, 稀疏卷积, 深度学习, 语义分割, 注意力机制

CLC Number:

TP391.41

Zhehan QIU, Yang LI. Real-time binocular foreground depth estimation algorithm based on sparse convolution[J]. Journal of Computer Applications, 2021, 41(12): 3680-3685.

邱哲瀚, 李扬. 基于稀疏卷积的前景实时双目深度估计算法[J]. 《计算机应用》唯一官方网站, 2021, 41(12): 3680-3685.

Figures/Tables 12

Fig. 1 SPSMNet model architecture

Fig. 2 Spatial feature extraction module

Fig. 3 Semantic attention module

Fig. 4 Disparity channel activation mode

Fig. 5 Disparity regression module

Fig. 6 Five group of samples in ApolloScape dataset

Fig. 7 Results of disparity estimation of PSMNet， GANet andproposed algorithms to Fig. 6

Tab. 1 Quantitative results on ApolloScape dataset

算法	评价指标/%					运行时间/s
算法	ARE	MAE	2PE	3PE	5PE	运行时间/s
PSMNet	9.76	171	25.71	14.48	5.73	0.105 2
GANet	9.07	155	22.96	12.50	5.08	0.617 3
本文算法	7.92	147	22.05	11.16	3.94	0.060 5

Fig. 8 Two group of samples of ApolloScape test dataset

Fig. 9 Results of disparity estimation of three models to Fig. 8

Tab. 2 Ablation experiment

方法种类	采用方法		评价指标/%
方法种类	稀疏卷积	语义注意力机制	2PE	3PE	5PE
Prototype 1			27.53	15.26	6.47
Prototype 2	√		24.25	13.02	4.93
Prototype 3	√	√	22.05	11.16	3.94

Fig. 10 Validation dagaset loss curves for three models

References 15

1	LAGA H， JOSPIN L V， BOUSSAID F， et al. A survey on deep learning techniques for stereo-based depth estimation［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2020（Early Access）： 1-1. 10.1109/TPAMI.2020.3032602
2	MAYER N， ILG E， HÄUSSER P， et al. A large dataset to train convolutional networks for disparity， optical flow， and scene flow estimation［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 4040-4048. 10.1109/cvpr.2016.438
3	CHANG J R， CHEN Y S. Pyramid stereo matching network［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 5410-5418. 10.1109/cvpr.2018.00567
4	YUE C， XU J R， LIN S， et al. GCNet： non-local networks meet squeeze-excitation networks and beyond［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop. Piscataway： IEEE， 2019： 1971-1980. 10.1109/iccvw.2019.00246
5	GRAHAM B. Spatially-sparse convolutional neural networks［EB/OL］. （2014-09-22）［2020-11-27］.. 10.5244/c.29.150
6	GRAHAM B. Sparse 3D convolutional neural networks［EB/OL］. （2015-08-25）［2020-11-27］.. 10.5244/c.29.150
7	GRAHAM B， ENGELCKE M， MAATEN L VAN DER. 3D semantic segmentation with submanifold sparse convolutional networks［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 9224-9232. 10.1109/cvpr.2018.00961
8	UHRIG J， SCHNEIDER N， SCHNEIDER L， et al. Sparsity invariant CNNs［C］// Proceedings of the 2017 International Conference on 3D Vision. Piscataway： IEEE， 2017： 11-20. 10.1109/3dv.2017.00012
9	WANG Y， LAI Z H， HUANG G， et al. Anytime stereo image depth estimation on mobile devices［C］// Proceedings of the 2019 International Conference on Robotics and Automation. Piscataway： IEEE， 2019： 5893-5900. 10.1109/icra.2019.8794003
10	FU J， LIU J， JIANG J， et al. Scene segmentation with dual relation-aware attention network［J］. IEEE Transactions on Neural Networks and Learning Systems， 2021， 32（6）： 2547-2560. 10.1109/tnnls.2020.3006524
11	WANG Y， ZHOU Q， LIU J， et al. LEDNet： a lightweight encoder-decoder network for real-time semantic segmentation［C］// Proceedings of the 2019 IEEE International Conference on Image Processing. Piscataway： IEEE， 2019： 1860-1864. 10.1109/icip.2019.8803154
12	GIRSHICK R. Fast R-CNN［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 1440-1448. 10.1109/iccv.2015.169
13	HUANG X Y， WANG P， CHENG X J， et al. The ApolloScape open dataset for autonomous driving and its application［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2020， 42（10）： 2702-2719. 10.1109/tpami.2019.2926463
14	PASZKE A， GROSS S， MASSA F， et al. PyTorch： an imperative style， high-performance deep learning library［EB/OL］. （2019-12-03）［2020-11-27］..
15	ZHANG F H， PRISACARIU V， YANG R G， et al. GA-Net： guided aggregation net for end-to-end stereo matching［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 185-194. 10.1109/cvpr.2019.00027

[1]	Shunyong LI, Shiyi LI, Rui XU, Xingwang ZHAO. Incomplete multi-view clustering algorithm based on self-attention fusion [J]. Journal of Computer Applications, 2024, 44(9): 2696-2703.
[2]	Yexin PAN, Zhe YANG. Optimization model for small object detection based on multi-level feature bidirectional fusion [J]. Journal of Computer Applications, 2024, 44(9): 2871-2877.
[3]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[4]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[5]	Xiyuan WANG, Zhancheng ZHANG, Shaokang XU, Baocheng ZHANG, Xiaoqing LUO, Fuyuan HU. Unsupervised cross-domain transfer network for 3D/2D registration in surgical navigation [J]. Journal of Computer Applications, 2024, 44(9): 2911-2918.
[6]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[7]	Yunchuan HUANG, Yongquan JIANG, Juntao HUANG, Yan YANG. Molecular toxicity prediction based on meta graph isomorphism network [J]. Journal of Computer Applications, 2024, 44(9): 2964-2969.
[8]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.
[9]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[10]	Yuhan LIU, Genlin JI, Hongping ZHANG. Video pedestrian anomaly detection method based on skeleton graph and mixed attention [J]. Journal of Computer Applications, 2024, 44(8): 2551-2557.
[11]	Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594.
[12]	Shangbin MO, Wenjun WANG, Ling DONG, Shengxiang GAO, Zhengtao YU. Single-channel speech enhancement based on multi-channel information aggregation and collaborative decoding [J]. Journal of Computer Applications, 2024, 44(8): 2611-2617.
[13]	Yanjie GU, Yingjun ZHANG, Xiaoqian LIU, Wei ZHOU, Wei SUN. Traffic flow forecasting via spatial-temporal multi-graph fusion [J]. Journal of Computer Applications, 2024, 44(8): 2618-2625.
[14]	Qianhong SHI, Yan YANG, Yongquan JIANG, Xiaocao OUYANG, Wubo FAN, Qiang CHEN, Tao JIANG, Yuan LI. Multi-granularity abrupt change fitting network for air quality prediction [J]. Journal of Computer Applications, 2024, 44(8): 2643-2650.
[15]	Yiqun ZHAO, Zhiyu ZHANG, Xue DONG. Anisotropic travel time computation method based on dense residual connection physical information neural networks [J]. Journal of Computer Applications, 2024, 44(7): 2310-2318.