基于注意力的毫米波-激光雷达融合目标检测

doi:10.11772/j.issn.1001-9081.2020081334

计算机应用 ›› 2021, Vol. 41 ›› Issue (7): 2137-2144.DOI: 10.11772/j.issn.1001-9081.2020081334

所属专题：前沿与综合应用

基于注意力的毫米波-激光雷达融合目标检测

李朝^1,2, 兰海¹, 魏宪¹

1. 中国科学院海西研究院泉州装备制造研究所, 福建泉州 362216;
2. 中北大学电气与控制工程学院, 太原 036005

收稿日期:2020-09-01 修回日期:2020-11-28 出版日期:2021-07-10 发布日期:2020-12-17
通讯作者: 兰海
作者简介:李朝(1994-),男,江西萍乡人,硕士研究生,主要研究方向:三维目标检测、传感器融合;兰海(1988-),男,福建莆田人,助理研究员,硕士,主要研究方向:机器学习与模式识别及其在医疗影像中的应用;魏宪(1986-),男,河南泌阳人,研究员,博士,CCF会员,主要研究方向:机器学习与模式识别及其在无人系统中的应用。
基金资助:
国家自然科学基金青年科学基金资助项目（61806186）；福建省智能物流产业技术研究院建设项目（2018H2001）；机器人技术与系统国家重点实验室（HIT）资助项目（SKLRS-2019-KF-15）；泉州市科技计划项目（2019C112，2019STS08）。

Attention-based object detection with millimeter wave radar-lidar fusion

LI Chao^1,2, LAN Hai¹, WEI Xian¹

1. Quanzhou Institute of Equipment Manufacturing, Haixi Institute, Chinese Academy of Sciences, Quanzhou Fujian 362216, China;
2. School of Electrical and Control Engineering, North University of China, Taiyuan Shanxi 036005, China

Received:2020-09-01 Revised:2020-11-28 Online:2021-07-10 Published:2020-12-17
Supported by:
This work is partially supported by the Youth Program of the National Science Foundation of China (61806186), the Construction Program of Fujian Intelligent Logistics Industry Technology Research Institute (2018H2001), the Programe of State Key Laboratory of Robotics and System (Harbin Institute of Technology) (SKLRS-2019-KF-15), the Quanzhou Science and Technology Program (2019C112, 2019STS08).

摘要/Abstract

摘要： 针对自动驾驶中使用激光雷达进行目标检测时漏检被遮挡目标、远距离目标和复杂天气场景下目标的问题，提出一种基于注意力机制的毫米波-激光雷达特征融合的目标检测方法。首先，将毫米波和激光雷达各自的扫描帧数据分别聚合到它们的标注帧上，并将毫米波和激光雷达的数据点进行空间对齐；其次，对两者进行聚合和空间对齐后的数据分别进行PointPillar点云柱快速编码，转换成伪图像；最后，通过中间卷积层提取两者的传感器特征，并利用注意力机制对两者的特征图进行融合，融合后的特征图通过单阶段检测器得到检测结果。实验结果显示，该融合算法在nuScenes数据集中的平均精度均值（mAP）高于PointPillar基础网络，而且注意力融合的检测方法的性能表现优于利用拼接融合、相乘融合、相加融合的检测方法。可视化结果显示所提方法是有效的，能提高网络对被遮挡目标、远处目标和雨雾天气下目标检测的鲁棒性。

关键词: 自动驾驶, 目标检测, 传感器融合, 注意力机制, 激光雷达, 毫米波雷达

Abstract: To address problems of missing occluded objects, distant objects and objects in extreme weather scenarios when using lidar for object detection in autonomous driving, an attention-based object detection method with millimeter wave radar-lidar feature fusion was proposed. Firstly, the scan frame data of millimeter wave radar and lidar were aggregated into their respective labeled frames, and the points of millimeter wave radar and lidar were spatially aligned, then PointPillar was employed to encode both the millimeter wave radar and lidar data into pseudo images. Finally, the features of both millimeter wave radar and lidar sensors were extracted by the middle convolution layer, and the features maps of them were fused by attention mechanism, and the fused feature map was passed through a single-stage detector to obtain detection results. Experimental results on nuScenes dataset show that compared to the basic PointPillar network, the mean Average Precision (mAP) of the proposed attention fusion algorithm is higher, which performs better than concatenation fusion, multiply fusion and add fusion methods. The visualization results show that the proposed method is effective and can improve the robustness of the network for detecting occluded objects, distant objects and objects surrounded by rain and fog.

Key words: autonomous driving, object detection, sensor fusion, attention mechanism, lidar, millimeter wave radar

中图分类号:

TP391.4

李朝, 兰海, 魏宪. 基于注意力的毫米波-激光雷达融合目标检测[J]. 计算机应用, 2021, 41(7): 2137-2144.

LI Chao, LAN Hai, WEI Xian. Attention-based object detection with millimeter wave radar-lidar fusion[J]. Journal of Computer Applications, 2021, 41(7): 2137-2144.

参考文献

[1] QI C R,SU H,MO K,et al. PointNet:deep learning on point sets for 3D classification and segmentation[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:652-660.
[2] ZHOU Y,TUZEL O. VoxelNet:end-to-end learning for point cloud based 3D object detection[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:4490-4499.
[3] 彭育辉, 郑玮鸿, 张剑锋.基于深度学习的道路障碍物检测方法[J]. 计算机应用,2020,40(8):2428-2433.(PENG Y H, ZHENG W H,ZHANG J F. Deep learning-based on-road obstacle detection method[J]. Journal of Computer Applications,2020,40(8):2428-2433)
[4] KU J,MOZIFIAN M,LEE J,et al. Joint 3D proposal generation and object detection from view aggregation[C]//Proceedings of the 2018 IEEE/RSJ Conference on Intelligent Robots and Systems. Piscataway:IEEE,2018:1-8.
[5] CHEN X,MA H,WAN J,et al. Multi-view 3D object detection network for autonomous driving[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:6526-6534.
[6] 杜佳, 宋春林. 一种改进的毫米波雷达多目标检测算法[J]. 通信技术,2015,48(7):808-813.(DU J,SONG C L. A modified millimeter-wave radar multi-target detection algorithm[J]. Communications Technology,2015,48(7):808-813)
[7] SCHUMANN O,WÖHLER C,HAHN M,et al. Comparison of random forest and long short-term memory network performances in classification tasks using radar[C]//Proceedings of the 2017 Workshop on Sensor Data Fusion:Trends, Solutions, Applications. Piscataway:IEEE,2017:1-6.
[8] SCHUMANN O, HAHN M, DICKMANN J, et al. Semantic segmentation on radar point clouds[C]//Proceedings of the 21st International Conference on Information Fusion. Piscataway:IEEE,2018:2179-2186.
[9] QI C R,YI L,SU H,et al. PointNet++:deep hierarchical feature learning on point sets in a metric space[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook,NY:Curran Associates Inc.,2017:5005-5114.
[10] DANZER A,GRIEBEL T,BACH M,et al. 2D car detection in radar data with PointNets[C]//Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference. Piscataway:IEEE,2019:61-66.
[11] GÖHRING D,WANG M,SCHNÜRMACHER M,et al. Radar/lidar sensor fusion for car-following on highways[C]//Proceedings of the 5th International Conference on Automation,Robotics and Applications. Piscataway:IEEE,2011:407-412.
[12] HAJRI H,RAHAL M C. Real time lidar and radar high-level fusion for obstacle detection and tracking with evaluation on a ground truth[J]. International Journal of Mechanical and Mechatronics Engineering,2018,129(8):821-827.
[13] LEE H,CHAE H,YI K. A geometric model based 2D LiDAR/Radar sensor fusion for tracking surrounding vehicles[J]. IFACPapersOnLine,2019,52(8):130-135.
[14] NABATI R,QI H. RRPN:radar region proposal network for object detection in autonomous vehicles[C]//Proceedings of the 2019 IEEE International Conference on Image Processing. Piscataway:IEEE,2019:3093-3097.
[15] CHADWICK S,MADDERN W,NEWMAN P. Distant vehicle detection using radar and vision[C]//Proceedings of the 2019 International Conference on Robotics and Automation. Piscataway:IEEE,2019:8311-8317.
[16] LIU W,ANGUELOV D,ERHAN D,et al. SSD:single shot MultiBox detector[C]//Proceedings of the 2016 European Conference on Computer Vision,LNCS 9905. Cham:Springer, 2016:21-37.
[17] CHANG S,ZHANG Y,ZHANG F,et al. Spatial attention fusion for obstacle detection using mmWave radar and vision sensor[J]. Sensors,2020,20(4):No. 956.
[18] JOHN V,MITA S. RVNet:deep sensor fusion of monocular camera and radar for image-based obstacle detection in challenging environments[C]//Proceedings of the 2019 Pacific-Rim Symposium on Image and Video Technology, LNCS 11854. Cham:Springer,2019:351-364.
[19] REDMON J,DIVVALA S,GIRSHICK R,et al. You only look once:unified,real-time object detection[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:779-788.
[20] NOBIS F,GEISSLINGER M,WEBER M,et al. A deep learning-based radar and camera sensor fusion architecture for object detection[C]//Proceedings of the 2019 Workshop on Sensor Data Fusion:Trends, Solutions, Applications. Piscataway:IEEE, 2019:1-7.
[21] VASWANI A,SHAZEER N,PARMAR N,et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook,NY:Curran Associates Inc.,2017:6000-6010.
[22] 卢玲, 杨武, 王远伦, 等. 结合注意力机制的长文本分类方法[J]. 计算机应用,2018,38(5):1272-1277.(LU L,YANG W, WANG Y L,et al. Long text classification combined with attention mechanism[J]. Journal of Computer Applications,2018,38(5):1272-1277.)
[23] XIAO T,XU Y,YANG K,et al. The application of two-level attention models in deep convolutional neural network for finegrained image classification[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2015:842-850.
[24] HU J,SHEN L,SUN G. Squeeze-and-excitation networks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:7132-7141.
[25] WANG X,GIRSHICK R,GUPTA A,et al. Non-local neural networks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:7794-7803.
[26] LANG A H,VORA S,CAESAR H,et al. PointPillars:fast encoders for object detection from point clouds[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2019:12689-12697.
[27] HE K,ZHANG X,REN S,et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE 2016:770-778.
[28] CAESAR H,BANKITI V,LANG A H,et al. nuScenes:a multimodal dataset for autonomous driving[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2020:11618-11628.
[29] LIN T Y,GOYAL P,GIRSHICK R,et al. Focal loss for dense object detection[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:2999-3007.
[30] YE Y,CHEN H,ZHANG C,et al. SARPNET:shape attention regional proposal network for liDAR-based 3D object detection[J]. Neurocomputing,2020,379:53-63.
[31] SIMONELLI A,BULÒ S R,PORZI L,et al. Disentangling monocular 3D object detection[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE,2019:1991-1999.

基于注意力的毫米波-激光雷达融合目标检测

Attention-based object detection with millimeter wave radar-lidar fusion

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	代雨柔, 杨庆, 张凤荔, 周帆. 基于自监督学习的社交网络用户轨迹预测模型[J]. 计算机应用, 2021, 41(9): 2545-2551.
[2]	马佳良, 陈斌, 孙晓飞. 基于改进的Faster R-CNN的通用目标检测框架[J]. 计算机应用, 2021, 41(9): 2712-2719.
[3]	刘雅璇, 钟勇. 基于头实体注意力的实体关系联合抽取方法[J]. 计算机应用, 2021, 41(9): 2517-2522.
[4]	李康康, 张静. 基于注意力机制的多层次编码和解码的图像描述模型[J]. 计算机应用, 2021, 41(9): 2504-2509.
[5]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[6]	徐江浪, 李林燕, 万新军, 胡伏原. 结合目标检测的室内场景识别方法[J]. 计算机应用, 2021, 41(9): 2720-2725.
[7]	陈静, 毛莺池, 陈豪, 王龙宝, 王子成. 基于改进单点多盒检测器的大坝缺陷目标检测方法[J]. 计算机应用, 2021, 41(8): 2366-2372.
[8]	党伟超, 李涛, 白尚旺, 高改梅, 刘春霞. 基于自注意力长短期记忆网络的Web软件系统实时剩余寿命预测方法[J]. 计算机应用, 2021, 41(8): 2346-2351.
[9]	樊玮, 李晨炫, 邢艳, 黄睿, 彭洪健. 航空发动机损伤图像的二分类到多分类递进式检测网络[J]. 计算机应用, 2021, 41(8): 2352-2357.
[10]	李扬志, 袁家政, 刘宏哲. 基于时空注意力图卷积网络模型的人体骨架动作识别算法[J]. 计算机应用, 2021, 41(7): 1915-1921.
[11]	张洋, 江铭虎. 基于注意力机制的文本作者识别[J]. 计算机应用, 2021, 41(7): 1897-1901.
[12]	黄少伟, 黄婉琳, 雷闰龙, 毛雪松. 基于脉冲位置幅度调制的距离速度同时测量[J]. 计算机应用, 2021, 41(7): 2145-2149.
[13]	侯笑晗, 金国栋, 谭力宁, 薛远亮. 基于自适应和最优特征的合成孔径雷达舰船检测方法[J]. 计算机应用, 2021, 41(7): 2150-2155.
[14]	肖振远, 王逸涵, 罗建桥, 熊鹰, 李柏林. 基于部分加权损失函数的RefineDet[J]. 计算机应用, 2021, 41(7): 1928-1932.
[15]	武维, 李泽平, 杨华蔚, 林川, 王忠德. 融合内容特征和时序信息的深度注意力视频流行度预测模型[J]. 计算机应用, 2021, 41(7): 1878-1884.