Infrared monocular ranging algorithm based on multiscale feature fusion

doi:10.11772/j.issn.1001-9081.2021040912

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (3): 804-809.DOI: 10.11772/j.issn.1001-9081.2021040912

Special Issue: 人工智能； 2021年中国计算机学会人工智能会议(CCFAI 2021)

• 2021 CCF Conference on Artificial Intelligence (CCFAI 2021) • Previous Articles Next Articles

Infrared monocular ranging algorithm based on multiscale feature fusion

Bin LIU¹^,², Gangqing LI¹^,²(), Chengquan AN¹, Shuigen WANG², Jiansheng WANG²

^1.College of Information and Communication Engineering，Harbin Engineering University，Harbin Heilongjiang 150001，China
^2.IRay Technology Company Limited，Yantai Shandong 264000，China

Received:2021-06-02 Revised:2021-07-14 Accepted:2021-07-15 Online:2022-04-09 Published:2022-03-10
Contact: Gangqing LI
About author:LIU Bin， born in 1998， M. S. candidate. His research interests include artificial intelligence， computer vision.
AN Chengquan， born in 1974， Ph. D.， associate professor. His research interests include wireless communication， spread spectrum communication， signal processing.
WANG Shuigen， born in 1990， Ph. D. His research interests include artificial intelligence， deep learning， feature learning， computer vision.
WANG Jiansheng， born in 1972， M. S. His research interests include image processing， deep learning， computer vision.

基于多尺度特征融合的红外单目测距算法

刘斌¹^,², 李港庆¹^,²(), 安澄全¹, 王水根², 王建生²

^1.哈尔滨工程大学信息与通信工程学院，哈尔滨 150001
^2.艾睿光电科技有限公司，山东烟台 264000

通讯作者: 李港庆
作者简介:刘斌（1998—），男，湖北荆州人，硕士研究生，CCF会员，主要研究方向：人工智能、计算机视觉
安澄全（1974—），男，黑龙江双城人，副教授，博士，主要研究方向：无线通信、扩频通信、信号处理
王水根（1990—），男，江西瑞金人，博士，主要研究方向：人工智能、深度学习、特征学习、计算机视觉
王建生（1972—），男，重庆人，硕士，主要研究方向：图像处理、深度学习、计算机视觉。

Abstract

Abstract:

Due to the introduction of MonoDepth2， unsupervised monocular ranging has made great progress in the field of visible light. However， visible light is not applicable in some scenes， such as at night and in some low-visibility environments. Infrared thermal imaging can obtain clear target images at night and under low-visibility conditions， so it is necessary to estimate the depth of infrared image. However， due to the different characteristics of visible and infrared images， it is unreasonable to migrate existing monocular depth estimation algorithms directly to infrared images. An infrared monocular ranging algorithm based on multiscale feature fusion after improving the MonoDepth2 algorithm can solve this problem. A new loss function， edge loss function， was designed for the low texture characteristic of infrared image to reduce pixel mismatch during image reprojection. The previous unsupervised monocular ranging simply upsamples the four-scale depth maps to the original image resolution to calculate projection errors， ignoring the correlation between scales and the contribution differences between different scales. A weighted Bi-directional Feature Pyramid Network （BiFPN） was applied to feature fusion of multiscale depth maps so that the blurring of depth map edge was solved. In addition， Residual Network （ResNet） structure was replaced by Cross Stage Partial Network （CSPNet） to reduce network complexity and increase operation speed. The experimental results show that edge loss is more suitable for infrared image ranging， resulting in better depth map quality. After adding BiFPN structure， the edge of depth image is clearer. After replacing ResNet with CSPNet， the inference speed is improved by about 20 percentage points. The proposed algorithm can accurately estimate the depth of the infrared image， solving the problem of depth estimation in night low-light scenes and some low-visibility scenes， and the application of this algorithm can also reduce the cost of assisted driving to a certain extent.

Key words: unsupervised, monocular ranging, infrared image, Bi-directional Feature Pyramid Network (BiFPN), Cross Stage Partial Network (CSPNet)

摘要：

由于MonoDepth2的提出，无监督单目测距在可见光领域取得了重大发展；然而在某些场景例如夜间以及一些低能见度的环境，可见光并不适用，而红外热成像可以在夜间和低能见度条件下获得清晰的目标图像，因此对于红外图像的深度估计显得尤为必要。由于可见光和红外图像的特性不同，直接将现有可见光单目深度估计算法迁移到红外图像是不合理的。针对该问题，对MonoDepth2算法进行改进，提出了基于多尺度特征融合的红外单目测距算法。针对红外图像低纹理的特性设计了一项新的损失函数边缘损失函数，旨在降低图像重投影时的像素误匹配。不同于以往的无监督单目测距单纯地将四个尺度的深度图统一上采样到原图像分辨率计算投影误差而忽略了尺度之间的关联性以及不同尺度之间的贡献差异，将加权的双向特征金字塔网络（BiFPN）应用于多尺度深度图的特征融合，解决了深度图边缘模糊问题。另外用跨阶段部分网络（CSPNet）替换残差网络（ResNet）结构，以降低网络复杂度并提高运算速度。实验结果表明，边缘损失更适合红外图像测距，使得深度图质量更高；在加入BiFPN结构之后，深度图像的边缘更加清晰；将ResNet替换为CSPNet之后，推理速度提高了大约20个百分点。该算法能够准确估计出红外图像的深度，解决夜间低光照场景以及一些低能见度场景下的深度估计难题；该算法的应用也可以在一定程度上降低汽车辅助驾驶的成本。

关键词: 无监督, 单目测距, 红外图像, 双向特征金字塔网络, 跨阶段部分网络

CLC Number:

TP183

Bin LIU, Gangqing LI, Chengquan AN, Shuigen WANG, Jiansheng WANG. Infrared monocular ranging algorithm based on multiscale feature fusion[J]. Journal of Computer Applications, 2022, 42(3): 804-809.

刘斌, 李港庆, 安澄全, 王水根, 王建生. 基于多尺度特征融合的红外单目测距算法[J]. 《计算机应用》唯一官方网站, 2022, 42(3): 804-809.

Figures/Tables 13

Fig. 1 Network structure of the proposed model

Fig. 2 Image edge extraction

Fig. 3 Basic module structure of ResNet

Fig. 4 Basic module structure of improved CSPNet

Fig. 5 Structure of BiFPN

Fig. 6 Structure of PANet

Fig. 7 Inference result

Fig. 8 Comparison of inference effects between proposed model and MonoDepth2 model

Fig. 9 Comparison of model inference effects before and after adding BIFPN structure

Fig. 10 Comparison of model inference effects before and after adding edge loss function

Tab. 1 Comparison of inference speed between two network structures

主干网络	Size	FPS
ResNet	640×192	35.0
CSPNet	640×192	42.0
ResNet	800×608	10.0
CSPNet	800×608	12.0
ResNet	1 024×640	3.0
CSPNet	1 024×640	3.7

Tab. 2 Comparison of inference precision between two network structures

主干网络	abs rel	sq rel	RMSE	$δ < 1.25$	$δ < 1.252$	$δ < 1.253$
ResNet	0.115	0.882	4.701	0.879	0.961	0.982
CSPNet	0.118	0.894	4.752	0.866	0.947	0.975

Tab. 2 Comparison of inference precision between two network structures

主干网络	abs rel	sq rel	RMSE	$δ < 1.25$	$δ < 1.252$	$δ < 1.253$
ResNet	0.115	0.882	4.701	0.879	0.961	0.982
CSPNet	0.118	0.894	4.752	0.866	0.947	0.975

Fig. 11 Comparison of training weight inference effects between two datasets

References 17

1	ZHAO C， SUN Q， ZHANG C， et al. Monocular depth estimation based on deep learning： an overview［J］. Science China （Technological Sciences）， 2020， 63： 2243-2249. 10.1007/s11431-020-1582-8
2	黄军，王聪，刘越，等.单目深度估计技术进展综述［J］.中国图象图形学报，2019，24（12）：2081-2097. 10.11834/jig.190455
	HUANG J， WANG C， LIU Y，et al. Overview of monocular depth estimation technology ［J］.Chinese Journal of Image and Graphics， 2019， 24（12）： 2081-2097. 10.11834/jig.190455
3	GODARD C， AODHA O MAC， FIRMAN M， et al. Digging into self-supervised monocular depth estimation ［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE，2019： 3827-3837. 10.1109/iccv.2019.00393
4	李旭，丁萌，魏东辉，等. VDAS中基于单目红外图像的深度估计方法［J］.系统工程与电子技术，2021，43（5）： 1210-1217. 10.12305/j.issn.1001-506X.2021.05.07
	LI X， DING M， WEI D H， et al. Depth estimation method based on monocular infrared image in VDAS ［J］. System Engineering and Electronic Technology， 2021，43（5）： 1210-1217. 10.12305/j.issn.1001-506X.2021.05.07
5	陈钱.红外图像处理技术现状及发展趋势［J］.红外技术，2013， 35（6）： 311-318. 10.1016/j.infrared.2013.07.006
	CHEN Q. Status and development trend of infrared image processing technology ［J］. Infrared Technology， 2013， 35（6）： 311-318. 10.1016/j.infrared.2013.07.006
6	GODARD C， AODHA O MAC， BROSTOW G J， et al. Unsupervised monocular depth estimation with left-right consistency ［C］// Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 6602-6611. 10.1109/cvpr.2017.699
7	周飞燕，金林鹏，董军. 卷积神经网络研究综述［J］.计算机学报， 2017， 40（6）： 1229-1251. 10.11897/SP.J.1016.2017.01229
	ZHOU F Y， JIN L P， DONG J. Review of convolutional neural network research ［J］. Chinese Journal of Computers， 2017， 40（6）： 1229-1251. 10.11897/SP.J.1016.2017.01229
8	ZHOU T H， BROWN M， SNAVELY N， et al. Unsupervised learning of depth and ego-motion from video ［C］// Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 6612-6619. 10.1109/cvpr.2017.700
9	李先真，张大波. 基于降采样聚类的双目图像测距算法研究［J］. 辽宁大学学报（自然科学版）， 2020， 47（3）： 277-283. 10.1109/icuems50872.2020.00147
	LI X Z， ZHANG D B. Research on binocular image ranging algorithm based on down sampling clustering ［J］. Journal of Liaoning University （Natural Science Edition）， 2020， 47（3）： 277-283. 10.1109/icuems50872.2020.00147
10	杨词银，张建萍，郭立红，等. 基于大气修正的目标红外辐射测量［J］.长春理工大学学报（自然科学版）， 2010， 33（4）： 1-3. 10.3969/j.issn.1672-9870.2010.04.001
	YANG C Y， ZHANG J P， GUO L H， et al. Target infrared radiation measurement based on atmospheric correction ［J］. Journal of Changchun University of Technology （Natural Science Edition）， 2010， 33（4）： 1-3. 10.3969/j.issn.1672-9870.2010.04.001
11	范延军. 基于机器视觉的先进辅助驾驶系统关键技术研究［D］. 南京：东南大学， 2016： 3-11.
	FAN Y J. Research on key technologies of advanced auxiliary driving system based on machine vision ［D］. Nanjing： Southeast University， 2016： 3-11.
12	陈雪松，栾明君. 基于对数调频生物仿生波信号的军事雷达测距优化方法分析［J］. 电子测试， 2013（22）： 45-46.
	CHEN X S， LUAN M J. Analysis of military radar ranging optimization method based on log FM biomimetic wave signal ［J］. Electronic Test， 2013（22）： 45-46.
13	TAN M， PANG R， LE Q V. EfficientDet： scalable and efficient object detection ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition， Piscataway： IEEE， 2020： 10778-10787. 10.1109/cvpr42600.2020.01079
14	WANG C-Y， MARK LIAO H-Y， WU Y-H， et al. CSPNet： a new backbone that can enhance learning capability of CNN ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2020： 1571-1580. 10.1109/cvprw50498.2020.00203
15	WANG Z， BOVIK A C， SHEIKH H R， et al. Image quality assessment： from error visibility to structural similarity ［J］. IEEE Transactions on Image Processing， 2004， 13（4）： 600-612. 10.1016/j.jvcir.2019.102655
16	LIU S， QI L， QIN H， et al. Path aggregation network for instance segmentation ［C］// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8759-8768. 10.1109/cvpr.2018.00913
17	EIGEN D， FERGUS R. Predicting depth， surface normals and semantic labels with a common multi-scale convolutional architecture ［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 2650-2658. 10.1109/iccv.2015.304

[1]	Yuxin HUANG, Jialong XU, Zhengtao YU, Shukai HOU, Jiaqi ZHOU. Unsupervised text sentiment transfer method based on generation prompt [J]. Journal of Computer Applications, 2024, 44(9): 2667-2673.
[2]	Jieru JIA, Jianchao YANG, Shuorui ZHANG, Tao YAN, Bin CHEN. Unsupervised person re-identification based on self-distilled vision Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2893-2902.
[3]	Xiawuji, Heming HUANG, Gengzangcuomao, Yutao FAN. Survey of extractive text summarization based on unsupervised learning and supervised learning [J]. Journal of Computer Applications, 2024, 44(4): 1035-1048.
[4]	Junjie ZHU, Li YU, Shengwen LI, Changzheng ZHOU. Technology term recognition with comprehensive constituency parsing [J]. Journal of Computer Applications, 2024, 44(4): 1072-1079.
[5]	Rui JIANG, Wei LIU, Cheng CHEN, Tao LU. Asymmetric unsupervised end-to-end image deraining network [J]. Journal of Computer Applications, 2024, 44(3): 922-930.
[6]	Jingxin LIU, Wenjing HUANG, Liangsheng XU, Chong HUANG, Jiansheng WU. Unsupervised feature selection model with dictionary learning and sample correlation preservation [J]. Journal of Computer Applications, 2024, 44(12): 3766-3775.
[7]	Yongjiang LIU, Bin CHEN. Pixel-level unsupervised industrial anomaly detection based on multi-scale memory bank [J]. Journal of Computer Applications, 2024, 44(11): 3587-3594.
[8]	Nengbing HU, Biao CAI, Xu LI, Danhua CAO. Graph classification method based on graph pooling contrast learning [J]. Journal of Computer Applications, 2024, 44(11): 3327-3334.
[9]	Pei ZHAO, Yan QIAO, Rongyao HU, Xinyu YUAN, Minyue LI, Benchu ZHANG. Multivariate time series anomaly detection based on multi-domain feature extraction [J]. Journal of Computer Applications, 2024, 44(11): 3419-3426.
[10]	Cheng WANG, Yang WANG, Yingjiao RONG. YOLOv7-MSBP target location algorithm for character recognition of power distribution cabinet [J]. Journal of Computer Applications, 2024, 44(10): 3191-3199.
[11]	Wei TONG, Liyang HE, Rui LI, Wei HUANG, Zhenya HUANG, Qi LIU. Efficient similar exercise retrieval model based on unsupervised semantic hashing [J]. Journal of Computer Applications, 2024, 44(1): 206-216.
[12]	Tian HE, Zongxin SHEN, Qianqian HUANG, Yanyong HUANG. Adaptive learning-based multi-view unsupervised feature selection method [J]. Journal of Computer Applications, 2023, 43(9): 2657-2664.
[13]	Menglin HUANG, Lei DUAN, Yuanhao ZHANG, Peiyan WANG, Renhao LI. Prompt learning based unsupervised relation extraction model [J]. Journal of Computer Applications, 2023, 43(7): 2010-2016.
[14]	Zhe XU, Zhihong WANG, Cunyu SHAN, Yaru SUN, Ying YANG. Unsupervised face forgery video detection based on reconstruction error [J]. Journal of Computer Applications, 2023, 43(5): 1571-1577.
[15]	Mengting GE, Minghua WAN. Feature extraction model based on neighbor supervised locally invariant robust principal component analysis [J]. Journal of Computer Applications, 2023, 43(4): 1013-1020.

Infrared monocular ranging algorithm based on multiscale feature fusion

基于多尺度特征融合的红外单目测距算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 17

Related Articles 15

Recommended Articles

Metrics