深度学习在单图像三维模型重建的应用

doi:10.11772/j.issn.1001-9081.2020010070

计算机应用 ›› 2020, Vol. 40 ›› Issue (8): 2351-2357.DOI: 10.11772/j.issn.1001-9081.2020010070

• 虚拟现实与多媒体计算 • 上一篇下一篇

深度学习在单图像三维模型重建的应用

张豪^1,2, 张强², 邵思羽², 丁海斌³

1. 空军工程大学研究生院, 西安 710038;
2. 空军工程大学防空反导学院, 西安 710038;
3. 陆军工程大学训练基地, 江苏徐州 221004

收稿日期:2020-01-22 修回日期:2020-03-31 出版日期:2020-08-10 发布日期:2020-03-31
通讯作者: 张豪(1995-),男,福建福州人,硕士研究生,主要研究方向:深度学习、模式识别;421821467@qq.com
作者简介:张强(1973-),男,陕西汉中人,副教授,博士,主要研究方向:深度学习、电力系统及其自动化;邵思羽(1991-),女,山东邹城人,讲师,博士,主要研究方向:基于深度学习、迁移学习的机电设备健康状态检测与故障诊断;丁海斌(1985-),男,山东荣成人,讲师,硕士,主要研究方向:防空反导模拟训练。
基金资助:
江苏省普通高校学术学位研究生科研创新计划项目（KYCX18_0072）。

Application of deep learning to 3D model reconstruction of single image

ZHANG Hao^1,2, ZHANG Qiang², SHAO Siyu², DING Haibin³

1. Graduate School, Air Force Engineering University, Xi'an Shannxi 710038, China;
2. Air and Missile Defense College, Air Force Engineering University, Xi'an Shannxi 710038, China;
3. Training Base, Army Engineering University of PLA, XuzhouJiangsu 221004, China

Received:2020-01-22 Revised:2020-03-31 Online:2020-08-10 Published:2020-03-31
Supported by:
This work is partially supported by the Scientific Research Innovation Plan for Graduate Students of Academic Degree in Colleges and Universities in Jiangsu Province (KYCX18_0072).

摘要/Abstract

摘要： 针对基于单图像重建的三维模型具有高度不确定性问题，提出了一种基于深度图像估计、球面投影映射、三维对抗生成网络相结合的网络模型算法。首先，通过深度估计器得到输入图像的深度图像，这有利于对图像进一步的分析；其次，将得到的深度图像通过球面投影映射转换为三维模型；最后，利用三维对抗生成网络对重建的三维模型的真实性进行判断，建立更逼真的三维模型。理论分析和仿真实验表明，与学习先验知识生成三维模型的算法LVP相比，所提模型在真实三维模型与重建三维模型的交并比（IoU）上提高了20.1%，倒角距离（CD）缩小了13.2%。实验结果表明，所提模型在单视图三维模型重建中具有良好的泛化能力。

关键词: 深度图像, 深度估计, 三维重建, 对抗生成网络, 球面投影

Abstract: To solve the problem that the reconstructed 3D model of a single image has high uncertainty, a network model based on depth image estimation, spherical projection mapping and 3D generative adversarial network was proposed. Firstly, the depth image of the input image was obtained by the depth estimator, which was helpful for the further analysis of the image. Secondly, the obtained depth image was converted into a 3D model by spherical projection mapping. Finally, 3D generative adversarial network was utilized to judge the authenticity of the reconstructed 3D model, so as to obtain 3D model closer to reality. In the comparison experiments with LVP algorithm which learning view priors for 3D reconstruction, the proposed model has the Intersection-over-Union (IoU) increased by 20.1% and the Charmfer Distance (CD) decreased by 13.2%. Theoretical analysis and simulation results show that the proposed model has good generalization ability in the 3D model reconstruction of a single image.

Key words: depth image, depth estimation, 3D reconstruction, Generative Adversarial Network (GAN), spherical projection

中图分类号:

TP391

张豪, 张强, 邵思羽, 丁海斌. 深度学习在单图像三维模型重建的应用[J]. 计算机应用, 2020, 40(8): 2351-2357.

ZHANG Hao, ZHANG Qiang, SHAO Siyu, DING Haibin. Application of deep learning to 3D model reconstruction of single image[J]. Journal of Computer Applications, 2020, 40(8): 2351-2357.

参考文献

[1] 周继来,周明全,耿国华,等. 基于曲度特征的三维模型检索算法[J]. 计算机应用, 2016, 36(7):1914-1917, 1922. (ZHOU J L, ZHOU M Q, GENG G H, et al. 3D model retrieval algorithm based on curvedness feature[J]. Journal of Computer Applications, 2016, 36(7):1914-1917, 1922.)
[2] 朱俊鹏,赵洪利,杨海涛. 基于卷积神经网络的视差图生成技术[J]. 计算机应用, 2018, 38(1):255-259, 289. (ZHU J P, ZHAO H L, YANG H T. Disparity map generation technology based on convolutional neural network[J]. Journal of Computer Applications, 2018, 38(1):255-259, 289.)
[3] CHANG A X, FUNKHOUSER T, GUIBAS L, et al. ShapeNet:an information-rich 3D model repository[EB/OL].[2019-11-21].https://arxiv.org/pdf/1512.03012.pdf.
[4] DENG X, SONG P, RODRIGUES M R D. RADAR:robust algorithm for depth image super resolution based on FRI theory and multimodal dictionary learning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019(Early Access):1-1.
[5] DREWS P L J, NASCIMENTO E R, BOTELHO S S C, et al. Underwater depth estimation and image restoration based on single images[J]. IEEE Computer Graphics and Applications, 2016, 36(2):24-35.
[6] 陈加,张玉麒,宋鹏,等. 深度学习在基于单幅图像的物体三维重建中的应用[J]. 自动化学报, 2019, 45(4):657-668. (CHEN J, ZHANG Y Q, SONG P, et al. Application of deep learning in 3D object reconstruction based on single image[J]. Acta Automatica Sinica, 2019, 45(4):657-668.)
[7] TULSIANI S, ZHOU T, EFROS A, et al. Multi-view supervision for single-view reconstruction via differentiable ray consistency[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019(Early Access):1-1.
[8] GROUEIX T, FISHER M, KIM V G, et al. AltasNet:a Papier-Mâché approach to learning 3D surface generation[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:216-224.
[9] HENDERSON P, FERRARI V. Learning single-image 3D reconstruction by generative modelling of shape, pose and shading[J]. International Journal of Computer Vision, 2020, 128(4):835-854.
[10] KATO H, HARADA T. Learning view priors for single-view 3D reconstruction[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2019:9770-9779.
[11] WU J, WANG Y, XUE T, et al. MarrNet:3D shape reconstruction via 2.5D sketches[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook:Curran Associates Inc., 2017:540-550.
[12] 李伟,张旭东. 基于卷积神经网络的深度图像超分辨率重建方法[J].电子测量与仪器学报, 2017, 31(12):1918-1928. (LI W, ZHANG X D. Depth image super-resolution reconstruction based on convolutional neural network[J]. Journal of Electronic Measurement and Instrumentation, 2017, 31(12):1918-1928)
[13] CHEN Y, SHI F, CHRISTODOULOU A G, et al. Efficient and accurate MRI super-resolution using a generative adversarial network and 3D multi-level densely connected network[C]//Proceedings of the 2018 International Conference on Medical Image Computing and Computer-Assisted Intervention, LNCS 11070. Cham:Springer, 2018:91-99.
[14] ZHANG K, SUN M, HAN T X, et al. Residual networks of residual networks:multilevel residual networks[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 28(6):1303-1314.
[15] RONNEBERGER O, FISCHER P, BROX T. U-Net:convolutional networks for biomedical image segmentation[C]//Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention, LNCS 9351. Cham:Springer, 2015:234-241.
[16] XIA Y, XIAO J, WANG Y. A fast registration algorithm of rock point cloud based on spherical projection and feature extraction[J]. Frontiers of Computer Science, 2019, 13(1):170-182.
[17] RAN L, ZHANG Y, ZHANG Q, et al. Convolutional neural network-based robot navigation using uncalibrated spherical images[J]. Sensors, 2017, 17(6):No.1341.
[18] GORBACHEV V A, OSOKIN I V. Detection and removal of foreground objects in spherical images for the synthesis of photorealistic intermediate images[J]. Pattern Recognition and Image Analysis, 2019, 29(3):471-485.
[19] BASHMAL L, BAZI Y, ALHICHRI H, et al. Siamese-GAN:learning invariant representations for aerial vehicle image categorization[J]. Remote Sensing, 2018, 10(3):No.351.
[20] CUI Z, ZHANG M, CAO Z, et al. Image data augmentation for SAR sensor via generative adversarial nets[J]. IEEE Access, 2019, 7:42255-42268.
[21] 曹仰杰,贾丽丽,陈永霞,等. 生成式对抗网络及其计算机视觉应用研究综述[J]. 中国图象图形学报, 2018, 23(10):1433-1449. (CAO Y J, JIA L L, CHEN Y X, et al. Review of computer vision based on generative adversarial networks[J]. Journal of Image and Graphics, 2018, 23(10):1433-1449.)

深度学习在单图像三维模型重建的应用

Application of deep learning to 3D model reconstruction of single image

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	温静, 李智宏. 基于带squeeze-and-excitation模块的ResNeXt的单目图像深度估计方法[J]. 计算机应用, 2021, 41(1): 215-219.
[2]	喻露, 胡剑锋, 姚磊岳. 基于人体骨架的非标准深蹲姿势检测方法[J]. 计算机应用, 2019, 39(5): 1448-1452.
[3]	莫赞, 盖彦蓉, 樊冠龙. 基于GAN-AdaBoost-DT不平衡分类算法的信用卡欺诈分类[J]. 计算机应用, 2019, 39(2): 618-622.
[4]	许德智, 孙季丰, 罗莎莎. 基于权重量化与信息压缩的车载图像超分辨率重建[J]. 计算机应用, 2019, 39(12): 3644-3649.
[5]	郭艺辉, 黄承慧, 钟雪灵, 陆寄远. 方向感知的网格模型特征识别[J]. 计算机应用, 2019, 39(12): 3673-3677.
[6]	廖斌, 李浩文. 基于多孔卷积神经网络的图像深度估计模型[J]. 计算机应用, 2019, 39(1): 267-274.
[7]	刘思奇, 张来峰, 范立成, 盛小明. 基于医学图像序列面绘制的骨组织快速成型方法[J]. 计算机应用, 2017, 37(5): 1456-1459.
[8]	李肖, 葛宝臻, 罗其俊, 李云鹏, 田庆国. 自由双目立体视觉摄像机动态外参数的获取[J]. 计算机应用, 2017, 37(10): 2888-2894.
[9]	徐海宁, 陈恩庆, 梁成武. 三维动作识别时空特征提取方法[J]. 计算机应用, 2016, 36(2): 568-573.
[10]	陆中秋, 侯振杰, 陈宸, 梁久祯. 基于深度图像与骨骼数据的行为识别[J]. 计算机应用, 2016, 36(11): 2979-2984.
[11]	尚常军, 丁瑞. 基于曲率局部二值模式的深度图像手势特征提取[J]. 计算机应用, 2016, 36(10): 2885-2889.
[12]	杨全彭进业. 基于深度图像信息的手语识别算法[J]. 计算机应用, 2013, 33(10): 2882-2885.
[13]	吴茜贾婧曹瑞芬裴曦吴爱东吴宜灿 FDS团队. 三维医学图像配准在图像引导放疗中的应用[J]. 计算机应用, 2013, 33(09): 2675-2678.
[14]	陈维涛陈东帆韩兴乾周晨高祥. 声带麻痹植入手术的有限元仿真模拟[J]. 计算机应用, 2013, 33(03): 896-900.
[15]	李怀泽沈会良程岳. 基于旋转多视角深度配准的三维重建方法[J]. 计算机应用, 2012, 32(12): 3365-3368.