Multi-view stereo method based on quadtree prior assistance

doi:10.11772/j.issn.1001-9081.2023111661

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (11): 3556-3564.DOI: 10.11772/j.issn.1001-9081.2023111661

• Multimedia computing and computer simulation • Previous Articles Next Articles

Multi-view stereo method based on quadtree prior assistance

Lihua HU¹(), Xiaoping LI¹, Jianhua HU², Sulan ZHANG¹

^1.College of Computer Science and Technology，Taiyuan University of Science and Technology，Taiyuan Shanxi 030024，China
^2.Institute of Automation，Chinese Academy of Sciences，Beijing 100190，China

Received:2023-12-04 Revised:2024-05-25 Accepted:2024-05-30 Online:2024-07-25 Published:2024-11-10
Contact: Lihua HU
About author:LI Xiaoping， born in 1998， M. S. candidate. Her research interests include computer vision.
HU Jianhua， born in 1987， Ph. D.， associate research fellow. His research interests include intelligent robot， 3D vision.
ZHANG Sulan， born in 1971， Ph. D.， professor. Her research interests include computer vision， machine learning.
Supported by:
National Natural Science Foundation of China(62273248);Natural Science Foundation of Shanxi Province(202103021224285);Science and Technology Service Network Initiative Program of Chinese Academy of Sciences(STS-HP-202202)

基于四叉树先验辅助的多视图立体方法

胡立华¹(), 李小平¹, 胡建华², 张素兰¹

^1.太原科技大学计算机科学与技术学院，太原 030024
^2.中国科学院自动化研究所，北京 100190

通讯作者: 胡立华
作者简介:李小平（1998—），女，山东青岛人，硕士研究生，主要研究方向：计算机视觉
胡建华（1987—），男，山西忻州人，副研究员，博士，主要研究方向：智能机器人、3D视觉
张素兰（1971—），女，山西长治人，教授，博士，CCF会员，主要研究方向：计算机视觉、机器学习。
基金资助:
国家自然科学基金资助项目(62273248);山西省自然科学基金资助项目(202103021224285);中国科学院科技服务网络计划项目(STS?HP?202202)

Abstract

Abstract:

PatchMatch-based Multi-View Stereo （MVS） method can estimate the depth of a scene based on multiple input images and is currently applied in large-scale 3D scene reconstruction. However， the existing methods have lower accuracy and completeness in depth estimation in low-texture regions due to unstable feature matching and unreliable reliance on photometric consistency alone. To address the above problems， an MVS method based on quadtree prior assistance was proposed. Firstly， the image pixel values were used to obtain local textures. Secondly， a coarse depth map was obtained by Adaptive Checkerboard sampling and Multi-Hypothesis joint view selection （ACMH）， which combined the structural information in the low-texture region to generate a priori plane hypothesis by using quadtree segmentation. Thirdly， by integrating the above information， a new multi-view matching cost function was designed to guide the low-texture regions for obtaining the best depth assumption， thereby improving the accuracy of stereo matching. Finally， comparison experiments were conducted with many existing traditional MVS methods on ETH3D， Tanks and Temples， and Chinese Academy of Sciences' ancient architecture datasets. The results demonstrate that the proposed method performs better， especially in ETH3D test dataset with error threshold of 2 cm， its F1 score and completeness are improved by 1.29 and 2.38 percentage points， respectively， compared with the current state-of-the-art multi-scale geometric consistency guided and planar prior assisted multi-view stereo method （ACMMP）.

Key words: Multi-View Stereo (MVS), depth estimation, matching cost, low-texture region, quadtree prior

摘要：

基于PatchMatch的多视图立体（MVS）方法依据输入多幅图像估计场景的深度，目前已应用于大规模场景三维重建。然而，由于特征匹配不稳定、仅依赖光度一致性不可靠等原因，现有方法在弱纹理区域的深度估计准确性和完整性较低。针对上述问题，提出一种基于四叉树先验辅助的MVS方法。首先，利用图像像素值获得局部纹理；其次，基于自适应棋盘网格采样的块匹配多视图立体视觉方法（ACMH）获得粗略的深度图，结合弱纹理区域中的结构信息，采用四叉树分割生成先验平面假设；再次，融合上述信息，设计一种新的多视图匹配代价函数，引导弱纹理区域得到最优深度假设，进而提高立体匹配的准确性；最后，在ETH3D、Tanks and Temples和中国科学院古建筑数据集上与多种现有的传统MVS方法进行对比实验。结果表明所提方法性能更优，特别是在ETH3D测试数据集中，当误差阈值为2 cm时，相较于当前先进的多尺度平面先验辅助方法（ACMMP），它的F1分数和完整性分别提高了1.29和2.38个百分点。

关键词: 多视图立体, 深度估计, 匹配代价, 弱纹理区域, 四叉树先验

CLC Number:

TP391.41

Lihua HU, Xiaoping LI, Jianhua HU, Sulan ZHANG. Multi-view stereo method based on quadtree prior assistance[J]. Journal of Computer Applications, 2024, 44(11): 3556-3564.

胡立华, 李小平, 胡建华, 张素兰. 基于四叉树先验辅助的多视图立体方法[J]. 《计算机应用》唯一官方网站, 2024, 44(11): 3556-3564.

Figures/Tables 16

References 39

1	XIAO X， XU C， WANG J， et al. Enhanced 3-D modeling for landmark image classification［J］. IEEE Transactions on Multimedia， 2012， 14（4）： 1246-1258.
2	MAIER-HEIN L， MOUNTNEY P， BARTOLI A， et al. Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery［J］. Medical Image Analysis， 2013， 17（8）： 974-996.
3	NEWCOMBE R A， LOVEGROVE S J， DAVISON A J. DTAM： dense tracking and mapping in real-time［C］// Proceedings of the 2011 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2011： 2320-2327.
4	WANG W， GAO W. Efficient multi-plane extraction from massive 3D points for modeling large-scale urban scenes［J］. The Visual Computer， 2019， 35（5）： 625-638.
5	KOSTRIKOV I， HORBERT E， LEIBE B. Probabilistic labeling cost for high-accuracy multi-view reconstruction［C］// Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2014： 1534-1541.
6	ESTEBAN C H， SCHMITT F. Silhouette and stereo fusion for 3D object modeling［C］// Proceedings of the 4th International Conference on 3-D Digital Imaging and Modeling. Piscataway： IEEE， 2003： 46-53.
7	FURUKAWA Y， PONCE J. Accurate， dense， and robust multiview stereopsis［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2010， 32（8）： 1362-1376.
8	WU P， LIU Y， YE M， et al. Fast and adaptive 3D reconstruction with extensively high completeness［J］. IEEE Transactions on Multimedia， 2017， 19（2）： 266-278.
9	SHEN S. Accurate multiple view 3D reconstruction using patch-based stereo for large-scale scenes［J］. IEEE Transactions on Image Processing， 2013， 22（5）： 1901-1914.
10	BLEYER M， RHEMANN C， ROTHER C. PatchMatch Stereo — stereo matching with slanted support windows［C］// Proceedings of the 2011 British Machine Vision Conference. Durham： BMVA Press， 2011： No.14.
11	XU Q， TAO W. Multi-scale geometric consistency guided multi-view stereo［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 5478-5487.
12	FISCHLER M A， BOLLES R C. Random sample consensus： a paradigm for model fitting with applications to image analysis and automated cartography［J］. Communications of the ACM， 1981， 24（6）： 381-395.
13	BARNES C， SHECHTMAN E， FINKELSTEIN A， et al. PatchMatch： a randomized correspondence algorithm for structural image editing［J］. ACM Transactions on Graphics， 2009， 28（3）： No.24.
14	ZHENG E， DUNN E， JOJIC V， et al. PatchMatch based joint view selection and depthmap estimation［C］// Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2014： 1510-1517.
15	NEAL R M， HINTON G E. A view of the EM algorithm that justifies incremental， sparse， and other variants［M］// JORDAN M I. Learning in Graphical Models， ASID 89. Dordrecht： Springer， 1998： 355-368.
16	GALLIANI S， LASINGER K， SCHINDLER K. Massively parallel multiview stereopsis by surface normal diffusion［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 873-881.
17	SCHÖNBERGER J L， ZHENG E， FRAHM J M， et al. Pixelwise view selection for unstructured multi-view stereo［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9907. Cham： Springer， 2016： 501-518.
18	WOODFORD O， TORR P， REID I， et al. Global stereo reconstruction under second-order smoothness priors［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2009， 31（12）： 2115-2128.
19	GALLUP D， FRAHM J M， POLLEFEYS M. Piecewise planar and non-planar stereo for urban scene reconstruction［C］// Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2010： 1418-1425.
20	XU Z， LIU Y， SHI X， et al. MARMVS： matching ambiguity reduced multiple view stereo for efficient large scale scene reconstruction［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 5980-5989.
21	SUN S， ZHENG Y， SHI X， et al. PHI-MVS： plane hypothesis inference multi-view stereo for large-scale scene reconstruction［EB/OL］. ［2024-05-22］..
22	STATHOPOULOU E K， BATTISTI R， CERNEA D， et al. Semantically derived geometric constraints for MVS reconstruction of textureless areas［J］. Remote Sensing， 2021， 13（6）： No.1053.
23	ROMANONI A， MATTEUCCI M. TAPA-MVS： textureless-aware PAtchMatch multi-view stereo［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 10412-10421.
24	XU Q， TAO W. Planar prior assisted PatchMatch multi-view stereo［C］// Proceedings of the 34th AAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 12516-12523.
25	MOULON P， MONASSE P， PERROT R， et al. OpenMVG： open multiple view geometry［C］// Proceedings of the 2016 International Conference on Reproducible Research in Pattern Recognition， LNCS 10214. Cham： Springer， 2017： 60-74.
26	中国科学院自动化研究所模式识别国家重点实验室. 中国古建筑三维重建数据集［DS/OL］. ［2023-08-23］..
	National Laboratory of Pattern Recognition of Institute of Automation of Chinese Academy of Sciences. Chinese ancient architecture 3D reconstruction dataset［DS/OL］. ［2023-08-23］..
27	SCHÖPS T， SCHÖNBERGER J L， GALLIANI S， et al. A multi-view stereo benchmark with high-resolution images and multi-camera videos［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2538-2547.
28	KNAPITSCH A， PARK J， ZHOU Q Y， et al. Tanks and temples： benchmarking large-scale scene reconstruction［J］. ACM Transactions on Graphics， 2017， 36（4）： No.78.
29	STRECHA C， VON HANSEN W， VAN GOOL L， et al. On benchmarking camera calibration and multi-view stereo for high resolution imagery［C］// Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2008： 1-8.
30	SEITZ S M， CURLESS B， DIEBEL J， et al. A comparison and evaluation of multi-view stereo reconstruction algorithms［C］// Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2006： 519-528.
31	FU C， HUANG N， HUANG Z， et al. Confidence-guided planar-recovering multiview stereo for weakly textured plane of high-resolution image scenes［J］. Remote Sensing， 2023， 15（9）： No.2474.
32	XU Q， KONG W， TAO W， et al. Multi-scale geometric consistency guided and planar prior assisted multi-view stereo［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2023， 45（4）： 4945-4963.
33	STATHOPOULOU E K， BATTISTI R， CERNEA D， et al. Multiple view stereo with quadtree-guided priors［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2023， 196： 197-209.
34	KONG W， XU Q， SU W， et al. LGP-MVS： combined local and global planar priors guidance for indoor multi-view stereo［J］. The Visual Computer， 2023， 39（12）： 6421-6433.
35	CERNEA D. OpenMVS： multi-view stereo reconstruction library［EB/OL］. ［2024-05-22］..
36	WANG Y， ZENG Z， GUAN T， et al. Adaptive patch deformation for textureless-resilient multi-view stereo［C］// Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2023： 1621-1630.
37	YUAN Z， CAO J， WANG Z， et al. TSAR-MVS： textureless-aware segmentation and correlative refinement guided multi-view stereo［J］. Pattern Recognition， 2024， 154： No.110565.
38	KUHN A， LIN S， ERDLER O. Plane completion and filtering for multi-view stereo reconstruction［C］// Proceedings of the 2019 DAGM German Conference on Pattern Recognition， LNCS 11824. Cham： Springer， 2019： 18-32.
39	Tanks and Temples Benchmark［DB/OL］. ［2024-04-23］. .

参数	值	意义
x	5	窗口大小
β	0.12	先验平面系数
α	0.08	像素局部纹理度系数
ε	0.1	与先验平面相关的代价项权重系数
ρ	0.18	多视图匹配代价系数
γ	0.15	可靠点代价阈值
thr	0.2	平面置信度阈值

参数	值	意义
x	5	窗口大小
β	0.12	先验平面系数
α	0.08	像素局部纹理度系数
ε	0.1	与先验平面相关的代价项权重系数
ρ	0.18	多视图匹配代价系数
γ	0.15	可靠点代价阈值
thr	0.2	平面置信度阈值

方法	所有场景			室内			室外
方法	F1分数	准确性	完整性	F1分数	准确性	完整性	F1分数	准确性	完整性
文献［31］方法	85.76	86.17	85.71	85.29	85.54	85.46	87.17	88.05	86.46
文献［32］方法	85.89	91.91	81.49	85.39	91.87	80.67	87.38	92.03	83.93
文献［33］方法	—	—	—	82.59	77.39	79.50	82.58	87.64	85.03
文献［24］方法	81.51	90.54	75.58	80.57	90.60	74.23	84.36	90.35	79.62
ACMH	75.89	89.34	68.62	73.93	91.14	64.81	81.77	83.96	80.03
文献［34］方法	81.27	91.47	75.82	82.08	91.68	74.58	84.50	90.82	79.56
文献［23］方法	81.84	80.24	84.18	80.70	78.74	83.43	85.27	84.73	86.44
文献［35］方法	79.77	81.98	78.54	78.33	82.00	75.92	84.09	81.93	86.41
本文方法	87.18	91.54	83.87	86.91	91.36	83.47	88.00	92.08	85.05

方法	所有场景			室内			室外
方法	F1分数	准确性	完整性	F1分数	准确性	完整性	F1分数	准确性	完整性
文献［31］方法	85.76	86.17	85.71	85.29	85.54	85.46	87.17	88.05	86.46
文献［32］方法	85.89	91.91	81.49	85.39	91.87	80.67	87.38	92.03	83.93
文献［33］方法	—	—	—	82.59	77.39	79.50	82.58	87.64	85.03
文献［24］方法	81.51	90.54	75.58	80.57	90.60	74.23	84.36	90.35	79.62
ACMH	75.89	89.34	68.62	73.93	91.14	64.81	81.77	83.96	80.03
文献［34］方法	81.27	91.47	75.82	82.08	91.68	74.58	84.50	90.82	79.56
文献［23］方法	81.84	80.24	84.18	80.70	78.74	83.43	85.27	84.73	86.44
文献［35］方法	79.77	81.98	78.54	78.33	82.00	75.92	84.09	81.93	86.41
本文方法	87.18	91.54	83.87	86.91	91.36	83.47	88.00	92.08	85.05

方法	平均 F1分数	Intermediate集合
方法	平均 F1分数	Family	Francis	Horse	Lighthouse	M60	Panther	Playground	Train
文献［20］方法	42.14	50.41	22.25	25.63	56.43	44.83	46.97	48.53	42.04
文献［32］方法	59.38	70.93	55.39	51.80	63.83	55.94	59.47	59.51	58.20
文献［24］方法	58.41	70.30	54.06	54.11	61.65	54.16	57.60	58.12	57.25
ACMH	54.82	69.99	49.45	45.12	59.04	52.64	52.37	58.34	51.61
文献［35］方法	55.11	—	—	—	—	—	—	—	—
文献［38］方法	55.88	70.99	49.60	40.34	63.44	57.79	58.91	56.59	49.40
本文方法	58.00	72.08	58.45	45.92	64.16	57.05	57.84	59.45	49.09

Multi-view stereo method based on quadtree prior assistance

基于四叉树先验辅助的多视图立体方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 39

Related Articles 7

Recommended Articles

Metrics

方法	F1分数	完整性	准确性
本文方法/EE	81.51	90.54	75.58
本文方法/Q	82.88	89.47	77.18
本文方法	87.18	91.54	83.87

数据集	场景	图像数	运行时间/s
数据集	场景	图像数	基线方法	本文方法
Strecha	fountain	11	199.51	231.72
Strecha	entry	10	173.02	209.16
ETH3D	electro	45	488.58	551.19
ETH3D	relief_2	31	648.75	739.72

[1]	Wei XIONG, Yibo CHEN, Lizhen ZHANG, Qian YANG, Qin ZOU. Self-supervised monocular depth estimation using multi-frame sequence images [J]. Journal of Computer Applications, 2024, 44(12): 3907-3914.
[2]	Meng ZHOU, Zhangjin HUANG. Focal stack depth estimation method based on defocus blur [J]. Journal of Computer Applications, 2023, 43(9): 2897-2903.
[3]	Wenju LI, Mengying LI, Liu CUI, Wanghui CHU, Yi ZHANG, Hui GAO. Monocular depth estimation method based on pyramid split attention network [J]. Journal of Computer Applications, 2023, 43(6): 1736-1742.
[4]	Wentao ZHANG, Yuanyu WANG, Saize LI. Depth estimation model of single haze image based on conditional generative adversarial network [J]. Journal of Computer Applications, 2022, 42(9): 2865-2875.
[5]	WEN Jing, LI Zhihong. Monocular image depth estimation method based on ResNeXt with squeeze-and-excitation module [J]. Journal of Computer Applications, 2021, 41(1): 215-219.
[6]	ZHANG Hao, ZHANG Qiang, SHAO Siyu, DING Haibin. Application of deep learning to 3D model reconstruction of single image [J]. Journal of Computer Applications, 2020, 40(8): 2351-2357.
[7]	LIAO Bin, LI Haowen. Image depth estimation model based on atrous convolutional neural network [J]. Journal of Computer Applications, 2019, 39(1): 267-274.