Design of augmented reality navigation simulation system for pelvic minimally invasive surgery based on stereoscopic vision

doi:10.11772/j.issn.1001-9081.2018020335

Journal of Computer Applications ›› 2018, Vol. 38 ›› Issue (9): 2660-2665.DOI: 10.11772/j.issn.1001-9081.2018020335

Previous Articles Next Articles

Design of augmented reality navigation simulation system for pelvic minimally invasive surgery based on stereoscopic vision

GAO Qinquan^1,2, HUANG Weiping^1,2, DU Min², WEI Mengyu^2,3, KE Dongzhong^1,2

1. School of Physics and Information Engineering, Fuzhou University, Fuzhou Fujian 350116, China;
2. Key Lab of Medical Instrumentation and Pharmaceutical Technology of Fujian Province, Fuzhou University, Fuzhou Fujian 350116, China;
3. State Key Laboratory of Analog and Mixed-Signal VLSL, University of Macau, Macao 999078, China

Received:2018-02-22 Revised:2018-05-04 Online:2018-09-10 Published:2018-09-06
Contact: 柯栋忠
Supported by:
This work is partially supported by the Natural Science Foundation of Fujian Province (2016J05157), the Young Teachers Education Research Foundation of Fujian Province (JAT160074).

基于双目视觉的盆腔微创手术增强现实导航仿真系统的设计

高钦泉^1,2, 黄伟萍^1,2, 杜民², 韦孟宇^2,3, 柯栋忠^1,2

1. 福州大学物理与信息工程学院, 福州 350116;
2. 福州大学福建省医疗器械与医药技术重点实验室, 福州 350116;
3. 澳门大学模拟与混合信号超大规模集成电路国家重点实验室, 澳门 999078

通讯作者: 柯栋忠
作者简介:高钦泉(1986—),男,福建福清人,副教授,博士,主要研究方向:计算机视觉、增强现实、医学图像处理;黄伟萍(1992—),男,福建泉州人,硕士研究生,主要研究方向:计算机视觉、增强现实;杜民(1955—),女,福建泉州人,教授,博士,主要研究方向:传感技术、生物医学仪器;韦孟宇(1964—),男,澳门人,教授,博士,主要研究方向:生物电子学、嵌入式系统;柯栋忠(1960—),男,福建福州人,高级工程师,主要研究方向:模式识别、计算机图像处理。
基金资助:
福建省自然科学基金资助项目（2016J05157）；福建省中青年教师教育科研项目基金资助项目（JAT160074）。

Abstract

Abstract: Minimally invasive endoscopic surgery always remains a challenge due to the complexity of the anatomical location and the limitations of endoscopic vision. An Augmented Reality (AR) navigation system was designed for simulation of pelvic minimally invasive surgery. Firstly, a 3D model of pelvis which was segmented and reconstructed from the preoperative CT (Computed Tomography) was textured mapping with the real pelvic surgical video, and then a surgical video with the ground truth pose was simulated. The blank model was initially registered with the intraoperative video by a 2D/3D registration based on color consistency of visible surface points. After that, an accurate tracking of intraoperative endoscopy was performed using a stereoscopic tracking algorithm. According to the multi-DOFs (Degree Of Freedoms) transformation matrix of endoscopy, the preoperative 3D model could then be fused to the intraoperative vision to achieve an AR navigation. The experimental results show that the root mean square error of the estimated trajectory compared to the ground truth is 2.3933 mm, which reveals that the system can achieve a good AR display for visual navigation.

Key words: 2D/3D registration, color-consistency, endoscopic tracking, augmented reality, surgery navigating system

摘要： 基于内窥镜视觉导航的盆腔微创手术，往往因为病灶的复杂解剖位置以及内窥镜视野的局限性，对手术的顺利开展提出了重要的挑战。针对这个问题，设计并开发一套基于立体视觉的盆腔微创手术的增强现实（AR）导航仿真系统。首先，利用术前的CT影像重建骨盆3D模型以及盆腔的真实手术视频，生成带有纹理信息的骨盆3D模型，仿真一套具有真实轨迹的手术视频；然后，利用基于可视点颜色一致性的2D/3D配准技术，实现术前重建模型与手术视野的初始化配准。利用立体视觉跟踪算法，对手术过程中的内窥镜位置进行跟踪，根据内窥镜多自由度的变换矩阵来实现术前3D模型与手术视野的融合与增强现实导航。估计轨迹与真实轨迹的均方根误差为2.3933 mm，仿真实验表明，导航系统为视觉导航提供良好的增强现实显示效果。

关键词: 2D/3D配准, 颜色一致性, 内窥镜跟踪, 增强现实, 手术导航仿真系统

CLC Number:

TP391.9

GAO Qinquan, HUANG Weiping, DU Min, WEI Mengyu, KE Dongzhong. Design of augmented reality navigation simulation system for pelvic minimally invasive surgery based on stereoscopic vision[J]. Journal of Computer Applications, 2018, 38(9): 2660-2665.

高钦泉, 黄伟萍, 杜民, 韦孟宇, 柯栋忠. 基于双目视觉的盆腔微创手术增强现实导航仿真系统的设计[J]. 计算机应用, 2018, 38(9): 2660-2665.

References

[1] KAPLAN J R, LEE Z, EUN D D, et al. Complications of minimally invasive surgery and their management[J]. Current Urology Reports, 2016, 17(6):1-10.
[2] FITZPATRICK J M, WEST J B, JR M C. Predicting error in rigid-body point-based registration[J]. IEEE Transactions on Medical Imaging, 1998, 17(5):694-702.
[3] 马帅依凡,赵子健.基于人工标记的手术导航仪[J].山东大学学报(工学版),2017,47(3):63-68.(MA S Y F, ZHAO Z J. Surgical navigation system based on an artificial marker[J]. Journal of Shandong University (Engineering Science), 2017, 47(3):63-68.)
[4] LIU X, KANG S, PLISHKER W, et al. Laparoscopic stereoscopic augmented reality:toward a clinically viable electromagnetic tracking solution[J]. Journal of Medical Imaging, 2016, 3(4):045001.
[5] PETERS T M. Image-guided surgery:from X-rays to virtual reality[J]. Computer Methods in Biomechanics and Biomedical Engineering, 2001, 4(1):27-57.
[6] VINOGRADSKIY Y Y, CASTILLO R, CASTILLO E, et al. Use of weekly 4DCT-based ventilation maps to quantify changes in lung function for patients undergoing radiation therapy[J]. Medical Physics, 2012, 39(1):289-298.
[7] ZHANG Z. A flexible new technique for camera calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11):1330-1334.
[8] CHANG P L, CHEN D, COHEN D, et al. 2D/3D registration of a preoperative model with endoscopic video using colour-consistency[C]//Proceedings of the 2011 Workshop on Augmented Environments for Computer-Assisted Interventions, LNCS 7264. Berlin:Springer, 2011:1-12.
[9] 和青芳.计算机图形学原理及算法教程[M].北京:清华大学出版社,2010:210-214.(HE Q F. Computer Graphics Principles and Algorithms Tutorial[M]. Beijing:Tsinghua University Press, 2010:210-214.)
[10] KLEIN G, MURRAY D. Parallel tracking and mapping for small AR workspaces[C]//ISMAR'07:Proceedings of the 20076th IEEE and ACM International Symposium on Mixed and Augmented Reality. Washington, DC:IEEE Computer Society, 2008:1-10.
[11] MUR-ARTAL R, TARDÓS J D. ORB-SLAM2:an open-source SLAM system for monocular, stereo, and RGB-D cameras[J]. IEEE Transactions on Robotics, 2017, 33(5):1255-1262.
[12] MUR-ARTAL R, MONTIEL J M M, TARDÓS J D. ORB-SLAM:a versatile and accurate monocular SLAM system[J]. IEEE Transactions on Robotics, 2017, 31(5):1147-1163.
[13] NEWCOMBE R A, LOVEGROVE S J, DAVISON A J. DTAM:dense tracking and mapping in real-time[C]//ICCV'11:Proceedings of the 2011 International Conference on Computer Vision. Washington, DC:IEEE Computer Society, 2011:2320-2327.
[14] FORSTER C, PIZZOLI M, SCARAMUZZA D. SVO:fast semi-direct monocular visual odometry[C]//Proceedings of the 2014 IEEE International Conference on Robotics and Automation. Piscataway, NJ:IEEE, 2014:15-22.
[15] FORSTER C, ZHANG Z, GASSNER M, et al. SVO:semidirect visual odometry for monocular and multicamera systems[J]. IEEE Transactions on Robotics, 2017, 33(2):249-265.
[16] DAVISON A J, REID I D, MOLTON N D, et al. MonoSLAM:Real-time single camera SLAM[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6):1052-1067.
[17] LOURAKIS M I A, ARGYROS A A. SBA:a software package for generic sparse bundle adjustment[J]. ACM Transactions on Mathematical Software, 2009, 36(1):Article No. 2.
[18] POWELL M J D. The BOBYQA algorithm for bound constrained optimization without derivatives[EB/OL].[2017-12-10]. http://www.optimization-online.org/DB_FILE/2010/05/2616.pdf.
[19] KVMMERLE R, GRISETTI G, STRASDAT H, et al. g²o:a general framework for graph optimization[EB/OL].[2017-12-10]. http://www.willowgarage.com/sites/default/files/g2o-icra11.pdf.

Design of augmented reality navigation simulation system for pelvic minimally invasive surgery based on stereoscopic vision

基于双目视觉的盆腔微创手术增强现实导航仿真系统的设计

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 12

Recommended Articles

Metrics

[1]	TAN Zhaoyi, CHEN Baifan. Enhanced vehicle 3D surround view based on coordinate inverse mapping [J]. Journal of Computer Applications, 2021, 41(4): 1165-1171.
[2]	SUN Qichang, MAI Yongfeng, CHEN Xiaojun. Fast calibration algorithm in surgical navigation system based on augmented reality [J]. Journal of Computer Applications, 2021, 41(3): 833-838.
[3]	LUO Bin, YU Bo. Computation offloading strategy based on particle swarm optimization in mobile edge computing [J]. Journal of Computer Applications, 2020, 40(8): 2293-2298.
[4]	ZHOU Xiang, TANG Liyu, LIN Ding. Virtual-real registration method of natural features based on binary robust invariant scalable keypoints and speeded up robust features [J]. Journal of Computer Applications, 2020, 40(5): 1403-1408.
[5]	LU Xiaoyong, YOU Bin, LIN Pei-Yu, CHEN Musheng. Augmented reality approach based on digital camera and temporal psycho-visual modulation [J]. Journal of Computer Applications, 2017, 37(8): 2298-2301.
[6]	LI Hua, WANG Xuyang, YANG Huamin, HAN Cheng. Illumination direction measurement based on halo analysis in high-dynamic range images [J]. Journal of Computer Applications, 2016, 36(5): 1387-1393.
[7]	WANG Haipeng, WANG Zhengliang, XU Weiwei, FAN Ran. Real-time face pose estimation system based on 3D face model on Android mobile platform [J]. Journal of Computer Applications, 2015, 35(8): 2321-2326.
[8]	LI Hong-bo WU Liang-liang WU Yu. Shadow generation algorithm of augmented reality using adaptive sampling and fusion [J]. Journal of Computer Applications, 2012, 32(07): 1860-1863.
[9]	. Design and implementation of augmented reality system based on hybrid tracking [J]. Journal of Computer Applications, 2009, 29(10): 2852-2854.
[10]	. Augmented reality application based on hand shape interaction and palmprint recognition [J]. Journal of Computer Applications, 2009, 29(08): 2083-2086.
[11]	Jianhua Jin;. Three dimensional registration techniques for augmented reality based on computer vision and magnetic tracking [J]. Journal of Computer Applications, 2006, 26(6): 1485-1489.
[12]	LIU Guo-yi, LI Hua. Planar pattern tracking and its application in augmented reality [J]. Journal of Computer Applications, 2005, 25(07): 1605-1607.