Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer

doi:10.11772/j.issn.1001-9081.2022040536

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (5): 1596-1605.DOI: 10.11772/j.issn.1001-9081.2022040536

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer

Jinwen GUO¹, Xinghua MA¹, Gongning LUO¹, Wei WANG¹, Yang CAO², Kuanquan WANG¹()

^1.Faculty of Computing，Harbin Institute of Technology，Harbin Heilongjiang 150001，China
^2.The First Affiliated Hospital of Harbin Medical University，Harbin Heilongjiang 150001，China

Received:2022-04-21 Revised:2022-07-05 Accepted:2022-07-08 Online:2022-08-05 Published:2023-05-10
Contact: Kuanquan WANG
About author:GUO Jinwen， born in 1997， M. S. candidate. His research interests include medical image processing， computer vision.
MA Xinghua， born in 1998， M. S. candidate. His research interests include computer vision， medical image analysis.
LUO Gongning， born in 1989， Ph. D.， associate professor. His research interests include computational cardiology， medical image analysis.
WANG Wei， born in 1989， Ph. D.， assistant professor. Her research interests include virtual heart， medical image processing.
CAO Yang， born in 1984， Ph. D.， professor. His research interests include coronary artery disease diagnosis/treatment， coronary interventional therapy.
WANG Kuanquan， born in 1964， Ph. D.， professor. His research interests include virtual reality and simulation， computational cardiology， artificial intelligence and smart healthcare.
Supported by:
National Natural Science Foundation of China(62001141);the Seventh Batch of Teaching Development Fund Projects in Harbin Institute of Technology(XYSZ2021048)

基于Transformer的结构强化IVOCT导丝伪影去除方法

郭劲文¹, 马兴华¹, 骆功宁¹, 王玮¹, 曹阳², 王宽全¹()

^1.哈尔滨工业大学计算学部，哈尔滨 150001
^2.哈尔滨医科大学附属第一医院，哈尔滨 150001

通讯作者: 王宽全
作者简介:郭劲文（1997—），男，宁夏银川人，硕士研究生，主要研究方向：医学图像处理、计算机视觉
马兴华（1998—），男，黑龙江哈尔滨人，硕士研究生，主要研究方向：计算机视觉、医学图像分析
骆功宁（1989—），男，山东烟台人，副教授，博士，主要研究方向：计算心脏学、医学图像分析
王玮（1989—），女，四川达州人，助理教授，博士，主要研究方向：虚拟心脏、医学图像处理
曹阳（1984—），男，黑龙江人，教授，博士，主要研究方向：冠心病诊疗、冠脉介入治疗
王宽全（1964—），男，四川达州人，教授，博士，CCF会员，主要研究方向：虚拟现实与仿真、计算心脏学、人工智能与智慧医疗。wangkq@hit.edu.cn
基金资助:
国家自然科学基金资助项目(62001141);哈尔滨工业大学第七批教学发展基金资助项目(XYSZ2021048)

Abstract

Abstract:

Improving the image quality of IntraVascular Optical Coherence Tomography （IVOCT） through guidewire artifact removal can assist physicians in diagnosing cardiovascular diseases more accurately， which reduces the probabilities of misdiagnosis and missed diagnosis. Aiming at the difficulties of complex structure information and a large proportion of artifact areas in IVOCT images， a Structure-Enhanced Transformer Network （SETN） using Generative Adversarial Network （GAN） architecture was proposed for guidewire artifact removal of IVOCT images. Firstly， based on the ORiginal Image （ORI） backbone generation network for extracting texture features， the generator of GAN was combined with RTV （Relative Total Variation） image enhanced generation network in parallel to obtain image structure information. Next， during the artifact area reconstruction of ORI/RTV image， Transformer encoders focusing on the temporal/spatial domain information respectively were introduced to capture the contextual information and the correlation between texture/structure features of IVOCT image sequence. Finally， the structural feature fusion module was used to integrate the structural features of different levels into the decoding stage of the ORI backbone generation network， so that the generator was cooperated with the discriminator for completing the image reconstruction of the guidewire artifact area. Experimental results show that the guidewire artifact removal results of SETN are excellent in both texture and structure reconstruction. Besides， the improvement of IVOCT image quality after guidewire artifact removal is positive for both vulnerable plaque segmentation and lumen contour extraction tasks of IVOCT image.

Key words: Generative Adversarial Network (GAN), Transformer, structure enhancement, IntraVascular Optical Coherence Tomography (IVOCT), artifact removal

摘要：

为去除导丝伪影以提高血管内光学相干断层扫描（IVOCT）的图像质量，辅助医师更加准确地诊断心血管疾病，降低误诊及漏诊的概率，针对IVOCT图像结构信息复杂且伪影区域占比大的难点，提出一种采用生成对抗网络（GAN）架构的基于Transformer的结构强化网络（SETN）。首先，GAN的生成器在提取纹理特征的原始图像（ORI）主干生成网络的基础上，并联了RTV（Relative Total Variation）图像强化生成网络用于获取图像的结构信息；其次，在ORI/RTV图像的伪影区域重建过程中，引入了分别关注时/空间域信息的Transformer编码器，用于捕获IVOCT图像序列的上下文信息以及纹理/结构特征之间的关联性；最后，利用结构特征融合模块将不同层次的结构特征融入ORI主干生成网络的解码阶段，配合判别器完成导丝伪影区域的图像重建。实验结果表明，SETN的导丝伪影去除结果在纹理和结构的重建上均十分优秀。此外，导丝伪影去除后IVOCT图像质量的提高，对于IVOCT图像的易损斑块分割及管腔轮廓线提取任务均具有积极意义。

关键词: 生成对抗网络, Transformer, 结构强化, 血管内光学相干断层扫描, 伪影去除

CLC Number:

TP183

Jinwen GUO, Xinghua MA, Gongning LUO, Wei WANG, Yang CAO, Kuanquan WANG. Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer[J]. Journal of Computer Applications, 2023, 43(5): 1596-1605.

郭劲文, 马兴华, 骆功宁, 王玮, 曹阳, 王宽全. 基于Transformer的结构强化IVOCT导丝伪影去除方法[J]. 《计算机应用》唯一官方网站, 2023, 43(5): 1596-1605.

Figures/Tables 15

Fig. 1 IVOCT image and guidewire artifact

Fig. 2 Generator network structure of SETN

Fig. 3 Transformer encoder and temporal/spatial domain feature integrators

Fig. 4 RTV results of IVOCT image

Fig. 5 Structural feature fusion module

Tab. 1 Parameter values of loss function

损失函数	参数名称	参数值
缺失区域与未缺失区域	$λ h o l e$	1.00
缺失区域与未缺失区域	$λ v a l i d$	1.00
ORI与RTV	$ω$	0.20
ORI与RTV	$τ$	0.20
重建与对抗	$λ r e c$	1.00
重建与对抗	$λ a d v$	0.01

Tab. 1 Parameter values of loss function

损失函数	参数名称	参数值
缺失区域与未缺失区域	$λ h o l e$	1.00
缺失区域与未缺失区域	$λ v a l i d$	1.00
ORI与RTV	$ω$	0.20
ORI与RTV	$τ$	0.20
重建与对抗	$λ r e c$	1.00
重建与对抗	$λ a d v$	0.01

Fig. 6 Flow of IVOCT image preprocessing

Fig. 7 Artifact trajectory and width statistics

Tab. 2 Evaluation results of guidewire artifact removal

方法	PSNR/dB	SSIM/%	MAE	FID
VINet^［20］	32.32	93.79	0.017 5	23.87
LGTSM^［21］	32.85	94.58	0.017 2	16.37
CAP^［22］	34.68	95.60	0.016 8	18.26
STTN^［26］	35.21	96.08	0.016 4	10.75
SETN_TR-	27.16	92.02	0.018 2	65.05
SETN_RTV-	35.51	96.11	0.016 2	10.99
SETN	36.03	96.25	0.015 4	9.32

Fig. 8 Box plots of experimental results

Fig. 9 Detail comparison of reconstruction results of guidewire artifact area

Fig. 10 Structural comparison of reconstruction results of guidewire artifact area

Tab. 3 Segmentation results of different input

方法	PA	MPA	IoU	DICE	Precision	Recall
未处理	94.67	92.15	86.95	84.31	91.60	87.41
STTN^［26］	98.20	97.20	95.05	94.11	96.82	95.42
SETN_RTV-	98.58	97.19	95.57	94.69	97.68	95.17
SETN	98.59	97.29	95.72	95.24	95.22	95.22

Fig. 11 Segmentation results before and after guidewire artifact removal

Fig. 12 Contour comparison of guidewire artifact area

References 26

1	中国心血管健康与疾病报告编写组. 中国心血管健康与疾病报告2020概要［J］. 中国循环杂志， 2021， 36（6）：521-545. 10.3969/j.issn.1000-3614.2021.06.001
	The Writing Committee of the Report on Cardiovascular Health and Diseases in China. Report on cardiovascular health and diseases burden in China： an updated summary of 2020［J］. Chinese Circulation Journal， 2021， 36（6）： 521-545. 10.3969/j.issn.1000-3614.2021.06.001
2	韩涛，邱建榕，王迪，等. 光学相干层析显微成像的技术与应用［J］. 中国激光， 2020， 47（2）： No.0207004. 10.3788/cjl202047.0207004
	HAN T， QIU J R， WANG D， et al. Optical coherence microscopy and its application［J］. Chinese Journal of Lasers， 2020， 47（2）： No.0207004. 10.3788/cjl202047.0207004
3	陆冬筱，房文汇，李玉瑶，等. 光学相干层析成像技术原理及研究进展［J］. 中国光学， 2020， 13（5）：919-935. 10.37188/CO.2020-0037
	LU D X， FANG W H， LI Y Y， et al. Optical coherence tomography： principles and recent developments［J］. Chinese Optics， 2020， 13（5）：919-935. 10.37188/CO.2020-0037
4	FERCHER A F， DREXLER W， HITZENBERGER C K， et al. Optical coherence tomography - principles and applications［J］. Reports on Progress in Physics， 2003， 66（2）： No.239. 10.1088/0034-4885/66/2/204
5	FERCHER A F. Optical coherence tomography - development， principles， applications［J］. Zeitschrift für Medizinische Physik， 2010， 20（4）： 251-276. 10.1016/j.zemedi.2009.11.002
6	JANG I K， BOUMA B E， KANG D H， et al. Visualization of coronary atherosclerotic plaques in patients using optical coherence tomography： comparison with intravascular ultrasound［J］. Journal of the American College of Cardiology， 2002， 39（4）： 604-609. 10.1016/s0735-1097(01)01799-5
7	TEARNEY G J， REGAR E， AKASAKA T， et al. Consensus standards for acquisition， measurement， and reporting of intravascular optical coherence tomography studies： a report from the international working group for intravascular optical coherence tomography standardization and validation［J］. Journal of the American College of Cardiology， 2012， 59（12）： 1058-1072. 10.1016/j.jacc.2011.09.079
8	REGAR E， AKASAKA T， ADRIAENSSENS T， et al. Consensus standards for acquisition， measurement， and reporting of intravascular optical coherence tomography studies［J］. Journal of the American College of Cardiology， 2012， 59： 1058-1072.
9	PRATI F， CERA M， RAMAZZOTTI V， et al. Safety and feasibility of a new non-occlusive technique for facilitated intracoronary Optical Coherence Tomography （OCT） acquisition in various clinical and anatomical scenarios［J］. EuroIntervention， 2007， 3（3）： 365-370. 10.4244/eijv3i3a66
10	XU L， YAN Q， XIA Y， et al. Structure extraction from texture via relative total variation［J］. ACM Transactions on Graphics， 2012， 31（6）： No.139. 10.1145/2366145.2366158
11	YU L F， LI H， MUELLER J， et al. Metal artifact reduction from reformatted projections for hip prostheses in multislice helical computed tomography： techniques and initial clinical results［J］. Investigative Radiology， 2009， 44（11）： 691-696. 10.1097/rli.0b013e3181b0a2f9
12	LIU J Y， YANG S， FANG Y M， et al. Structure-guided image inpainting using homography transformation［J］. IEEE Transactions on Multimedia， 2018， 20（12）： 3252-3265. 10.1109/tmm.2018.2831636
13	DING D， RAM S， RODRIGUEZ J J. Image inpainting using nonlocal texture matching and nonlinear filtering［J］. IEEE Transactions on Image Processing， 2019， 28（4）： 1705-1719. 10.1109/tip.2018.2880681
14	LI H， LUO W， HUANG J. Localization of diffusion-based inpainting in digital images［J］. IEEE Transactions on Information Forensics and Security， 2017， 12（12）： 3050-3064. 10.1109/tifs.2017.2730822
15	SRIDEVI G， SRINIVAS KUMAR S. Image inpainting based on fractional-order nonlinear diffusion for image reconstruction［J］. Circuits， Systems， and Signal Processing， 2019， 38（8）： 3802-3817. 10.1007/s00034-019-01029-w
16	XIE J Y， XU L L， CHEN E H. Image denoising and inpainting with deep neural networks［C］// Proceedings of the 25th International Conference on Neural Information Processing Systems — Volume 1. Red Hook， NY： Curran Associates Inc.， 2012： 341-349.
17	FAVORSKAYA M， DAMOV M， ZOTIN A. Intelligent texture reconstruction of missing data in video sequences using neural networks［M］// TWEEDALE J W， JAIN， L C. Advanced Techniques for Knowledge Engineering and Innovative Applications： 16th International Conference， KES 2012， San Sebastian， Spain， September 10-12， 2012， Revised Selected Papers， CCIS 246. Berlin： Springer， 2013： 163-176. 10.1007/978-3-642-42017-7_12
18	KÖHLER R， SCHULER C， SCHÖLKOPF B， et al. Mask-specific inpainting with deep neural networks［C］// Proceedings of the 2014 German Conference on Pattern Recognition， LNCS 8753. Cham： Springer， 2014： 523-534.
19	GOODFELLOW I J， POUGET-ABADIE J， MIRZA M， et al. Generative adversarial nets［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems — Volume 2. Cambridge： MIT Press， 2014： 2672-2680.
20	KIM D， WOO S， LEE J Y， et al. Deep video inpainting［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 5785-5794. 10.1109/cvpr.2019.00594
21	CHANG Y L， LIU Z Y， LEE K Y， et al. Free-form video inpainting with 3D gated convolution and Temporal PatchGAN［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 9065-9074. 10.1109/iccv.2019.00916
22	LEE S， OH S W， WON D， et al. Copy-and-paste networks for deep video inpainting［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 4412-4420. 10.1109/iccv.2019.00451
23	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 6000-6010.
24	DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An image is worth 16x16 words： Transformers for image recognition at scale［EB/OL］. （2021-06-03）［2022-03-22］..
25	JIANG Y F， CHANG S Y， WANG Z Y. TransGan： two pure transformers can make one strong GAN， and that can scale up［C/OL］// Proceedings of the 35th Conference on Neural Information Processing Systems ［2022-03-22］. .
26	ZENG Y H， FU J L， CHAO H Y. Learning joint spatial-temporal transformations for video inpainting［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12361. Cham： Springer， 2020： 528-543.

[1]	Jiepo FANG, Chongben TAO. Hybrid internet of vehicles intrusion detection system for zero-day attacks [J]. Journal of Computer Applications, 2024, 44(9): 2763-2769.
[2]	Yunchuan HUANG, Yongquan JIANG, Juntao HUANG, Yan YANG. Molecular toxicity prediction based on meta graph isomorphism network [J]. Journal of Computer Applications, 2024, 44(9): 2964-2969.
[3]	Xin YANG, Xueni CHEN, Chunjiang WU, Shijie ZHOU. Short-term traffic flow prediction of urban highway based on variant residual model and Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2947-2951.
[4]	Jinjin LI, Guoming SANG, Yijia ZHANG. Multi-domain fake news detection model enhanced by APK-CNN and Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2674-2682.
[5]	Liehong REN, Lyuwen HUANG, Xu TIAN, Fei DUAN. Multivariate long-term series forecasting method with DFT-based frequency-sensitive dual-branch Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2739-2746.
[6]	Jieru JIA, Jianchao YANG, Shuorui ZHANG, Tao YAN, Bin CHEN. Unsupervised person re-identification based on self-distilled vision Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2893-2902.
[7]	Yuwei DING, Hongbo SHI, Jie LI, Min LIANG. Image denoising network based on local and global feature decoupling [J]. Journal of Computer Applications, 2024, 44(8): 2571-2579.
[8]	Kaili DENG, Weibo WEI, Zhenkuan PAN. Industrial defect detection method with improved masked autoencoder [J]. Journal of Computer Applications, 2024, 44(8): 2595-2603.
[9]	Fan YANG, Yao ZOU, Mingzhi ZHU, Zhenwei MA, Dawei CHENG, Changjun JIANG. Credit card fraud detection model based on graph attention Transformation neural network [J]. Journal of Computer Applications, 2024, 44(8): 2634-2642.
[10]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[11]	Dahai LI, Zhonghua WANG, Zhendong WANG. Dual-branch low-light image enhancement network combining spatial and frequency domain information [J]. Journal of Computer Applications, 2024, 44(7): 2175-2182.
[12]	Shibin LI, Jun GONG, Shengjun TANG. Semi-supervised heterophilic graph representation learning model based on Graph Transformer [J]. Journal of Computer Applications, 2024, 44(6): 1816-1823.
[13]	Xiting LYU, Jinghua ZHAO, Haiying RONG, Jiale ZHAO. Information diffusion prediction model based on Transformer and relational graph convolutional network [J]. Journal of Computer Applications, 2024, 44(6): 1760-1766.
[14]	Xun YAO, Zhongzheng QIN, Jie YANG. Generative label adversarial text classification model [J]. Journal of Computer Applications, 2024, 44(6): 1781-1785.
[15]	Junfeng SHEN, Xingchen ZHOU, Can TANG. Dual-channel sentiment analysis model based on improved prompt learning method [J]. Journal of Computer Applications, 2024, 44(6): 1796-1806.

Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer

基于Transformer的结构强化IVOCT导丝伪影去除方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 26

Related Articles 15

Recommended Articles

Metrics