Pixel-level registration technology for multi-modal weld seam images

doi:10.11772/j.issn.1001-9081.2024040524

Journal of Computer Applications ›› 0, Vol. ›› Issue (): 229-233.DOI: 10.11772/j.issn.1001-9081.2024040524

• Multimedia computing and computer simulation • Previous Articles Next Articles

Pixel-level registration technology for multi-modal weld seam images

Zhenrong HUANG¹, Yao HUANG², Wang TU², Fei WANG², Bin CHEN¹^,³()

^1.International Institute of Artificial Intelligence，Harbin Institute of Technology，Shenzhen，Shenzhen Guangdong 518055，China
^2.Shanghai Spaceflight Precision Machinery Institute，Shanghai Academy of Spaceflight Technology，Shanghai 201600，China
^3.Chongqing Research Institute，Harbin Institute of Technology，Chongqing 401151，China

Received:2024-04-26 Revised:2024-07-01 Accepted:2024-07-04 Online:2025-01-24 Published:2024-12-31
Contact: Bin CHEN

多模态焊缝图像像素级配准技术

黄振荣¹, 黄瑶², 涂旺², 王飞², 陈斌¹^,³()

^1.哈尔滨工业大学（深圳）国际人工智能研究院，广东深圳 518055
^2.上海航天技术研究院上海航天精密机械研究所，上海 201600
^3.哈尔滨工业大学重庆研究院，重庆 401151

通讯作者: 陈斌
作者简介:黄振荣（2000—），男，广东梅州人，硕士研究生，主要研究方向：多模态图像配准、图像生成
黄瑶（1996—），女，江苏南通人，工程师，硕士，主要研究方向：自动化、智能化检测
涂旺（1998—），男，江西宜春人，助理工程师，硕士，主要研究方向：射线检测
王飞（1986—），男，江苏南通人，研究员，硕士，主要研究方向：无损检测
陈斌（1970—），男，四川广汉人，研究员，博士，主要研究方向：工业视觉。
基金资助:
深圳市2022年高校稳定支持计划项目(GXWD?20220811170603002)

Abstract

Abstract:

To exploit the visual features of multi-modal industrial weld seam images and further improve the registration effect through modal translation， a modal translation-based network for pixel-level registration of multi-modal weld seam images was proposed. Firstly， a cross-modal translation module was designed to make the network have the capability to capture shared features of different modalities of industrial images. Then， the shared features were captured to perform multi-modal image registration. At the same time， adversarial loss and multi-level contrastive loss were used to improve the modal translation effect. Additionally， the cross-modal translation module was integrated with the unimodal image registration module， and reconstruction loss was employed to improve pixel-level registration performance. Finally， a multi-modal industrial weld seam image dataset was constructed， and experiments were conducted using this dataset for comparison. Experimental results demonstrate that the proposed network significantly outperforms the existing advanced multi-modal image registration models such as DFMIR （Discriminator-Free-Medical-Image-Registration） and IMSE （Indescribable Multi-modal Spatial Evaluator）， achieving 3.9 and 3.2 percentage point increases in mean Intersection over Union （mIoU） and 16- and 11-pixel registration accuracy improvements in average Euclidean distance （aEd）， thereby obtaining good results in pixel-level registration.

Key words: industrial weld seam imaging, multi-modal image registration, modal translation, shared feature, multi-level contrastive learning

摘要：

为了挖掘多模态工业焊缝图像视觉特征并通过模态翻译的方式进一步提高多模态工业图像的配准效果，提出一种基于模态翻译的多模态焊缝图像像素级配准网络。首先，通过设计跨模态翻译模块赋予网络捕获不同模态工业图像共享特征的感知能力；其次，捕获共享特征以进行多模态图像配准，并利用对抗性损失和多层级对比损失提高模态翻译效果；同时，结合跨模态翻译模块与单模态图像配准模块，并通过重构损失提升像素级配准性能；最后，构建多模态工业焊缝图像数据集，并基于此数据集开展对比实验。实验结果表明，相较于DFMIR（Discriminator-Free-Medical-Image-Registration）和IMSE（Indescribable Multi-modal Spatial Evaluator）等现有的先进多模态图像配准模型，所提网络在平均交并比（mIoU）上提升了3.9、3.2个百分点，在平均欧氏距离（aEd）上提升了约16、11个像素点的配准精度，在像素级别配准上取得了较好的结果。

关键词: 工业焊缝成像, 多模态图像配准, 模态翻译, 共享特征, 多层级对比学习

CLC Number:

TP391.4

Zhenrong HUANG, Yao HUANG, Wang TU, Fei WANG, Bin CHEN. Pixel-level registration technology for multi-modal weld seam images[J]. Journal of Computer Applications, 0, (): 229-233.

黄振荣, 黄瑶, 涂旺, 王飞, 陈斌. 多模态焊缝图像像素级配准技术[J]. 《计算机应用》唯一官方网站, 0, (): 229-233.

Figures/Tables 7

References 18

1	黄子杰，欧阳，江德港，等. 面向牵引座焊缝表面质量检测的轻量型深度学习算法［J］. 计算机应用， 2024， 44（3）： 983-988.
2	陶显，侯伟，徐德. 基于深度学习的表面缺陷检测方法综述［J］. 自动化学报， 2021， 47（5）：1017-1034.
3	ARAR M， GINGER Y， DANON D， et al. Unsupervised multi-modal image registration via geometry preserving image-to-image translation［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020：13410-13419.
4	CHEN Z， WEI J， LI R. Unsupervised multi-modal medical image registration via discriminator-free image-to-image translation［C］// Proceedings of the 31st International Joint Conferences on Artificial Intelligence. California： ijcai.org， 2022：834-840.
5	KONG L， QI X S， SHEN Q， et al. Indescribable multi-modal spatial evaluator［C］// Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2023： 9853-9862.
6	KONG L， LIAN C， HUANG D， et al. Breaking the dilemma of medical image-to-image translation［EB/OL］. ［2023-12-30］..
7	PIELAWSKI N， WETZER E， ÖFVERSTEDT J， et al. CoMIR： contrastive multimodal image representation for registration［C］// Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2020： 18433-18444.
8	QIN C， SHI B， LIAO R， et al. Unsupervised deformable registration for multi-modal images via disentangled representations［C］// Proceedings of the 2019 International Conference on Information Processing in Medical Imaging， LNCS 11492. Cham： Springer， 2019： 249-261.
9	PLUIM J P W， MAINTZ J B A， VIERGEVER M A. Mutual-information-based registration of medical images： a survey［J］. IEEE Transactions on Medical Imaging， 2003， 22（8）： 986-1004.
10	SARVAIYA J N， PATNAIK S， BOMBAYWALA S. Image registration by template matching using normalized cross-correlation［C］// Proceedings of the 2009 International Conference on Advances in Computing， Control， and Telecommunication Technologies. Piscataway： IEEE， 2009： 819-822.
11	HOOPES A， HOFFMANN M， FISCHL B， et al. HyperMorph： amortized hyperparameter learning for image registration［C］// Proceedings of the 2021 International Conference on Information Processing in Medical Imaging， LNCS 12729. Cham： Springer， 2021： 3-17.
12	PARK T， EFROS A A， ZHANG R， et al. Contrastive learning for unpaired image-to-image translation［EB/OL］. ［2023-12-30］..
13	HE K， ZHANG X， REN S， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778.
14	RONNCBERGER O， FISCHER P， BROX T. U-net： convolutional networks for biomedical image segmentation［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9351. Cham： Springer， 2015： 234-241.
15	MARSTAL K， BERENDSEN F， STARING M， et al. SimpleElastix： a user-friendly， multi-lingual library for medical image registration［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 574-582.
16	HEINRICH M P， JENKINSON M， BHUSHAN M， et al. MIND： modality independent neighbourhood descriptor for multi-modal deformable registration［J］. Medical Image Analysis， 2012， 16（7）： 1423-1435.
17	BALAKRISHNAN G， ZHAO A， SABUNCU M R， et al. VoxelMorph： a learning framework for deformable medical image registration［J］. IEEE Transactions on Medical Imaging， 2019， 38（8）： 1788-1800.
18	ZHU J Y， PARK T， ISOLA P， et al. Unpaired image-to-image translation using cycle-consistent adversarial networks［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2242-2251.

模型	mIoU	aEd
初始值+目标检测	0.445	100.79
Affine^［15］+目标检测	0.483	94.05
MIND^［16］+目标检测	0.503	83.34
CGAN^［18］+目标检测	0.607	51.62
RGPT^［3］+目标检测	0.796	35.06
DFMIR^［4］+目标检测	0.892	20.67
IMSE^［5］+目标检测	0.899	15.46
本文网络	0.931	4.18

模型	mIoU	aEd
初始值+目标检测	0.445	100.79
Affine^［15］+目标检测	0.483	94.05
MIND^［16］+目标检测	0.503	83.34
CGAN^［18］+目标检测	0.607	51.62
RGPT^［3］+目标检测	0.796	35.06
DFMIR^［4］+目标检测	0.892	20.67
IMSE^［5］+目标检测	0.899	15.46
本文网络	0.931	4.18

目标检测模块	跨模态翻译模块	mIoU	aEd
		0.931	4.18
√		0.591	88.18
	√	0.801	30.78
√	√	0.446	95.97

目标检测模块	跨模态翻译模块	mIoU	aEd
		0.931	4.18
√		0.591	88.18
	√	0.801	30.78
√	√	0.446	95.97

[1]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[2]	Sailong SHI, Zhiwen FANG. Gaze estimation model based on multi-scale aggregation and shared attention [J]. Journal of Computer Applications, 2024, 44(7): 2047-2054.
[3]	HUANG Xueyu, XU Haote, TAO Jianwen. Multi-source adaptation classification framework with feature selection [J]. Journal of Computer Applications, 2020, 40(9): 2499-2506.
[4]	WANG Lifang, CHENG Xi, QIN Pinle, GAO Yuan. Non-rigid multi-modal medical image registration based on multi-channel sparse coding [J]. Journal of Computer Applications, 2018, 38(4): 1127-1133.

Pixel-level registration technology for multi-modal weld seam images

多模态焊缝图像像素级配准技术

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 18

Related Articles 4

Recommended Articles

Metrics