Review of multi-modal medical image segmentation based on deep learning

doi:10.11772/j.issn.1001-9081.2022101636

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (11): 3385-3395.DOI: 10.11772/j.issn.1001-9081.2022101636

• Artificial intelligence • Previous Articles Next Articles

Review of multi-modal medical image segmentation based on deep learning

Meng DOU¹^,², Zhebin CHEN¹^,², Xin WANG³, Jitao ZHOU³, Yu YAO¹^,²()

^1.Chengdu Institute of Computer Application，Chinese Academy of Sciences，Chengdu Sichuan 610213，China
^2.School of Computer Science and Technology，University of Chinese Academy of Sciences，Beijing 100049，China
^3.Department of Abdominal Oncology，West China Hospital of Sichuan University，Chengdu Sichuan 610041，China

Received:2022-11-04 Revised:2023-04-26 Accepted:2023-05-04 Online:2023-05-24 Published:2023-11-10
Contact: Yu YAO
About author:DOU Meng， born in 1993， Ph. D. candidate. His research interests include medical image analysis， deep learning.
CHEN Zhebin， born in 1993， Ph. D. candidate. His research interests include medical image analysis， deep learning.
WANG Xin， born in 1977， Ph. D.， professor. Her research interests include diagnosis and treatment of rectal tumors.
ZHOU Jitao， born in 1985， Ph. D.， attending physician. Her research interests include diagnosis and treatment of rectal tumors.
YAO Yu， born in 1980， Ph. D.， professor. His research interests include medical image analysis， machine learning.
Supported by:
National Natural Science Foundation of China(82073338);Key Research and Development Project of Sichuan Science and Technology Plan(2022YFS0217)

基于深度学习的多模态医学图像分割综述

窦猛¹^,², 陈哲彬¹^,², 王辛³, 周继陶³, 姚宇¹^,²()

^1.中国科学院成都计算机应用研究所, 成都 610213
^2.中国科学院大学计算机科学与技术学院, 北京 100049
^3.四川大学华西医院腹部肿瘤科, 成都 610041

通讯作者: 姚宇
作者简介:窦猛（1993—），男，山东滨州人，博士研究生，主要研究方向：医学图像分析、深度学习
陈哲彬（1993—），男，贵州遵义人，博士研究生，主要研究方向：医学图像分析、深度学习
王辛（1977—），女，四川成都人，教授，博士，主要研究方向：直肠肿瘤诊断与治疗
周继陶（1985—），女，四川成都人，主治医师，博士，主要研究方向：直肠肿瘤诊断与治疗
姚宇（1980—），男，四川宜宾人，教授，博士，主要研究方向：医学图像分析、机器学习。 Casitmed2022@163.com
基金资助:
国家自然科学基金资助项目(82073338);四川省科技计划项目重点研发项目(2022YFS0217)

Abstract

Abstract:

Multi-modal medical images can provide clinicians with rich information of target areas （such as tumors， organs or tissues）. However， effective fusion and segmentation of multi-modal images is still a challenging problem due to the independence and complementarity of multi-modal images. Traditional image fusion methods have difficulty in addressing this problem， leading to widespread research on deep learning-based multi-modal medical image segmentation algorithms. The multi-modal medical image segmentation task based on deep learning was reviewed in terms of principles， techniques， problems， and prospects. Firstly， the general theory of deep learning and multi-modal medical image segmentation was introduced， including the basic principles and development processes of deep learning and Convolutional Neural Network （CNN）， as well as the importance of the multi-modal medical image segmentation task. Secondly， the key concepts of multi-modal medical image segmentation was described， including data dimension， preprocessing， data enhancement， loss function， and post-processing， etc. Thirdly， different multi-modal segmentation networks based on different fusion strategies were summarized and analyzed. Finally， several common problems in medical image segmentation were discussed， the summary and prospects for future research were given.

Key words: deep learning, multi-modal, medical image, image fusion, image segmentation

摘要：

多模态医学图像可以为临床医生提供靶区（如肿瘤、器官或组织）的丰富信息。然而，由于多模态图像之间相互独立且仅有互补性，如何有效融合多模态图像并进行分割仍是亟待解决的问题。传统的图像融合方法难以有效解决此问题，因此基于深度学习的多模态医学图像分割算法得到了广泛的研究。从原理、技术、问题及展望等方面对基于深度学习的多模态医学图像分割任务进行了综述。首先，介绍了深度学习与多模态医学图像分割的一般理论，包括深度学习与卷积神经网络（CNN）的基本原理与发展历程，以及多模态医学图像分割任务的重要性；其次，介绍了多模态医学图像分割的关键概念，包括数据维度、预处理、数据增强、损失函数以及后处理等；接着，对基于不同融合策略的多模态分割网络进行综述，对不同方式的融合策略进行分析；最后，对医学图像分割过程中常见的几个问题进行探讨，并对今后研究作了总结与展望。

关键词: 深度学习, 多模态, 医学图像, 图像融合, 图像分割

CLC Number:

TP183

Meng DOU, Zhebin CHEN, Xin WANG, Jitao ZHOU, Yu YAO. Review of multi-modal medical image segmentation based on deep learning[J]. Journal of Computer Applications, 2023, 43(11): 3385-3395.

窦猛, 陈哲彬, 王辛, 周继陶, 姚宇. 基于深度学习的多模态医学图像分割综述[J]. 《计算机应用》唯一官方网站, 2023, 43(11): 3385-3395.

Figures/Tables 16

References 80

1	范立南，王晶，王银玲.多模态医学图像融合技术综述［J］.沈阳大学学报，2009，21（3）：96-100. 10.3969/j.issn.1008-9225.2009.03.027
	FAN L N， WANG J， WANG Y L. A summary of multimodality medical image fusion technology［J］. Journal of Shenyang University， 2009， 21（3）： 96-100. 10.3969/j.issn.1008-9225.2009.03.027
2	邹诚实，岑婷.腹部CT及MRI图像融合配准在临床中的应用［J］.上海医学影像，2002，11（4）：250-253. 10.3969/j.issn.1008-617X.2002.04.003
	ZOU C S， CEN T. Abdomen CT and MRI image fusion technique and its clinical application ［J］. Shanghai Medical Imaging， 2002，11（4）： 250-253. 10.3969/j.issn.1008-617X.2002.04.003
3	GUO Z， LI X， HUANG H， et al. Deep learning-based image segmentation on multimodal medical imaging ［J］. IEEE Transactions on Radiation and Plasma Medical Sciences， 2019， 3（2）： 162-169. 10.1109/trpms.2018.2890359
4	KHALEGHI B， KHAMIS A， KARRAY F O， et al. Multisensor data fusion： a review of the state-of-the-art ［J］. Information Fusion， 2013， 14（1）： 28-44. 10.1016/j.inffus.2011.08.001
5	赵澄东，王旭辉，杨静宇.一种基于改进的概率图像融合方法［J］.计算机工程与应用，2012，48（8）：182-184. 10.3778/j.issn.1002-8331.2012.08.052
	ZHAO C D， WANG X H， YANG J Y. Method of image fusion based on improved probability theory ［J］. Computer Engineering and Applications， 2012， 48（8）： 182-184. 10.3778/j.issn.1002-8331.2012.08.052
6	DAS S， KUNDU MK. A neuro-fuzzy approach for medical image fusion ［J］. IEEE Transactions on Biomedical Engineering， 2013， 60（12）： 3347-3353. 10.1109/tbme.2013.2282461
7	BALASUBRAMANIAM P， ANANTHI V P. Image fusion using intuitionistic fuzzy sets ［J］. Information Fusion， 2014， 20： 21-30. 10.1016/j.inffus.2013.10.011
8	SMETS P. The combination of evidence in the transferable belief model ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 1990， 12（5）： 447-458. 10.1109/34.55104
9	LIAN C， RUAN S， DENŒUX T， et al. Joint tumor segmentation in PET-CT images using co-clustering and fusion based on belief functions ［J］. IEEE Transactions on Image Processing， 2018， 28（2）： 755-766. 10.1109/tip.2018.2872908
10	VÁZQUEZ-REINA A， GELBART M， HUANG D， et al. Segmentation fusion for connectomics ［C］// Proceedings of the 2011 International Conference on Computer Vision. Piscataway： IEEE， 2011： 177-184. 10.1109/iccv.2011.6126240
11	肖儿良，周莹，简献忠.迁移学习与GAN结合的医学图像融合模型［J］.小型微型计算机系统，2020，41（9）：1972-1978. 10.3969/j.issn.1000-1220.2020.09.029
	XIAO E L， ZHOU Y， JIAN X Z. Medical image fusion model combining transfer learning and generative adversarial network ［J］. Journal of Chinese Computer Systems，2020，41（9）：1972-1978. 10.3969/j.issn.1000-1220.2020.09.029
12	李鸿，邹俊颖，谭茜成，等.面向医学图像分割的多注意力融合网络［J］.计算机应用，2022，42（12）：3891-3899. 10.11772/j.issn.1001-9081.2021101737
	LI H， ZOU J Y， TAN X C， et al. Multi-attention fusion network for medical image segmentation ［J］. Journal of Computer Applications， 2022， 42（12）： 3891-3899. 10.11772/j.issn.1001-9081.2021101737
13	ZHANG N， RUAN S， LEBONVALLET S， et al. Kernel feature selection to fuse multi-spectral MRI images for brain tumor segmentation ［J］. Computer Vision and Image Understanding， 2011， 115（2）： 256-269. 10.1016/j.cviu.2010.09.007
14	KRIZHEVSKY A， SUTSKEVER I， HINTON G E. ImageNet classification with deep convolutional neural networks ［J］. Communications of the ACM， 2017， 60（6）： 84-90. 10.1145/3065386
15	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition ［EB/OL］. ［2023-04-24］. .
16	SZEGEDY C， LIU W， JIA Y， et al. Going deeper with convolutions ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Washington， DC： IEEE Computer Society， 2015： 1-9. 10.1109/cvpr.2015.7298594
17	HE K， ZHANG X， REN S， et al. Identity mappings in deep residual networks ［C］// Proceedings of the 2016 European Conference on Computer Vision， LNIP 9908. Cham：Springer， 2016： 630-645.
18	HUANG G， LIU Z， L van der MAATEN， et al. Densely connected convolutional networks ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Washington， DC： IEEE Computer Society， 2017： 2261-2269. 10.1109/cvpr.2017.243
19	LONG J， SHELHAMER E， DARRELL T. Fully convolutional networks for semantic segmentation ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Washington， DC： IEEE Computer Society， 2015： 3431-3440. 10.1109/cvpr.2015.7298965
20	RONNEBERGER O， FISCHER P， BROX T. U-Net： convolutional networks for biomedical image segmentation ［C］// Proceedings of 18th International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9351. Cham： Springer， 2015： 234-241.
21	MENZE B H， JAKAB A， BAUER S， et al. The multimodal brain tumor image segmentation benchmark （BRATS）［J］. IEEE Transactions on Medical Imaging， 2015， 34（10）： 1993-2024. 10.1109/tmi.2014.2377694
22	PEREIRA S， PINTO A， ALVES V， et al. Brain tumor segmentation using convolutional neural networks in MRI images ［J］. IEEE Transactions on Medical Imaging， 2016， 35（5）： 1240-1251. 10.1109/tmi.2016.2538465
23	HAVAEI M， DAVY A， WARDE-FARLEY D， et al. Brain tumor segmentation with deep neural networks ［J］. Medical Image Analysis， 2017， 35： 18-31. 10.1016/j.media.2016.05.004
24	KALINOVSKY A， KOVALEV V. Lung image segmentation using deep learning methods and convolutional neural networks ［C］// Proceedings of the 13th International Conference on Pattern Recognition and Information Processing. Berlin： Springer， 2016： 21-24.
25	ROTH H R， LU L， FARAG A， et al. DeepOrgan： multi-level deep convolutional networks for automated pancreas segmentation ［C］// Proceedings of 18th International Conference on Medical Image Computing and Computer-Assisted Intervention， LNIP 9349. Cham： Springer， 2015： 556-564.
26	FU M， WU W， HONG X， et al. Hierarchical combinatorial deep learning architecture for pancreas segmentation of medical computed tomography cancer images ［J］. BMC Systems Biology， 2018， 12： No.56. 10.1186/s12918-018-0572-z
27	YU L， YANG X， CHEN H， et al. Volumetric ConvNets with mixed residual connections for automated prostate segmentation from 3D MR images ［C］// Proceedings of the 31st AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2017： 66-72. 10.1609/aaai.v31i1.10510
28	ZHOU X， TAKAYAMA R， WANG S， et al. Deep learning of the sectional appearances of 3D CT images for anatomical structure segmentation based on an FCN voting method ［J］. Medical Physics， 2017， 44（10）： 5221-5233. 10.1002/mp.12480
29	STOYANOV D， TAYLOR Z， CARNEIRO G， et al. Deep learning in medical image analysis and multimodal learning for clinical decision support ［C］// Proceedings of the 4th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， DLMIA 2018， and 8th International Workshop， ML-CDS 2018， Held in Conjunction with MICCAI 2018， LNCS 11045. Cham： Springer， 2018： 3-11.
30	ZHOU T， RUAN S， CANU S. A review： deep learning for medical image segmentation using multi-modality fusion ［J］. Array， 2019，（3/4）： No.100004. 10.1016/j.array.2019.100004
31	HATAMIZADEH A， NATH V， TANG Y， et al. Swin UNETR： Swin transformers for semantic segmentation of brain tumors in MRI images ［C］// Proceedings of the 7th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2021， Held in Conjunction with MICCAI 2021， LNCS 12962. Cham： Springer， 2022： 272-284. 10.1007/978-3-031-08999-2_22
32	MENDRIK A M， VINCKEN K L， KUIJF H J， et al. MRBrainS challenge： online evaluation framework for brain image segmentation in 3T MRI scans ［J］. Computational Intelligence and Neuroscience， 2015， 2015： No.813696. 10.1155/2015/813696
33	IŠGUM I， BENDERS M J N L， AVANTS B， et al. Evaluation of automatic neonatal brain segmentation algorithms： the NeoBrainS12 challenge ［J］. Medical Image Analysis， 2015， 20（1）： 135-151. 10.1016/j.media.2014.11.001
34	WANG L， NIE D， LI G， et al. Benchmark on automatic six-month-old infant brain segmentation algorithms： the iSeg-2017 challenge ［J］. IEEE Transactions on Medical Imaging， 2019， 38（9）： 2219-2230. 10.1109/tmi.2019.2901712
35	DOLZ J， GOPINATH K， YUAN J， et al. HyperDense-Net： a hyper-densely connected CNN for multi-modal image segmentation ［J］. IEEE Transactions on Medical Imaging， 2019， 38（5）： 1116-1126. 10.1109/tmi.2018.2878669
36	CUI S， MAO L， JIANG J， et al. Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network ［J］. Journal of Healthcare Engineering， 2018， 2018： No.4940593. 10.1155/2018/4940593
37	DOLZ J， DESROSIERS C， AYED I BEN. IVD-Net： intervertebral disc localization and segmentation in MRI with a multi-modal UNet ［C］// Proceedings of 5th International Workshop and Challenge， CSI 2018， Held in Conjunction with MICCAI 2018， LNCS 11397. Cham： Springer， 2019： 130-143. 10.1007/978-3-030-13736-6_11
38	NIE D， WANG L， GAO Y， et al. Fully convolutional networks for multi-modality isointense infant brain image segmentation［C］// Proceedings of 2016 IEEE International Symposium on Biomedical Imaging. Piscataway： IEEE， 2016： 1342-1345. 10.1109/isbi.2016.7493515
39	WANG G， LI W， OURSELIN S， et al. Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks ［C］// Proceedings of the Third International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2017， Held in Conjunction with MICCAI 2017， LNCS 10670. Cham： Springer， 2018： 178-190. 10.1007/978-3-319-75238-9_16
40	KAMNITSAS K， LEDIG C， NEWCOMBE V F J， et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation ［J］. Medical Image Analysis， 2017， 36： 61-78. 10.1016/j.media.2016.10.004
41	ZHAO X， WU Y， SONG G， et al. A deep learning model integrating FCNNs and CRFs for brain tumor segmentation ［J］. Medical Image Analysis， 2018， 43： 98-111. 10.1016/j.media.2017.10.002
42	MLYNARSKI P， DELINGETTE H， CRIMINISI A， et al. 3D convolutional neural networks for tumor segmentation using long-range 2D context ［J］. Computerized Medical Imaging and Graphics， 2019， 73： 60-72. 10.1016/j.compmedimag.2019.02.001
43	WANG J， PEREZ L. The effectiveness of data augmentation in image classification using deep learning ［EB/OL］. ［2022-05-09］. .
44	ISENSEE F， KICKINGEREDER P， WICK W， et al. Brain tumor segmentation and radiomics survival prediction： Contribution to the BraTS 2017 challenge ［C］// Proceedings of the Third International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2017， Held in Conjunction with MICCAI 2017， LNCS 10670. Cham： Springer， 2018： 287-297. 10.1007/978-3-319-75238-9_25
45	ISENSEE F， KICKINGEREDER P， WICK W， et al. No new-net ［C］// Proceedings of the 4th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2018， Held in Conjunction with MICCAI 2018， LNCS 11384. Cham： Springer， 2019： 234-244. 10.1007/978-3-030-11726-9_21
46	MYRONENKO A. 3D MRI brain tumor segmentation using autoencoder regularization ［C］// Proceedings of the 4th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2018， Held in Conjunction with MICCAI 2018， LNCS 11384. Cham： Springer， 2019： 311-320. 10.1007/978-3-030-11726-9_28
47	ZHOU C， DING C， WANG X， et al. One-pass multi-task networks with cross-task guided attention for brain tumor segmentation ［J］. IEEE Transactions on Image Processing， 2020， 29： 4516-4529. 10.1109/tip.2020.2973510
48	ZHAO C， ZHAO Z， ZENG Q， et al. MVP U-Net： multi-view pointwise U-net for brain tumor segmentation ［C］// Proceedings of the 6th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2020， Held in Conjunction with MICCAI 2020， LNCS 12659. Cham： Springer， 2021： 93-103. 10.1007/978-3-030-72087-2_9
49	DING Y， ZHENG W， GENG J， et al. MVFusFra： a multi-view dynamic fusion framework for multimodal brain tumor segmentation ［J］. IEEE Journal of Biomedical and Health Informatics， 2021， 26（4）： 1570-1581. 10.1109/jbhi.2021.3122328
50	QIN Y， KAMNITSAS K， ANCHA S， et al. Autofocus layer for semantic segmentation ［C］// Proceedings of 21th International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 11072. Cham： Springer， 2018： 603-611.
51	LIN F， WU Q， LIU J， et al. Path aggregation U-Net model for brain tumor segmentation ［J］. Multimedia Tools and Applications， 2021， 80（15）： 22951-22964. 10.1007/s11042-020-08795-9
52	GOODFELLOW I， POUGET-ABADIE J， MIRZA M， et al. Generative adversarial networks ［J］. Communications of the ACM， 2020， 63（11）： 139-144. 10.1145/3422622
53	YANG H-Y， YANG J. Automatic brain tumor segmentation with contour aware residual network and adversarial training［C］// Proceedings of the 4th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2018， Held in Conjunction with MICCAI 2018， LNCS 11384. Cham： Springer， 2019： 267-278. 10.1007/978-3-030-11726-9_24
54	HUO Y， XU Z， BAO S， et al. Splenomegaly segmentation on multi-modal MRI using deep convolutional networks ［J］. IEEE Transactions on Medical Imaging， 2019， 38（5）： 1185-1196. 10.1109/tmi.2018.2881110
55	ISOLA P， ZHU J Y， ZHOU T， et al. Image-to-image translation with conditional adversarial networks ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Washington， DC： IEEE Computer Society， 2017： 5967-5976. 10.1109/cvpr.2017.632
56	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need ［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 6000-6010.
57	DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An image is worth 16x16 words： Transformers for image recognition at scale ［EB/OL］. ［2023-04-24］. .
58	LIU Z， LIN Y， CAO Y， et al. Swin Transformer： hierarchical vision Transformer using shifted windows ［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2021： 10012-10022. 10.1109/iccv48922.2021.00986
59	JIA Q， SHU H. BiTr-Unet： a CNN-Transformer combined network for MRI brain tumor segmentation ［C］// Proceedings of the 7th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2021， Held in Conjunction with MICCAI 2021， LNCS 12963. Cham： Springer， 2022： 3-14. 10.1007/978-3-031-09002-8_1
60	CHEN J， LU Y， YU Q， et al. TransUnet： Transformers make strong encoders for medical image segmentation ［EB/OL］. ［2023-04-25］. . 10.48550/arXiv.2102.04306
61	WANG W， CHEN C， DING M， et al. TransBTS： multimodal brain tumor segmentation using transformer ［C］// Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention， LNIP 12901. Cham： Springer， 2021： 109-119.
62	HATAMIZADEH A， NATH V， TANG Y， et al. Swin UNETR： Swin Transformers for semantic segmentation of brain tumors in MRI images ［C］// Proceedings of the 7th International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2021， Held in Conjunction with MICCAI 2021， LNCS 12962. Cham： Springer， 2022： 272-284. 10.1007/978-3-031-08999-2_22
63	JIANG Y， ZHANG Y， LIN X， et al. SwinBTS： a method for 3D multimodal brain tumor segmentation using Swin Transformer ［J］. Brain Sciences， 2022， 12（6）： No.797. 10.3390/brainsci12060797
64	CHEN L， WU Y， DSOUZA A M， et al. MRI tumor segmentation with densely connected 3D CNN ［C］// Proceedings of the 2018 Medical Imaging： Image Processing， SPIE 10574. Bellingham， WA： SPIE， 2018： 357-364.
65	CHEN Y， CHEN J， WEI D， et al. OctopusNet： a deep learning segmentation network for multi-modal medical images ［C］// Proceedings of the First International Workshop on Multiscale Multimodal Medical Imaging， LNCS 11977. Cham： Springer， 2020： 17-25.
66	YUE M， DAI Z， HE J， et al. MRI-guided automated delineation of gross tumor volume for nasopharyngeal carcinoma using deep learning［C］// Proceedings of the 35th IEEE International Symposium on Computer-Based Medical Systems. Piscataway： IEEE， 2022： 292-296. 10.1109/cbms55023.2022.00058
67	LI C， SUN H， LIU Z， et al. Learning cross-modal deep representations for multi-modal MR image segmentation ［C］// Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention， LNCS 11765. Cham： Springer， 2019： 57-65.
68	ZHANG Y， YANG J， TIAN J， et al. Modality-aware mutual learning for multi-modal medical image segmentation ［C］// Proceedings of the 24th International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 12901. Cham： Springer， 2021： 589-599.
69	DOU Q， LIU Q， HENG P A， et al. Unpaired multi-modal segmentation via knowledge distillation ［J］. IEEE Transactions on Medical Imaging， 2020， 39（7）： 2415-2425. 10.1109/tmi.2019.2963882
70	WANG H， ZHANG J， CHEN Y， et al. Uncertainty-aware multi-modal learning via cross-modal random network prediction［C］// Proceedings of the 17th European Conference on Computer Vision， LNCS 13697. Cham： Springer， 2022： 200-217.
71	ROKACH L. Ensemble-based classifiers ［J］. Artificial Intelligence Review， 2010， 33（1/2）： 1-39. 10.1007/s10462-009-9124-7
72	KAMNITSAS K， BAI W， FERRANTE E， et al. Ensembles of multiple models and architectures for robust brain tumour segmentation ［C］// Proceedings of the Third International Workshop on Brainlesion： Glioma， Multiple Sclerosis， Stroke and Traumatic Brain Injuries， BrainLes 2017， Held in Conjunction with MICCAI 2017， LNCS 10670. Berlin： Springer， 2018： 450-462. 10.1007/978-3-319-75238-9_38
73	AYGÜN M， ŞAHIN Y H， Ünal G. Multi modal convolutional neural networks for brain tumor segmentation ［EB/OL］. ［2023-04-24］. .
74	DOUZAS G， BACAO F. Effective data generation for imbalanced learning using conditional generative adversarial networks ［J］. Expert Systems with Applications， 2018， 91（C）： 464-471. 10.1016/j.eswa.2017.09.030
75	CHAWLA N V， BOWYER K W， HALL L O， et al. SMOTE： synthetic minority over-sampling technique ［J］. Journal of Artificial Intelligence Research， 2002， 16（1）： 321-357. 10.1613/jair.953
76	CLÈRIGUES A， VALVERDE S， BERNAL J， et al. Acute and sub-acute stroke lesion segmentation from multimodal MRI ［J］. Computer Methods and Programs in Biomedicine， 2020， 194： No.105521. 10.1016/j.cmpb.2020.105521
77	HE Y， YANG D， ROTH H， et al. DiNTS： differentiable neural network topology search for 3D medical image segmentation ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington， DC： IEEE Computer Society， 2021： 5841-5850. 10.1109/cvpr46437.2021.00578
78	CHANG P D. Fully convolutional neural networks with hyperlocal features for brain tumor segmentation ［C］// Proceedings of the 19th International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin： Springer， 2016： 4-9.
79	ZHOU T， RUAN S， VERA P， et al. A tri-attention fusion guided multi-modal segmentation network ［J］. Pattern Recognition， 2022， 124（C）： No.108417. 10.1016/j.patcog.2021.108417
80	ZHANG D， HUANG G， ZHANG Q， et al. Cross-modality deep feature learning for brain tumor segmentation ［J］. Pattern Recognition， 2021， 110： No.107562. 10.1016/j.patcog.2020.107562

数据集	图像数		分割任务	图像种类
数据集	训练集	测试集	分割任务	图像种类
BraTS 2012	35	15	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2013	35	25	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2014	200	38	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2015	200	53	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2016	200	191	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2017	285	146	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2018	285	191	脑肿瘤	T1w， T1ce， T2w， Flair
ISLES 2015	28	36	缺血性中风病变	T1w， T2w， TSE， Flair， DWI， TFE/TSE
MRBrainS13	5	15	脑组织	T1w， T1_IR， Flair
NeoBrainS12	20	5	脑组织	T1w， T2w
Iseg-2017	10	13	脑组织	T1w， T2w
CHAOS	20	20	腹部器官	CT， T1-DUAL， T2-SPIR
IVDM3Seg	16	8	椎间盘	In-phase， Opposed-phase， Fat， Water

数据集	图像数		分割任务	图像种类
数据集	训练集	测试集	分割任务	图像种类
BraTS 2012	35	15	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2013	35	25	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2014	200	38	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2015	200	53	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2016	200	191	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2017	285	146	脑肿瘤	T1w， T1ce， T2w， Flair
BraTS 2018	285	191	脑肿瘤	T1w， T1ce， T2w， Flair
ISLES 2015	28	36	缺血性中风病变	T1w， T2w， TSE， Flair， DWI， TFE/TSE
MRBrainS13	5	15	脑组织	T1w， T1_IR， Flair
NeoBrainS12	20	5	脑组织	T1w， T2w
Iseg-2017	10	13	脑组织	T1w， T2w
CHAOS	20	20	腹部器官	CT， T1-DUAL， T2-SPIR
IVDM3Seg	16	8	椎间盘	In-phase， Opposed-phase， Fat， Water

方法来源	数据集	数据	网络结构	融合方式	分割精度（DICE）
文献［22］	BraTS2013	3D	CNN	输入级融合	ET：0.770； WT：0.880； TC：0.830
文献［40］	BraTS2015	3D patch	CNN+CRF	输入级融合	ET：0.728； WT：0.901； TC：0.754
文献［78］	BraTS2016	2D	FCN	输入级融合	ET：0.720； WT：0.870； TC：0.810
文献［72］	BraTS2017	3D	U-Net/FCN	输入级融合	ET：0.729； WT：0.886； TC：0.785
文献［46］	BraTS2018	3D	U-Net+VAE	输入级融合	ET：0.766； WT：0.884； TC：0.815
文献［67］	BraTS2018	2D slice	CNN+Attention	中间级融合	ET：0.734； WT：0.834； TC：0.783
文献［79］	BraTS2018	3D patch	U-Net+Attention	中间级融合	ET：0.688； WT：0.876； TC：0.784
文献［80］	BraTS2018	2D slice	U-Net+GAN	决策级融合	ET：0.831； WT：0.873； TC：0.656
文献［61］	BraTS2020	3D patch	U-Net+Transformer	输入级融合	ET：0.787； WT：0.901； TC：0.817
文献［35］	MRBrainS13	3D patch	CNN+DenseNet	中间级融合	CSF：0.834； WM：0.895； GM：0.863
文献［37］	IVD	2D slice	U-Net+DenseNet	中间级融合	0.919 ± 0.018

方法来源	数据集	数据	网络结构	融合方式	分割精度（DICE）
文献［22］	BraTS2013	3D	CNN	输入级融合	ET：0.770； WT：0.880； TC：0.830
文献［40］	BraTS2015	3D patch	CNN+CRF	输入级融合	ET：0.728； WT：0.901； TC：0.754
文献［78］	BraTS2016	2D	FCN	输入级融合	ET：0.720； WT：0.870； TC：0.810
文献［72］	BraTS2017	3D	U-Net/FCN	输入级融合	ET：0.729； WT：0.886； TC：0.785
文献［46］	BraTS2018	3D	U-Net+VAE	输入级融合	ET：0.766； WT：0.884； TC：0.815
文献［67］	BraTS2018	2D slice	CNN+Attention	中间级融合	ET：0.734； WT：0.834； TC：0.783
文献［79］	BraTS2018	3D patch	U-Net+Attention	中间级融合	ET：0.688； WT：0.876； TC：0.784
文献［80］	BraTS2018	2D slice	U-Net+GAN	决策级融合	ET：0.831； WT：0.873； TC：0.656
文献［61］	BraTS2020	3D patch	U-Net+Transformer	输入级融合	ET：0.787； WT：0.901； TC：0.817
文献［35］	MRBrainS13	3D patch	CNN+DenseNet	中间级融合	CSF：0.834； WM：0.895； GM：0.863
文献［37］	IVD	2D slice	U-Net+DenseNet	中间级融合	0.919 ± 0.018

[1]	Ziyi HE, Yan YANG, Yiling ZHANG. Multi-view clustering network with deep fusion [J]. Journal of Computer Applications, 2023, 43(9): 2651-2656.
[2]	Juntao CHEN, Ziqi ZHU. Image copy-move forgery detection based on multi-scale feature extraction and fusion [J]. Journal of Computer Applications, 2023, 43(9): 2919-2924.
[3]	Xiaolin LI, Songjia YANG. Hybrid beamforming for multi-user mmWave relay networks using deep learning [J]. Journal of Computer Applications, 2023, 43(8): 2511-2516.
[4]	Xiang GUO, Wengang JIANG, Yuhang WANG. Encrypted traffic classification method based on improved Inception-ResNet [J]. Journal of Computer Applications, 2023, 43(8): 2471-2476.
[5]	Yumeng CUI, Jingya WANG, Xiaowen LIU, Shangyi YAN, Zhizhong TAO. General text classification model combining attention and cropping mechanism [J]. Journal of Computer Applications, 2023, 43(8): 2396-2405.
[6]	Kun ZHANG, Fengyu YANG, Fa ZHONG, Guangdong ZENG, Shijian ZHOU. Source code vulnerability detection based on hybrid code representation [J]. Journal of Computer Applications, 2023, 43(8): 2517-2526.
[7]	Yi WANG, Jie XIE, Jia CHENG, Liwei DOU. Review of object pose estimation in RGB images based on deep learning [J]. Journal of Computer Applications, 2023, 43(8): 2546-2555.
[8]	Yuxin TUO, Tao XUE. Joint triple extraction model combining pointer network and relational embedding [J]. Journal of Computer Applications, 2023, 43(7): 2116-2124.
[9]	Jiaming HE, Jucheng YANG, Chao WU, Xiaoning YAN, Nenghua XU. Person re-identification method based on multi-modal graph convolutional neural network [J]. Journal of Computer Applications, 2023, 43(7): 2182-2189.
[10]	Min LIANG, Jiayi LIU, Jie LI. Image super-resolution reconstruction method based on iterative feedback and attention mechanism [J]. Journal of Computer Applications, 2023, 43(7): 2280-2287.
[11]	Kunpei YE, Xi XIONG, Zhe DING. Recruitment recommendation model based on field fusion and time weight [J]. Journal of Computer Applications, 2023, 43(7): 2133-2139.
[12]	Libin CEN, Jingdong LI, Chunbo LIN, Xiaoling WANG. Approximate query processing approach based on deep autoregressive model [J]. Journal of Computer Applications, 2023, 43(7): 2034-2039.
[13]	Shuai ZHENG, Xiaolong ZHANG, He DENG, Hongwei REN. 3D liver image segmentation method based on multi-scale feature fusion and grid attention mechanism [J]. Journal of Computer Applications, 2023, 43(7): 2303-2310.
[14]	Runting DONG, Li WU, Xiaoying WANG, Tengfei CAO, Jianqiang HUANG, Qin GUAN, Jiexia WU. Review of application analysis and research progress of deep learning in weather forecasting [J]. Journal of Computer Applications, 2023, 43(6): 1958-1968.
[15]	Yichi CHEN, Bin CHEN. Review of lifelong learning in computer vision [J]. Journal of Computer Applications, 2023, 43(6): 1785-1795.

Review of multi-modal medical image segmentation based on deep learning

基于深度学习的多模态医学图像分割综述

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 80

Related Articles 15

Recommended Articles

Metrics