Transformer based U-shaped medical image segmentation network： a survey

doi:10.11772/j.issn.1001-9081.2022040530

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (5): 1584-1595.DOI: 10.11772/j.issn.1001-9081.2022040530

• Multimedia computing and computer simulation • Previous Articles Next Articles

Transformer based U-shaped medical image segmentation network： a survey

Liyao FU¹, Mengxiao YIN¹^,², Feng YANG¹^,²()

^1.School of Computer，Electronics and Information，Guangxi University，Nanning Guangxi 530004，China
^2.Guangxi Key Laboratory of Multimedia Communications and Network Technology （Guangxi University），Nanning Guangxi 530004，China

Received:2022-04-18 Revised:2022-07-02 Accepted:2022-07-04 Online:2022-07-26 Published:2023-05-10
Contact: Feng YANG
About author:FU Liyao， born in 1998， M. S. candidate. Her research interests include computer vision， medical image segmentation.
YIN Mengxiao， born in 1978， Ph. D.， associate professor. Her research interests include computer graphics and virtual reality， digital geometry processing， image and video editing， graph theory and its applications.
YANG Feng， born in 1979， Ph. D.， associate professor. His research interests include artificial intelligence， network information security， big data and high-performance computing， precision medicine.
Supported by:
National Natural Science Foundation of China(61861004)

基于Transformer的U型医学图像分割网络综述

傅励瑶¹, 尹梦晓¹^,², 杨锋¹^,²()

^1.广西大学计算机与电子信息学院，南宁 530004
^2.广西多媒体通信与网络技术重点实验室（广西大学），南宁 530004

通讯作者: 杨锋
作者简介:傅励瑶（1998—），女，重庆人，硕士研究生，主要研究方向：计算机视觉、医学图像分割
尹梦晓（1978—），女，河南南阳人，副教授，博士，CCF会员，主要研究方向：计算机图形学与虚拟现实、数字几何处理、图像与视频编辑、图论及其应用
杨锋（1979—），男，广西玉林人，副教授，博士，CCF会员，主要研究方向：人工智能、网络信息安全、大数据与高性能计算、精准医学。fyang@foxmail.com
基金资助:
国家自然科学基金资助项目(61861004)

Abstract

Abstract:

U-shaped Network （U-Net） based on Fully Convolutional Network （FCN） is widely used as the backbone of medical image segmentation models， but Convolutional Neural Network （CNN） is not good at capturing long-range dependency， which limits the further performance improvement of segmentation models. To solve the above problem， researchers have applied Transformer to medical image segmentation models to make up for the deficiency of CNN， and U-shaped segmentation networks combining Transformer have become the hot research topics. After a detailed introduction of U-Net and Transformer， the related medical image segmentation models were categorized by the position in which the Transformer module was located， including only in the encoder or decoder， both in the encoder and decoder， as a skip-connection， and others， the basic contents， design concepts and possible improvement aspects about these models were discussed， the advantages and disadvantages of having Transformer in different positions were also analyzed. According to the analysis results， it can be seen that the biggest factor to decide the position of Transformer is the characteristics of the target segmentation task， and the segmentation models of Transformer combined with U-Net can make better use of the advantages of CNN and Transformer to improve segmentation performance of models， which has great development prospect and research value.

Key words: deep learning, Convolutional Neural Network (CNN), medical image segmentation, U-shaped Network (U-Net), Transformer

摘要：

目前，医学图像分割模型广泛采用基于全卷积网络（FCN）的U型网络（U-Net）作为骨干网，但卷积神经网络（CNN）在捕捉长距离依赖能力上的劣势限制了分割模型性能的进一步提升。针对上述问题，研究者们将Transformer应用到医学图像分割模型中以弥补CNN的不足，结合Transformer和U型结构的分割网络成为研究热点之一。在详细介绍U-Net和Transformer之后，按医学图像分割模型中Transformer模块所处的位置，包括仅在编码器或解码器、同时在编码器和解码器、作为过渡连接和其他位置进行分类，讨论各模型的基本内容、设计理念以及可改进的地方，并分析了Transformer处于不同位置的优缺点。根据分析结果可知，决定Transformer所在位置的最大因素是目标分割任务的特点，而且Transformer结合U-Net的分割模型能更好地利用CNN和Transformer各自的优势，提高模型的分割性能，具有较大的发展前景和研究价值。

关键词: 深度学习, 卷积神经网络, 医学图像分割, U型网络, Transformer

CLC Number:

TP183

Liyao FU, Mengxiao YIN, Feng YANG. Transformer based U-shaped medical image segmentation network： a survey[J]. Journal of Computer Applications, 2023, 43(5): 1584-1595.

傅励瑶, 尹梦晓, 杨锋. 基于Transformer的U型医学图像分割网络综述[J]. 《计算机应用》唯一官方网站, 2023, 43(5): 1584-1595.

Figures/Tables 6

References 95

1	LONG J， SHELHAMER E， DARRELL T. Fully convolutional networks for semantic segmentation［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 3431-3440. 10.1109/cvpr.2015.7298965
2	RONNEBERGER O， FISCHER P， BROX T. U-net： convolutional networks for biomedical image segmentation［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9351. Cham： Springer， 2015： 234-241.
3	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017：6000-6010.
4	DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An image is worth 16x16 words： Transformers for image recognition at scale［EB/OL］. （2021-06-03）［2022-03-22］. .
5	ZHENG S X， LU J C， ZHAO H S， et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 6877-6886. 10.1109/cvpr46437.2021.00681
6	CARION N， MASSA F， SYNNAEVE G， et al. End-to-end object detection with transformers［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12346. Cham： Springer， 2020： 213-229.
7	CHEN C， LIU Q D， JIN Y M， et al. Source-free domain adaptive fundus image segmentation with denoised pseudo-labeling［C］// Proceedings of the 2021 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 12905. Cham： Springer， 2021： 225-235.
8	XIE Y T， ZHANG J P， XIA Y， et al. Unified 2D and 3D pre-training for medical image classification and segmentation［EB/OL］. ［2021-12-17］. .
9	ZHOU B， LIU C， DUNCAN J S. Anatomy-constrained contrastive learning for synthetic segmentation without ground-truth［C］// Proceedings of the 2021 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 12901. Cham： Springer， 2021： 47-56.
10	ZHOU Z W， SHIN J， ZHANG L， et al. Fine-tuning convolutional neural networks for biomedical image analysis： actively and incrementally［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 4761-4772. 10.1109/cvpr.2017.506
11	LUO X D， CHEN J N， SONG T， et al. Semi-supervised medical image segmentation through dual-task consistency［C］// Proceedings of the 35th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2021： 8801-8809. 10.1609/aaai.v35i10.17066
12	HONG S M， BONKHOFF A， HOOPES A， et al. Hypernet-ensemble learning of segmentation probability for medical image segmentation with ambiguous labels［EB/OL］. （2021-12-13）［2022-03-24］. .
13	YAO H F， HU X W， LI X M. Enhancing pseudo label quality for semi-supervised domain-generalized medical image segmentation［C］// Proceedings of the 36th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2022： 3099-3107. 10.1609/aaai.v36i3.20217
14	LUO X D， HU M H， SONG T， et al. Semi-supervised medical image segmentation via cross teaching between CNN and Transformer［C］// Proceedings of the 5th International Conference on Medical Imaging with Deep Learning. New York： JMLR.org， 2022： 820-833.
15	XU G， WU X， ZHANG X， et al. LeVit-UNet： make faster encoders with transformer for medical image segmentation［EB/OL］. （2021-07-19）［2022-03-22］. . 10.2139/ssrn.4116174
16	HUANG X H， DENG Z F， LI D D， et al. MISSFormer： an effective medical image segmentation transformer［EB/OL］. （2021-12-19）［2022-03-20］. . 10.1109/tmi.2022.3230943
17	朱锴，付忠良，陈晓清. 基于卷积神经网络的超声图像左心室分割方法［J］. 计算机应用， 2019， 39（7）：2121-2124. 10.11772/j.issn.1001-9081.2018112321
	ZHU K， FU Z L， CHEN X Q. Left ventricular segmentation method of ultrasound image based on convolutional neural network［J］. Journal of Computer Applications， 2019， 39（7）：2121-2124. 10.11772/j.issn.1001-9081.2018112321
18	王雪. 基于U-Net多尺度和多维度特征融合的皮肤病变分割方法［J］.吉林大学学报（理学版）， 2021， 59（1）： 123-127.
	WANG X. Skin lesion segmentation method based on U-Net with multi-scale and multi-dimensional feature fusion［J］. Journal of Jilin University （Science Edition）， 2021， 59（1）：123-127.
19	董阳，潘海为，崔倩娜，等. 面向多模态磁共振脑瘤图像的小样本分割方法［J］. 计算机应用， 2021， 41（4）：1049-1054. 10.11772/j.issn.1001-9081.2020081388
	DONG Y， PAN H W， CUI Q N， et al. Few-shot segmentation method for multi-modal magnetic resonance images of brain tumor［J］. Journal of Computer Applications， 2021， 41（4）：1049-1054. 10.11772/j.issn.1001-9081.2020081388
20	CHEN J N， LU Y Y， YU Q H， et al. TransUNet： Transformers make strong encoders for medical image segmentation［EB/OL］. （2021-02-08）［2022-02-10］. .
21	YAO C， HU M H， ZHAI G T， et al. TransClaw U-Net： Claw U-Net with transformers for medical image segmentation［EB/OL］. （2021-07-12）［2022-02-10］. . 10.1109/icicsp55539.2022.10050624
22	VALANARASU J M J， OZA P， HACIHALILOGLU I， et al. Medical transformer： gated axial-attention for medical image segmentation［C］// Proceedings of the 2021 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 12901. Cham： Springer， 2021： 36-46.
23	ZHOU Z W， RAHMAN SIDDIQUEE M M， TAJBAKHSH N， et al. UNet++： a nested U-Net architecture for medical image segmentation［C］// Proceedings of the 2018 International Workshop on Deep Learning in Medical Image Analysis and the 2018 International Workshop on Multimodal Learning for Clinical Decision Support， LNCS 11045. Cham： Springer， 2018： 3-11.
24	HUANG G， LIU Z， VAN DER MAATEN L， et al. Densely connected convolutional networks［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2261-2269. 10.1109/cvpr.2017.243
25	ZHANG Z X， LIU Q J， WANG Y H. Road extraction by deep residual U-Net［J］. IEEE Geoscience and Remote Sensing Letters， 2018， 15（5）： 749-753. 10.1109/lgrs.2018.2802944
26	HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
27	JHA D， SMEDSRUD P H， RIEGLER M A， et al. ResUNet++： an advanced architecture for medical image segmentation［C］// Proceedings of the 2019 IEEE International Symposium on Multimedia. Piscataway： IEEE， 2019： 225-230. 10.1109/ism46123.2019.00049
28	HU J， SHEN L， SUN G. Squeeze-and-excitation networks［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7132-7141. 10.1109/cvpr.2018.00745
29	MILLETARI F， NAVAB N， AHMADI S A. V-Net： fully convolutional neural networks for volumetric medical image segmentation［C］// Proceedings of the 4th International Conference on 3D Vision. Piscataway： IEEE， 2016： 565-571. 10.1109/3dv.2016.79
30	ÇIÇEK Ö， ABDULKADIR A， LIENKAMP S S， et al. 3D U-Net： learning dense volumetric segmentation from sparse annotation［C］// Proceedings of the 2016 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9901. Cham： Springer， 2016： 424-432.
31	ISENSEE F， JAEGER P F， KOHL S A A， et al. Automated design of deep learning methods for biomedical image segmentation［EB/OL］. （2020-04-02）［2022-01-24］. . 10.1038/s41592-020-01008-z
32	BA J L， KIROS J R， HINTON G E. Layer normalization［EB/OL］. （2016-07-21）［2022-02-11］. .
33	IOFFE S， SZEGEDY C. Batch normalization： accelerating deep network training by reducing internal covariate shift［C］// Proceedings of the 32nd International Conference on Machine Learning. New York： JMLR.org， 2015： 448-456.
34	LIU Z， LIN Y T， CAO Y， et al. Swin Transformer： hierarchical vision transformer using shifted windows［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2021： 9992-10002. 10.1109/iccv48922.2021.00986
35	HO J， KALCHBRENNER N， WEISSENBORN D， et al. Axial attention in multidimensional transformers［EB/OL］. （2019-12-20）［2022-02-11］. .
36	WANG H Y， ZHU Y K， GREEN B， et al. Axial-DeepLab： stand-alone axial-attention for panoptic segmentation［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12349. Cham： Springer， 2020： 108-126. 10.1007/978-3-030-58548-8_7
37	LI Y Y， WANG Z Y， YIN L， et al. X-Net： a dual encoding- decoding method in medical image segmentation［J/OL］. The Visual Computer （2021-11-05）［2022-02-21］. . 10.1007/s00371-021-02328-7
38	SZEGEDY C， LIU W， JIA Y Q， et al. Going deeper with convolutions［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 1-9. 10.1109/cvpr.2015.7298594
39	LIANG J J， YANG C H， ZENG M J， et al. TransConver： transformer and convolution parallel network for developing automatic brain tumor segmentation in MRI images［J］. Quantitative Imaging in Medicine and Surgery， 2022， 12（4）： 2397-2415. 10.21037/qims-21-919
40	GRAHAM B， EL-NOUBY A， TOUVRON H， et al. LeViT： a vision Transformer in ConvNet’s clothing for faster inference［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2021： 12259-12269. 10.1109/iccv48922.2021.01204
41	ZHANG Y D， LIU H Y， HU Q. TransFuse： fusing Transformers and CNNs for medical image segmentation［C］// Proceedings of the 2021 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 12901. Cham： Springer， 2021： 14-24.
42	TOUVRON H， CORD M， DOUZE M， et al. Training data-efficient image transformers & distillation through attention［C］// Proceedings of the 38th International Conference on Machine Learning. New York： JMLR.org， 2021： 10347-10357. 10.1109/iccv48922.2021.00091
43	TANG Y C， YANG D， LI W Q， et al. Self-supervised pre-training of Swin Transformers for 3D medical image analysis［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2022： 12259-12269. 10.1109/cvpr52688.2022.02007
44	LI S H， SUI X C， LUO X D， et al. Medical image segmentation using squeeze-and-expansion transformers［C］// Proceedings of the 30th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2021： 807-815. 10.24963/ijcai.2021/112
45	LEE J， LEE Y， KIM J， et al. Set Transformer： a framework for attention-based permutation-invariant neural networks［C］// Proceedings of the 36th International Conference on Machine Learning. New York： JMLR.org， 2019：3744-3753.
46	ZHOU H Y， GUO J S， ZHANG Y H， et al. nnFormer： interleaved transformer for volumetric segmentation［EB/OL］. （2022-02-04）［2022-02-21］. .
47	WU Y X， LIAO K L， CHEN J T， et al. D-Former： a U-shaped dilated transformer for 3D medical image segmentation［J］. Neural Computing and Applications， 2023， 35（2）： 1931-1944. 10.1007/s00521-022-07859-1
48	CAO H， WANG Y Y， CHEN J， et al. Swin-Unet： Unet-like pure Transformer for medical image segmentation［EB/OL］. （2021-05-12）［2022-02-21］. . 10.1007/978-3-031-25066-8_9
49	WANG H Y， XIE S A， LIN L F， et al. Mixed Transformer U-Net for medical image segmentation［C］// Proceedings of the 2022 IEEE International Conference on Acoustics， Speech and Signal Processing. Piscataway： IEEE， 2022： 2390-2394. 10.1109/icassp43922.2022.9746172
50	GUO M H， LIU Z N， MU T J， et al. Beyond self-attention： external attention using two linear layers for visual tasks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022（Early Access）： 1-13. 10.1109/tpami.2022.3211006
51	CHEN D， YANG W Z， WANG L J， et al. PCAT-UNet： UNet-like network fused convolution and Transformer for retinal vessel segmentation［J］. PLoS ONE， 2022， 17（1）： No.e0262689. 10.1371/journal.pone.0262689
52	LUO X D， HU M H， SONG T， et al. Semi-supervised medical image segmentation via cross teaching between CNN and Transformer［C］// Proceedings of the 5th International Conference on Medical Imaging with Deep Learning. New York： JMLR.org， 2022： 820-833.
53	LIN A L， CHEN B Z， XU J Y， et al. DS-TransUNet： dual Swin Transformer U-Net for medical image segmentation［J］. IEEE Transactions on Instrumentation and Measurement， 2022， 71： No.4005615. 10.1109/tim.2022.3178991
54	GUO D F， TERZOPOULOS D. A transformer-based network for anisotropic 3D medical image segmentation［C］// Proceedings of the 25th International Conference on Pattern Recognition. Piscataway： IEEE， 2021： 8857-8861. 10.1109/icpr48806.2021.9411990
55	LUO C， ZHANG J， CHEN X L， et al. UCATR： based on CNN and Transformer encoding and cross-attention decoding for lesion segmentation of acute ischemic stroke in non-contrast computed tomography images［C］// Proceedings of the 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway： IEEE， 2021： 3565-3568. 10.1109/embc46164.2021.9630336
56	WANG W X， CHEN C， DING M， et al. TransBTS： multimodal brain tumor segmentation using Transformer［C］// Proceedings of the 2021 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 12901. Cham： Springer， 2021： 109-119.
57	WANG B， WANG F， DONG P W， et al. Multiscale TransUNet++： dense hybrid U-Net with transformer for medical image segmentation［J］. Signal， Image and Video Processing， 2022， 16（6）： 1607-1614. 10.1007/s11760-021-02115-w
58	LIN T Y， GOYAL P， GIRSHICK R， et al. Focal loss for dense object detection［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2999-3007. 10.1109/iccv.2017.324
59	WANG Z， SIMONCELLI E P， BOVIK A C. Multiscale structural similarity for image quality assessment［C］// Proceedings of the 37th Asilomar Conference on Signals， Systems and Computers — Volume 2. Piscataway： IEEE， 2003： 1398-1402. 10.1109/acssc.2003.1292181
60	YU J H， JIANG Y N， WANG Z Y， et al. UnitBox： an advanced object detection network［C］// Proceedings of the 24th ACM International Conference on Multimedia. New York： ACM， 2016： 516-520. 10.1145/2964284.2967274
61	LI J Y， WANG W X， CHEN C， et al. TransBTSV2： wider instead of deeper transformer for medical image segmentation［EB/OL］. （2022-05-17）［2022-03-10］. .
62	LIU Y Y， YANG Y， JIANG W， et al. 3D deep attentive U-Net with transformer for breast tumor segmentation from automated breast volume scanner［C］// Proceedings of the 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway： IEEE， 2021： 4011-4014. 10.1109/embc46164.2021.9629523
63	ZHANG J X， JIANG Z K， DONG J， et al. Attention gate ResU-Net for automatic MRI brain tumor segmentation［J］. IEEE Access， 2020， 8： 58533-58545. 10.1109/access.2020.2983075
64	ZHANG Y L， HIGASHITA R， FU H Z， et al. A multi-branch hybrid transformer network for corneal endothelial cell segmentation［C］// Proceedings of the 2021 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 12901. Cham： Springer， 2021： 99-108.
65	YAN X Y， TANG H， SUN S L， et al. AFTer-UNet： axial fusion transformer UNet for medical image segmentation［C］// Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2022： 3270-3280. 10.1109/wacv51458.2022.00333
66	XIE Y T， ZHANG J P， SHEN C H， et al. CoTr： efficiently bridging CNN and Transformer for 3D medical image segmentation［C］// Proceedings of the 2021 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 12903. Cham： Springer， 2021： 171-180.
67	ZHU X Z， SU W J， LU L W， et al. Deformable DETR： deformable Transformers for end-to-end object detection［EB/OL］. （2021-03-18）［2022-03-10］. . 10.1609/aaai.v36i1.19893
68	JI Y F， ZHANG R M， WANG H J， et al. Multi-compound transformer for accurate biomedical image segmentation［C］// Proceedings of the 2021 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 12901. Cham： Springer， 2021： 326-336.
69	CHEN B Z， LIU Y S， LI Y J， et al. TransAttUnet： multi-level attention-guided U-Net with Transformer for medical image segmentation［EB/OL］. （2022-07-09）［2022-02-21］. .
70	MA M J， XIA H Y， TAN Y M， et al. HT-Net： hierarchical context-attention transformer network for medical CT image segmentation［J］. Applied Intelligence， 2022， 52（9）： 10692-10705. 10.1007/s10489-021-03010-0
71	HE K M， ZHANG X Y， REN S Q， et al. Spatial pyramid pooling in deep convolutional networks for visual recognition［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2015， 37（9）： 1904-1916. 10.1109/tpami.2015.2389824
72	HUANG S Q， LI J N， XIAO Y Z， et al. RTNet： relation Transformer network for diabetic retinopathy multi-lesion segmentation［J］. IEEE Transactions on Medical Imaging， 2022， 41（6）： 1596-1607. 10.1109/tmi.2022.3143833
73	LANDMAN B， XU Z， IGELSIAS J， et al. MICCAI multi-atlas labeling beyond the cranial vault — workshop and challenge［C］// Proceedings of the 2015 International MICCAI Brainlesion Workshop. Cham： Springer， 2015， 5： 12.
74	BERNARD O， LALANDE A， ZOTTI C， et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis： is the problem solved？［J］. IEEE Transactions on Medical Imaging， 2018， 37（11）： 2514-2525. 10.1109/tmi.2018.2837502
75	VALANARASU J M J， YASARLA R， WANG P Y， et al. Learning to segment brain anatomy from 2D ultrasound with less data［J］. IEEE Journal of Selected Topics in Signal Processing， 2020， 14（6）： 1221-1234. 10.1109/jstsp.2020.3001513
76	SIRINUKUNWATTANA K， PLUIM J P W， CHEN H， et al. Gland segmentation in colon histology images： the GLaS challenge contest［J］. Medical Image Analysis， 2017， 35： 489-502. 10.1016/j.media.2016.08.008
77	CAICEDO J C， GOODMAN A， KARHOHS K W， et al. Nucleus segmentation across imaging experiments： the 2018 Data Science Bowl［J］. Nature Methods， 2019， 16（12）： 1247-1253. 10.1038/s41592-019-0612-7
78	NAYLOR P， LAÉ M， REYAL F， et al. Segmentation of nuclei in histopathology images by deep regression of the distance map［J］. IEEE Transactions on Medical Imaging， 2019， 38（2）： 448-459. 10.1109/tmi.2018.2865709
79	WENINGER L， RIPPEL O， KOPPERS S， et al. Segmentation of brain tumors and patient survival prediction： methods for the BraTS 2018 challenge［C］// Proceedings of the 2018 International MICCAI Brainlesion Workshop， LNCS 11384. Cham： Springer， 2019： 3-12.
80	ZHAO Y X， ZHANG Y M， LIU C L. Bag of tricks for 3D MRI brain tumor segmentation［C］// Proceedings of the 2019 International MICCAI Brainlesion Workshop， LNCS 11992. Cham： Springer， 2020： 210-220.
81	JHA D， SMEDSRUD P H， RIEGLER M A， et al. Kvasir-SEG： a segmented polyp dataset［C］// Proceedings of the 2020 International Conference on Multimedia Modeling， LNCS 11962. Cham： Springer， 2020： 451-462.
82	BERSETH M. ISIC 2017 - skin lesion analysis towards melanoma detection［EB/OL］. （2017-03-01）［2022-02-21］. . 10.1109/iesys.2017.8233570
83	STAAL J， ABRÀMOFF M D， NIEMEIJER M， et al. Ridge-based vessel segmentation in color images of the retina［J］. IEEE Transactions on Medical Imaging， 2004， 23（4）： 501-509. 10.1109/tmi.2004.825627
84	HOOVER A D， KOUZNETSOVA V， GOLDBAUM M. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response［J］. IEEE Transactions on Medical imaging， 2000， 19（3）： 203-210. 10.1109/42.845178
85	OWEN C G， RUDNICKA A R， MULLEN R， et al. Measuring retinal vessel tortuosity in 10-year-old children： validation of the Computer-Assisted Image Analysis of the Retina （CAIAR） program［J］. Investigative Ophthalmology and Visual Science， 2009， 50（5）： 2004-2010. 10.1167/iovs.08-3018
86	CODELLA N， ROTEMBERG V， TSCHANDL P， et al. Skin lesion analysis toward melanoma detection 2018： a challenge hosted by the International Skin Imaging Collaboration （ISIC）［EB/OL］. （2019-03-29）［2022-03-23］. .
87	ANTONELLI M， REINKE A， BAKAS S， et al. The medical segmentation decathlon［J］. Nature Communications， 2022， 13： No.4128. 10.1038/s41467-022-30695-9
88	LITJENS G， TOTH R， van de WENDY， et al. Evaluation of prostate segmentation algorithms for MRI： the PROMISE12 challenge［J］. Medical Image Analysis， 2014， 18（2）： 359-373. 10.1016/j.media.2013.12.002
89	BILIC P， CHRIST P， LI H B， et al. The Liver Tumor Segmentation benchmark （LiTS）［J］. Medical Image Analysis， 2023， 84： No. 102680 .
90	RUGGERI A， SCARPA F， DE LUCA M， et al. A system for the automatic estimation of morphometric parameters of corneal endothelium in alizarine red-stained images［J］. British Journal of Ophthalmology， 2010， 94（5）： 643-647. 10.1136/bjo.2009.166561
91	CHEN X M， SUN S L， BAI N， et al. A deep learning-based auto-segmentation system for organs-at-risk on whole-body computed tomography images for radiation therapy［J］. Radiotherapy and Oncology， 2021， 160： 175-184. 10.1016/j.radonc.2021.04.019
92	GAMPER J， ALEMI KOOHBANANI N， BENET K， et al. PanNuke： an open pan-cancer histology dataset for nuclei instance segmentation and classification［C］// Proceedings of the 2019 European Congress on Digital Pathology， LNCS 11435. Cham： Springer， 2019： 11-19.
93	HELLER N， SATHIANATHEN N， KALAPARA A， et al. The KiTS19 challenge data： 300 kidney tumor cases with clinical context， CT semantic segmentations， and surgical outcomes［EB/OL］. （2020-03-15）［2022-03-12］. . 10.1200/jco.2020.38.6_suppl.626
94	PORWAL P， PACHADE S， KOKARE M， et al. IDRiD： diabetic retinopathy - segmentation and grading challenge［J］. Medical Image Analysis， 2019， 59： No.101561. 10.3390/data3030025
95	LI T， GAAO Y Q， WANG K， et al. Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening［J］. Information Sciences， 2019， 501：511-522. 10.1016/j.ins.2019.06.011

Transformer位置	模型	数据集	分割任务
编码器	TransUNet^［20］	BCV^［73］/ACDC^［74］	腹部多器官分割/心脏分割
	MedT^［22］	Brain US^［75］/GLAS^［76］	脑室隔膜分割/腺体分割
	X-Net^［37］	DSB18^［77］/TNBC^［78］	细胞核分割/细胞核分割
	TransConver^［39］	BraTS18^［79］/BraTS19^［80］	脑部肿瘤分割
	TransFuse^［41］	Kvasir^［81］/ISIC2017^［82］	胃肠息肉分割/皮肤病变域分割
	LeViT-UNet^［15］	BCV/ACDC	腹部多器官分割/心脏分割
	TransClaw^［21］	BCV/ACDC	腹部多器官分割/心脏分割
	Swin UNETR^［43］	BCV	腹部多器官分割
解码器	Segtran^［44］	Kvasir/BraTS19	胃肠息肉分割/脑部肿瘤分割
编码器和解码器	nnFormer^［46］	BCV/ACDC	腹部多器官分割/心脏分割
	D-Former^［47］	BCV/ACDC	腹部多器官分割/心脏分割
	MISSFormer^［16］	BCV/ACDC	腹部多器官分割/心脏分割
	Swin UNet^［48］	BCV/ACDC	腹部多器官分割/心脏分割
	MT Net^［49］	BCV/ACDC	腹部多器官分割/心脏分割
	PCAT UNet^［51］	DRIVE^［83］/STARE^［84］/CHASE_DB1^［85］	视网膜血管分割
	文献［54］模型	ACDC	心脏分割
	DS-TransUNet^［53］	Kvasir/GLAS/ISIC2018^［86］	胃肠息肉分割/腺体分割/皮肤病变域分割
过渡连接	UCATR^［55］	The Segmentation Decathlon^［87］	肝癌病变域分割
	TransBTSv1^［56］	BraTS19	脑部肿瘤分割
	MS-TransUNet++^［57］	PROMISE12^［88］/LiTS^［89］	前列腺分割/肝脏肿瘤分割
	TransBTSv2^［61］	BraTS19	脑部肿瘤分割
	文献［62］模型	ABVS	胸部肿瘤分割
	MBT-Net^［64］	Alizarine^［90］	角膜内皮细胞分割
	AFTER-UNet^［65］	BCV/Thorax-85^［91］	腹部多器官分割/胸部多器官分割
	MCTrans^［68］	PanNuke^［92］/ISIC2018	细胞分割/皮肤病变域分割
	TransAtt-UNet^［69］	ISIC2018/GLAS/DSB18	皮肤病变域分割/腺体分割/细胞核分割
跳跃连接	HTNet^［70］	KiTS19^［93］	肾脏肿瘤分割
输出块	RTNet^［72］	IDRiD^［94］/DDR^［95］	眼底病变域分割

Transformer位置	模型	数据集	分割任务
编码器	TransUNet^［20］	BCV^［73］/ACDC^［74］	腹部多器官分割/心脏分割
	MedT^［22］	Brain US^［75］/GLAS^［76］	脑室隔膜分割/腺体分割
	X-Net^［37］	DSB18^［77］/TNBC^［78］	细胞核分割/细胞核分割
	TransConver^［39］	BraTS18^［79］/BraTS19^［80］	脑部肿瘤分割
	TransFuse^［41］	Kvasir^［81］/ISIC2017^［82］	胃肠息肉分割/皮肤病变域分割
	LeViT-UNet^［15］	BCV/ACDC	腹部多器官分割/心脏分割
	TransClaw^［21］	BCV/ACDC	腹部多器官分割/心脏分割
	Swin UNETR^［43］	BCV	腹部多器官分割
解码器	Segtran^［44］	Kvasir/BraTS19	胃肠息肉分割/脑部肿瘤分割
编码器和解码器	nnFormer^［46］	BCV/ACDC	腹部多器官分割/心脏分割
	D-Former^［47］	BCV/ACDC	腹部多器官分割/心脏分割
	MISSFormer^［16］	BCV/ACDC	腹部多器官分割/心脏分割
	Swin UNet^［48］	BCV/ACDC	腹部多器官分割/心脏分割
	MT Net^［49］	BCV/ACDC	腹部多器官分割/心脏分割
	PCAT UNet^［51］	DRIVE^［83］/STARE^［84］/CHASE_DB1^［85］	视网膜血管分割
	文献［54］模型	ACDC	心脏分割
	DS-TransUNet^［53］	Kvasir/GLAS/ISIC2018^［86］	胃肠息肉分割/腺体分割/皮肤病变域分割
过渡连接	UCATR^［55］	The Segmentation Decathlon^［87］	肝癌病变域分割
	TransBTSv1^［56］	BraTS19	脑部肿瘤分割
	MS-TransUNet++^［57］	PROMISE12^［88］/LiTS^［89］	前列腺分割/肝脏肿瘤分割
	TransBTSv2^［61］	BraTS19	脑部肿瘤分割
	文献［62］模型	ABVS	胸部肿瘤分割
	MBT-Net^［64］	Alizarine^［90］	角膜内皮细胞分割
	AFTER-UNet^［65］	BCV/Thorax-85^［91］	腹部多器官分割/胸部多器官分割
	MCTrans^［68］	PanNuke^［92］/ISIC2018	细胞分割/皮肤病变域分割
	TransAtt-UNet^［69］	ISIC2018/GLAS/DSB18	皮肤病变域分割/腺体分割/细胞核分割
跳跃连接	HTNet^［70］	KiTS19^［93］	肾脏肿瘤分割
输出块	RTNet^［72］	IDRiD^［94］/DDR^［95］	眼底病变域分割

[1]	Qinghai XU, Shifei DING, Tongfeng SUN, Jian ZHANG, Lili GUO. Improved capsule network based on multipath feature [J]. Journal of Computer Applications, 2023, 43(5): 1330-1335.
[2]	Rui XU, Shuang LIANG, Hang WAN, Yimin WEN, Shiming SHEN, Jian LI. Extraction of PM_2.5 diffusion characteristics based on candlestick pattern matching [J]. Journal of Computer Applications, 2023, 43(5): 1394-1400.
[3]	Jiahong SUI, Yingchi MAO, Huimin YU, Zicheng WANG, Ping PING. Global image captioning method based on graph attention network [J]. Journal of Computer Applications, 2023, 43(5): 1409-1415.
[4]	Jianhui HE, Chunlong HU, Xin SHU. Multi-task age estimation method based on multi-peak label distribution learning [J]. Journal of Computer Applications, 2023, 43(5): 1578-1583.
[5]	Xianlan WANG, Jinkun ZHOU, Nan MU, Chen WANG. Cross-view geo-localization method based on multi-task joint learning [J]. Journal of Computer Applications, 2023, 43(5): 1625-1635.
[6]	Jinwen GUO, Xinghua MA, Gongning LUO, Wei WANG, Yang CAO, Kuanquan WANG. Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer [J]. Journal of Computer Applications, 2023, 43(5): 1596-1605.
[7]	Yang LIU, Zhiyang LU, Jun WANG, Jun SHI. Gibbs artifact removal algorithm for magnetic resonance imaging based on self-attention connection UNet [J]. Journal of Computer Applications, 2023, 43(5): 1606-1611.
[8]	Haiyu YANG, Wenpu GUO, Kai KANG. Signal modulation recognition method based on convolutional long short-term deep neural network [J]. Journal of Computer Applications, 2023, 43(4): 1318-1322.
[9]	Guangyi DOU, Fanan WEI, Chuangyi QIU, Jianshu CHAO. Tracking appearance features based on attention self-correlation mechanism [J]. Journal of Computer Applications, 2023, 43(4): 1248-1254.
[10]	Jianqing GAO, Yanhui TU, Feng MA, Zhonghua FU. Progressive ratio mask-based adaptive noise estimation method [J]. Journal of Computer Applications, 2023, 43(4): 1303-1308.
[11]	Xu ZHANG, Long SHENG, Haifang ZHANG, Feng TIAN, Wei WANG. Pre-hospital emergency text classification model based on label confusion [J]. Journal of Computer Applications, 2023, 43(4): 1050-1055.
[12]	Xiaoyu FAN, Suzhen LIN, Yanbo WANG, Feng LIU, Dawei LI. Reconstruction algorithm for highly undersampled magnetic resonance images based on residual graph convolutional neural network [J]. Journal of Computer Applications, 2023, 43(4): 1261-1268.
[13]	Zhoubo XU, Puqing CHEN, Huadong LIU, Xin YANG. Deep graph matching model based on self-attention network [J]. Journal of Computer Applications, 2023, 43(4): 1005-1012.
[14]	Yongbing GAO, Juntian GAO, Rong MA, Lidong YANG. User granularity-level personalized social text generation model [J]. Journal of Computer Applications, 2023, 43(4): 1021-1028.
[15]	Cheng FANG, Bei LI, Ping HAN, Qiong WU. Fine-grained emotion classification of Chinese microblog based on syntactic dependency graph [J]. Journal of Computer Applications, 2023, 43(4): 1056-1061.

Transformer based U-shaped medical image segmentation network： a survey

基于Transformer的U型医学图像分割网络综述

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 6

References 95

Related Articles 15

Recommended Articles

Metrics