3D liver image segmentation method based on multi-scale feature fusion and grid attention mechanism

doi:10.11772/j.issn.1001-9081.2022060803

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (7): 2303-2310.DOI: 10.11772/j.issn.1001-9081.2022060803

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

3D liver image segmentation method based on multi-scale feature fusion and grid attention mechanism

Shuai ZHENG¹^,²^,³, Xiaolong ZHANG¹^,²^,³(), He DENG¹^,²^,³, Hongwei REN⁴

^1.School of Computer Science and Technology，Wuhan University of Science and Technology，Wuhan Hubei 430065，China
^2.Institute of Big Data Science and Engineering，Wuhan University of Science and Technology，Wuhan Hubei 430065，China
^3.Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System （Wuhan University of Science and Technology），Wuhan Hubei 430065，China
^4.Tianyou Hospital Affiliated to Wuhan University of Science and Technology，Wuhan Hubei 430064，China

Received:2022-06-06 Revised:2022-08-04 Accepted:2022-08-11 Online:2022-08-26 Published:2023-07-10
Contact: Xiaolong ZHANG
About author:ZHENG Shuai， born in 1998， M. S. candidate. His research interests include computer vision， deep learning.
ZHANG Xiaolong， born in 1963， Ph. D.， professor. His research interests include artificial intelligence， machine learning， data mining， biological information processing.
DENG He， born in 1977， Ph. D.， professor. His research interests include deep learning， image processing.
REN Hongwei， born in 1978， M. S.， deputy chief physician. Her research interests include imaging diagnosis of digestive system， central nervous system and musculoskeletal system.
Supported by:
National Natural Science Foundation of China(61972299)

基于多尺度特征融合和网格注意力机制的三维肝脏影像分割方法

郑帅¹^,²^,³, 张晓龙¹^,²^,³(), 邓鹤¹^,²^,³, 任宏伟⁴

^1.武汉科技大学计算机科学与技术学院, 武汉 430065
^2.武汉科技大学大数据科学与工程研究院, 武汉 430065
^3.智能信息处理与实时工业系统湖北省重点实验室(武汉科技大学), 武汉 430065
^4.武汉科技大学附属天佑医院, 武汉 430064

通讯作者: 张晓龙
作者简介:郑帅（1998—），男，湖北孝感人，硕士研究生，主要研究方向：计算机视觉、深度学习；
张晓龙（1963—），男，江西吉安人，教授，博士，主要研究方向：人工智能、机器学习、数据挖掘、生物信息处理；
邓鹤（1977—），湖北孝感人，教授，博士，主要研究方向：深度学习、图像处理；
任宏伟（1978—），女，河北滦城人，副主任医师，硕士，主要研究方向：消化系统、中枢神经系统及骨肌系统的影像学诊断。
基金资助:
国家自然科学基金资助项目(61972299)

Abstract

Abstract:

Due to the high similarity of gray values among liver and adjacent organs in Computed Tomography （CT） and Magnetic Resonance Imaging （MRI） images， a 3D liver image segmentation method based on multi-scale feature fusion and grid attention mechanism， namely MAGNet （Multi-scale feature fusion And Grid attention mechanism Network）， was proposed to segment liver automatically and accurately. Firstly， high-level features and low-level features were connected by the attention-guided concatenation module to extract important context information， and the grid attention mechanism was introduced in the attention-guided concatenation module to focus on the segmentation region of interest. Then， the multi-scale feature fusion module was formed by the layered connection in a single feature map according to the number of channels， and this module was used to replace the basic convolutional block to obtain multi-scale semantic information. Finally， the deep supervision mechanism was utilized to solve the problems of vanishing gradient， exploding gradient and slow convergence. Experimental results show that on 3DIRCADb dataset， compared with the U³-Net+DC method， MAGNet improves the Dice Similarity Coefficient （DSC） metric by 0.10 percentage points and reduces the Relative Volume Difference （RVD） metric by 1.97 percentage points； on Sliver07 dataset， compared with the CANet method， MAGNet improves the DSC metrics by 0.30 percentage points， reduces Volumetric Overlap Error （VOE） metrics by 0.68 percentage points， and reduces the Average Symmetric Surface Distance （ASD） and Root Mean Square Symmetric Surface Distance （RMSD） metrics 0.03 mm and 0.22 mm respectively； on the liver MRI dataset of a hospital， MAGNet also has good results on all metrics. Besides， MAGNet was applied to a mixed dataset of 3DIRCADb dataset and the hospital liver MRI dataset above， and a competitive segmentation result was also achieved.

Key words: 3D liver medical image, semantic segmentation, deep learning, multi-scale feature fusion, attention mechanism

摘要：

在计算机断层扫描（CT）和磁共振成像（MRI）的影像中肝脏与邻近脏器的灰度值相似性都比较高，为自动精确地分割肝脏，提出一种基于多尺度特征融合和网格注意力机制的三维肝脏影像分割方法MAGNet （Multi-scale feature fusion And Grid attention mechanism Network）。首先，通过注意力引导连接模块来连接高层特征和低层特征以提取出重要的上下文信息，并且在注意力引导连接模块中引入网格注意力机制来关注感兴趣的分割区域；然后，通过在单个特征图中按通道数进行分层连接形成多尺度特征融合模块，并用该模块替换基础卷积块以获取多尺度语义信息；最后，利用深度监督机制解决梯度消失、梯度爆炸和收敛过慢等问题。实验结果表明：在3DIRCADb数据集上，与U³-Net+DC方法相比，MAGNet在Dice相似系数（DSC）指标上提升了0.10个百分点，在相对体积差（RVD）指标上降低了1.97个百分点；在Sliver07数据集上，与CANet方法相比，MAGNet在DSC指标上提升了0.30个百分点，在体素重叠误差（VOE）指标上降低了0.68个百分点，在平均对称表面距离（ASD）和对称位置表面距离的均方根（RMSD）指标上分别降低了0.03 mm和0.22 mm；在某医院肝脏MRI数据集上，MAGNet在所有指标上也均具有良好的结果。另外，将MAGNet应用于3DIRCADb数据集和某医院肝脏MRI数据集进行混合形成的数据集，也取得了非常有竞争力的分割效果。

关键词: 三维肝脏医疗影像, 语义分割, 深度学习, 多尺度特征融合, 注意力机制

CLC Number:

TP391.4

Shuai ZHENG, Xiaolong ZHANG, He DENG, Hongwei REN. 3D liver image segmentation method based on multi-scale feature fusion and grid attention mechanism[J]. Journal of Computer Applications, 2023, 43(7): 2303-2310.

郑帅, 张晓龙, 邓鹤, 任宏伟. 基于多尺度特征融合和网格注意力机制的三维肝脏影像分割方法[J]. 《计算机应用》唯一官方网站, 2023, 43(7): 2303-2310.

Figures/Tables 11

References 27

1	MOGHBEL M， MASHOHOR S， MAHMUD R， et al. Review of liver segmentation and computer assisted detection/diagnosis methods in computed tomography［J］. Artificial Intelligence Review， 2018， 50（4）： 497-537. 10.1007/s10462-017-9550-x
2	MASOUMI H， BEHRAD A， POURMINA M A， et al. Automatic liver segmentation in MRI images using an iterative watershed algorithm and artificial neural network［J］. Biomedical Signal Processing and Control， 2012， 7（5）： 429-437. 10.1016/j.bspc.2012.01.002
3	JIN Q G， MENG Z P， SUN C M， et al. RA-UNet： a hybrid deep attention-aware network to extract liver and tumor in CT scans［J］. Frontiers in Bioengineering and Biotechnology， 2020， 8： No.605132. 10.3389/fbioe.2020.605132
4	高琳，罗晓辉，何立新.水平集方法在CT肝脏图像分割中的应用［J］.计算机工程与应用， 2005， 41（36）： 201-202， 232. 10.3321/j.issn:1002-8331.2005.36.064
	GAO L， LUO X H， HE L X. Application of level-set method in CT liver image segmentation［J］. Computer Engineering and Applications， 2005， 41（36）： 201-202， 232. 10.3321/j.issn:1002-8331.2005.36.064
5	郑睿，陈雷霆，房春兰，等.基于水平集方法的软组织图像序列分割［J］.计算机工程与设计， 2007， 28（15）： 3629-3631， 3726. 10.3969/j.issn.1000-7024.2007.15.028
	ZHENG R， CHEN L T， FANG C L， et al. Soft tissue image sequences segmentation based on level set method［J］. Computer Engineering and Design， 2007， 28（15）： 3629-3631， 3726. 10.3969/j.issn.1000-7024.2007.15.028
6	FU M， XU P， LI X D， et al. Fast crowd density estimation with convolutional neural networks［J］. Engineering Applications of Artificial Intelligence， 2015， 43： 81-88. 10.1016/j.engappai.2015.04.006
7	LONG J， SHELHAMER E， DARRELL T. Fully convolutional networks for semantic segmentation ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 3431-3440. 10.1109/cvpr.2015.7298965
8	RONNEBERGER O， FISCHER P， BROX T. U-Net： convolutional networks for biomedical image segmentation ［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9351. Cham： Springer， 2015： 234-241.
9	ZHOU Z W， RAHMAN SIDDIQUEE M M， TAJBAKHSH N， et al. UNet++： a nested U-Net architecture for medical image segmentation ［C］// Proceedings of the 2018 International Workshop on Deep Learning in Medical Image Analysis/ International Workshop on Multimodal Learning for Clinical Decision Support， LNCS 11045. Cham： Springer， 2018： 3-11.
10	FU J， LIU J， TIAN H J， et al. Dual attention network for scene segmentation ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3141-3149. 10.1109/cvpr.2019.00326
11	刘哲，张晓林，宋余庆，等.结合改进的U-Net和Morphsnakes的肝脏分割［J］.中国图象图形学报， 2018， 23（8）： 1254-1262.
	LIU Z， ZHANG X L， SONG Y Q， et al. Liver segmentation with improved U-Net and Morphsnakes algorithm［J］. Journal of Image and Graphics， 2018， 23（8）： 1254-1262.
12	LI X M， CHEN H， QI X J， et al. H-DenseUNet： hybrid densely connected UNet for liver and tumor segmentation from CT volumes［J］. IEEE Transactions on Medical Imaging， 2018， 37（12）： 2663-2674. 10.1109/tmi.2018.2845918
13	CHEN Y L， WANG K， LIAO X Y， et al. Channel-UNet： a spatial channel-wise convolutional neural network for liver and tumors segmentation［J］. Frontiers in Genetics， 2019， 10： No.1110. 10.3389/fgene.2019.01110
14	SZEGEDY C， VANHOUCKE V， IOFFE S， et al. Rethinking the inception architecture for computer vision ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 2818-2826. 10.1109/cvpr.2016.308
15	YAN Q S， WANG B， ZHANG W， et al. Attention-guided deep neural network with multi-scale feature fusion for liver vessel segmentation［J］. IEEE Journal of Biomedical and Health Informatics， 2021， 25（7）： 2629-2642. 10.1109/jbhi.2020.3042069
16	SALEHI S S M， ERDOGMUS D， GHOLIPOUR A. Tversky loss function for image segmentation using 3D fully convolutional deep networks ［C］// Proceedings of the 2017 International Workshop on Machine Learning in Medical Imaging， LNCS 10541. Cham： Springer， 2017： 379-387.
17	CHLEBUS G， MEINE H， MOLTZ J H， et al. Neural network-based automatic liver tumor segmentation with random forest-based candidate filtering［EB/OL］. （2017-06-27）［2021-12-15］. . 10.1038/s41598-018-33860-7
18	HAN X. Automatic liver lesion segmentation using a deep convolutional neural network method［EB/OL］. （2017-04-24）［2021-12-17］. .
19	CHRIST P F， ELSHAER M E A， ETTLINGER F， et al. Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields ［C］// Proceedings of the 2016 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9901. Cham： Springer， 2016： 415-423.
20	TRAN S T， CHENG C H， LIU D G. A multiple layer U-Net， Uⁿ-Net， for liver and liver tumor segmentation in CT［J］. IEEE Access， 2021， 9： 3752-3764. 10.1109/access.2020.3047861
21	LU F， WU F， HU P J， et al. Automatic 3D liver location and segmentation via convolutional neural network and graph cut［J］. International Journal of Computer Assisted Radiology and Surgery， 2017， 12（2）： 171-182. 10.1007/s11548-016-1467-3
22	AL-SHAIKHLI S D S， YANG M Y， ROSENHAHN B. Automatic 3D liver segmentation using sparse representation of global and local image information via level set formulation［EB/OL］. （2015-10-04）［2021-12-25］. . 10.1016/j.cmpb.2016.09.007
23	DONG C H， CHEN Y W， FORUZAN A H， et al. Segmentation of liver and spleen based on computational anatomy models［J］. Computers in Biology and Medicine， 2015， 67： 146-160. 10.1016/j.compbiomed.2015.10.007
24	HU P J， WU F， PENG J L， et al. Automatic 3D liver segmentation based on deep learning and globally optimized surface evolution［J］. Physics in Medicine and Biology， 2016， 61（24）： No.8676. 10.1088/1361-6560/61/24/8676
25	LU X S， XIE Q L， ZHA Y F， et al. Fully automatic liver segmentation combining multi-dimensional graph cut with shape information in 3D CT images［J］. Scientific Reports， 2018， 8： No.10700. 10.1038/s41598-018-28787-y
26	GUO X T， SCHWARTZ L H， ZHAO B S. Automatic liver segmentation by integrating fully convolutional networks into active contour models［J］. Medical Physics， 2019， 46（10）： 4455-4469. 10.1002/mp.13735
27	SHAO S， ZHANG X L， CHENG R Q， et al. Semantic segmentation method of 3D liver image based on contextual attention model ［C］// Proceedings of the 2021 IEEE International Conference on Systems， Man， and Cybernetics. Piscataway： IEEE， 2021： 3042-3049. 10.1109/smc52423.2021.9659018

方法	VOE/%	RVD/%	ASD/mm	RMSD/mm	DSC
UNet ^［17］	14.21 ± 5.71	0.05 ± 0.10	4.33 ± 3.39	8.35 ± 7.54	0.923 ± 0.03
ResNet^［18］	11.65 ± 4.06	0.03 ± 0.06	3.91 ± 3.95	8.11 ± 9.68	0.938 ± 0.02
Christ方法^［19］	10.7	1.4	1.5	24.0	0.943
H-DenseUNet^［12］	10.20 ± 3.44	0.01 ± 0.05	4.06 ± 3.85	9.63 ± 10.41	0.947 ± 0.01
Channel-Unet^［13］	9.52 ± 4.65	0.02 ± 0.07	8.43 ± 9.39	14.21 ± 5.71	0.946 ± 0.03
U³-Net+DC^［20］	6.14	1.98	—	—	0.964
本文方法	6.59 ± 1.97	0.01 ± 0.04	1.13 ± 0.34	2.68 ± 1.39	0.965 ± 0.01

方法	VOE/%	RVD/%	ASD/mm	RMSD/mm	DSC
UNet ^［17］	14.21 ± 5.71	0.05 ± 0.10	4.33 ± 3.39	8.35 ± 7.54	0.923 ± 0.03
ResNet^［18］	11.65 ± 4.06	0.03 ± 0.06	3.91 ± 3.95	8.11 ± 9.68	0.938 ± 0.02
Christ方法^［19］	10.7	1.4	1.5	24.0	0.943
H-DenseUNet^［12］	10.20 ± 3.44	0.01 ± 0.05	4.06 ± 3.85	9.63 ± 10.41	0.947 ± 0.01
Channel-Unet^［13］	9.52 ± 4.65	0.02 ± 0.07	8.43 ± 9.39	14.21 ± 5.71	0.946 ± 0.03
U³-Net+DC^［20］	6.14	1.98	—	—	0.964
本文方法	6.59 ± 1.97	0.01 ± 0.04	1.13 ± 0.34	2.68 ± 1.39	0.965 ± 0.01

方法	图片序列数	VOE/%	RVD/%	ASD/mm	RMSD/mm	DSC
Lu方法^［21］	78	5.90	2.70	0.91	1.88	0.969
Al-Shaikhli方法^［22］	20	6.44	1.53	0.95	1.58	0.966
Dong方法^［23］	38	6.44	0.01	0.98	1.87	0.966
Hu方法^［24］	109	5.35	-0.17	0.84	1.78	0.972
Lu方法^［25］	40	5.92	1.03	1.06	1.68	—
Guo方法^［26］	77	—	—	1.40	2.30	0.962
CANet^［27］	111	5.89	0.01	1.00	2.57	0.970
本文方法	111	5.21	0.01	0.97	2.35	0.973

方法	图片序列数	VOE/%	RVD/%	ASD/mm	RMSD/mm	DSC
Lu方法^［21］	78	5.90	2.70	0.91	1.88	0.969
Al-Shaikhli方法^［22］	20	6.44	1.53	0.95	1.58	0.966
Dong方法^［23］	38	6.44	0.01	0.98	1.87	0.966
Hu方法^［24］	109	5.35	-0.17	0.84	1.78	0.972
Lu方法^［25］	40	5.92	1.03	1.06	1.68	—
Guo方法^［26］	77	—	—	1.40	2.30	0.962
CANet^［27］	111	5.89	0.01	1.00	2.57	0.970
本文方法	111	5.21	0.01	0.97	2.35	0.973

方法	VOE/%	RVD/%	ASD/mm	RMSD/mm	DSC
3D U-Net	13.23±7.89	0.05±0.13	3.76±3.05	7.23±6.34	0.924±0.04
3D U-Net+MFF	8.10±3.52	0.02±0.12	1.48±0.99	3.78±3.17	0.957±0.01
3D U-Net+GA+AGC	8.73±5.13	0.01±0.08	1.91±2.18	4.70±4.96	0.953±0.03
本文方法	7.04±1.97	0.01±0.04	1.13±0.34	2.68±1.39	0.965±0.01

3D liver image segmentation method based on multi-scale feature fusion and grid attention mechanism

基于多尺度特征融合和网格注意力机制的三维肝脏影像分割方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 27

Related Articles 15

Recommended Articles

Metrics

数据集	方法	VOE/%	RVD/%	ASD/mm	RMSD/mm	DSC
某医院肝脏MRI数据集	3D U-Net	8.96±3.46	0.03±0.08	0.97±0.38	2.40±0.67	0.952±0.01
	3D U-Net+MFF	6.66±2.68	0.01±0.06	0.67±0.27	1.92±0.68	0.965±0.01
	3D U-Net+GA+AGC	6.40±2.39	0.01±0.05	0.65±0.16	1.95±0.60	0.966±0.01
	本文方法	6.17±2.08	0.01±0.02	0.64±0.20	1.90±0.60	0.968±0.01
CT与MRI混合数据集	3D U-Net	11.91±7.89	0.06±0.14	1.40±0.80	3.35±1.50	0.934±0.05
	3D U-Net+MFF	10.33±5.98	0.04±0.12	1.23±0.78	3.06±1.73	0.944±0.03
	3D U-Net+GA+AGC	11.03±4.43	0.03±0.08	1.35±0.55	3.13±1.00	0.941±0.02
	本文方法	9.19±3.35	0.02±0.04	1.08±0.39	2.55±0.55	0.951±0.01

[1]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[2]	Xiyuan WANG, Zhancheng ZHANG, Shaokang XU, Baocheng ZHANG, Xiaoqing LUO, Fuyuan HU. Unsupervised cross-domain transfer network for 3D/2D registration in surgical navigation [J]. Journal of Computer Applications, 2024, 44(9): 2911-2918.
[3]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[4]	Yexin PAN, Zhe YANG. Optimization model for small object detection based on multi-level feature bidirectional fusion [J]. Journal of Computer Applications, 2024, 44(9): 2871-2877.
[5]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[6]	Shunyong LI, Shiyi LI, Rui XU, Xingwang ZHAO. Incomplete multi-view clustering algorithm based on self-attention fusion [J]. Journal of Computer Applications, 2024, 44(9): 2696-2703.
[7]	Yunchuan HUANG, Yongquan JIANG, Juntao HUANG, Yan YANG. Molecular toxicity prediction based on meta graph isomorphism network [J]. Journal of Computer Applications, 2024, 44(9): 2964-2969.
[8]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.
[9]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[10]	Yuhan LIU, Genlin JI, Hongping ZHANG. Video pedestrian anomaly detection method based on skeleton graph and mixed attention [J]. Journal of Computer Applications, 2024, 44(8): 2551-2557.
[11]	Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594.
[12]	Shangbin MO, Wenjun WANG, Ling DONG, Shengxiang GAO, Zhengtao YU. Single-channel speech enhancement based on multi-channel information aggregation and collaborative decoding [J]. Journal of Computer Applications, 2024, 44(8): 2611-2617.
[13]	Yanjie GU, Yingjun ZHANG, Xiaoqian LIU, Wei ZHOU, Wei SUN. Traffic flow forecasting via spatial-temporal multi-graph fusion [J]. Journal of Computer Applications, 2024, 44(8): 2618-2625.
[14]	Qianhong SHI, Yan YANG, Yongquan JIANG, Xiaocao OUYANG, Wubo FAN, Qiang CHEN, Tao JIANG, Yuan LI. Multi-granularity abrupt change fitting network for air quality prediction [J]. Journal of Computer Applications, 2024, 44(8): 2643-2650.
[15]	Zheng WU, Zhiyou CHENG, Zhentian WANG, Chuanjian WANG, Sheng WANG, Hui XU. Deep learning-based classification of head movement amplitude during patient anaesthesia resuscitation [J]. Journal of Computer Applications, 2024, 44(7): 2258-2263.