Medical MRI image super-resolution reconstruction based on multi-receptive field generative adversarial network

doi:10.11772/j.issn.1001-9081.2021040629

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (3): 938-945.DOI: 10.11772/j.issn.1001-9081.2021040629

• Multimedia computing and computer simulation • Previous Articles Next Articles

Medical MRI image super-resolution reconstruction based on multi-receptive field generative adversarial network

Pengwei LIU¹^,²^,³, Yuan GAO¹^,²^,³(), Pinle QIN¹^,²^,³, Zhe YIN¹^,²^,³, Lifang WANG¹^,²^,³

^1.Shanxi Medical Imaging and Data Analysis Engineering Research Center（North University of China），Taiyuan Shanxi 030051，China
^2.School of Data Science and Technology，North University of China，Taiyuan Shanxi 030051，China
^3.Shanxi Medical Imaging Artificial Intelligence Engineering Technology Research Center（North University of China），Taiyuan Shanxi 030051，China

Received:2021-04-21 Revised:2021-06-23 Accepted:2021-06-23 Online:2022-04-09 Published:2022-03-10
Contact: Yuan GAO
About author:LIU Pengwei， born in 1996， M. S. candidate. His research interests include deep learning， computer vision.
QIN Pinle， born in 1978， Ph. D.， professor. His research interests include machine vision， big data processing.
YIN Zhe， born in 1997， M. S. candidate. Her research interests include image fusion， deep learning
WANG Lifang， born in 1977， Ph. D.， associate professor. Her research interests include machine vision， big data processing.
Supported by:
Natural Science Foundation of Shanxi Province(201901D111152)

基于多感受野的生成对抗网络医学MRI影像超分辨率重建

刘朋伟¹^,²^,³, 高媛¹^,²^,³(), 秦品乐¹^,²^,³, 殷喆¹^,²^,³, 王丽芳¹^,²^,³

^1.山西省医学影像与数据分析工程研究中心(中北大学), 太原 030051
^2.中北大学大数据学院, 太原 030051
^3.山西省医学影像人工智能工程技术研究中心(中北大学), 太原 030051

通讯作者: 高媛
作者简介:刘朋伟（1996—），男，陕西咸阳人，硕士研究生，CCF会员，主要研究方向：深度学习、计算机视觉
秦品乐（1978—），男，山西长治人，教授，博士，主要研究方向：机器视觉、大数据处理
殷喆（1997—），女，山西大同人，硕士研究生，主要研究方向：图像融合、深度学习
王丽芳（1977—），女，山西长治人，副教授，博士，主要研究方向：机器视觉、大数据处理。
基金资助:
山西省自然科学基金资助项目(201901D111152)

Abstract

Abstract:

To solve the problems of image detail loss and unclear texture caused by interference factors such as noise， imaging technology and imaging principles in the medical Magnetic Resonance Imaging （MRI） process， a multi-receptive field generative adversarial network for medical MRI image super-resolution reconstruction was proposed. First， the multi-receptive field feature extraction block was used to obtain the global feature information of the image under different receptive fields. In order to avoid the loss of detailed texture due to too small or too large receptive fields， each set of features was divided into two groups， and one of which was used to feedback global feature information under different scales of receptive fields， and the other group was used to enrich the local detailed texture information of the next set of features； then， the multi-receptive field feature extraction block was used to construct feature fusion group， and spatial attention module was added to each feature fusion group to adequately obtain the spatial feature information of the image， reducing the loss of shallow and local features in the network， and achieving a more realistic degree in the details of the image. Secondly， the gradient map of the low-resolution image was converted into the gradient map of the high-resolution image to assist the reconstruction of the super-resolution image. Finally， the restored gradient map was integrated into the super-resolution branch to provide structural prior information for super-resolution reconstruction， which was helpful to generate high quality super-resolution images. The experimental results show that compared with the Structure-Preserving Super-Resolution with gradient guidance （SPSR） algorithm， the proposed algorithm improves the Peak Signal-to-Noise Ratio （PSNR） by 4.8%， 2.7% and 3.5% at ×2， ×3 and ×4 scales， respectively， and the reconstructed medical MRI images have richer texture details and more realistic visual effects.

Key words: super-resolution, multi-receptive field, dilated convolution, spatial attention mechanism, gradient map

摘要：

针对医学磁共振成像（MRI）过程中由于噪声、成像技术和成像原理等干扰因素引起的图像细节丢失、纹理不清晰等问题，提出了基于多感受野的生成对抗网络医学MRI影像超分辨率重建算法。首先，利用多感受野特征提取块获取不同感受野下图像的全局特征信息，为避免感受野过小或过大导致图像的细节纹理丢失，将每组特征分为两组，其中一组用于反馈不同尺度感受野下的全局特征信息，另一组用于丰富下一组特征的局部细节纹理信息；然后，使用多感受野特征提取块构建特征融合组，并在每个特征融合组中添加空间注意力模块，充分获取图像的空间特征信息，减少了浅层和局部特征在网络中的丢失，在图像的细节上取得了更逼真的还原度；其次，将低分辨率图像的梯度图转化为高分辨率图像的梯度图辅助重建超分辨率图像；最终将恢复后的梯度图集成到超分辨率分支中，为超分辨率重建提供结构先验信息，有助于生成高质量的超分辨率图像。实验结果表明，相比基于梯度引导的结构保留超分辨率算法（SPSR），所提算法在×2、×3、×4尺度下的峰值信噪比（PSNR）分别提升了4.8%、2.7%、3.5%，重建出的医学MRI影像纹理细节更加丰富、视觉效果更加逼真。

关键词: 超分辨率, 多感受野, 空洞卷积, 空间注意力机制, 梯度图

CLC Number:

TP391

Pengwei LIU, Yuan GAO, Pinle QIN, Zhe YIN, Lifang WANG. Medical MRI image super-resolution reconstruction based on multi-receptive field generative adversarial network[J]. Journal of Computer Applications, 2022, 42(3): 938-945.

刘朋伟, 高媛, 秦品乐, 殷喆, 王丽芳. 基于多感受野的生成对抗网络医学MRI影像超分辨率重建[J]. 《计算机应用》唯一官方网站, 2022, 42(3): 938-945.

Figures/Tables 11

References 21

1	PARK S C， PARK M K， KANG M G. Super-resolution image reconstruction： a technical overview［J］．IEEE Signal Processing Magazine， 2003， 20（3）：21-36． 10.1109/msp.2003.1203207
2	DONG C， LOY C C， HE K， et al. Learning a deep convolutional network for image super-resolution［C］// Proceedings of the 13th European Conference on Computer Vision. Cham： Springer， 2014： 184-199. 10.1007/978-3-319-10593-2_13
3	KIM J， LEE J K， LEE K M. Deeply-recursive convolutional network for image super-resolution［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016：1637-1645. 10.1109/cvpr.2016.181
4	KIM J， LEE J K， LEE K M. Accurate image super-resolution using very deep convolutional networks［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 1646-1654. 10.1109/cvpr.2016.182
5	ZHANG Y， TIAN Y， KONG Y， ZHONG B， et al. Residual dense network for image super-resolution［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 2472-2481. 10.1109/cvpr.2018.00262
6	ZHANG Y， LI K， LI K， et al. Image super-resolution using very deep residual channel attention networks［C］// Proceedings of the 2018 European Conference on Computer Vision. Cham： Springer， 2018： 286-301. 10.1007/978-3-030-01234-2_18
7	高媛，王晓晨，秦品乐，等. 基于深度可分离卷积和宽残差网络的医学影像超分辨率重建［J］.计算机应用， 2019， 39（9）： 2731-2737. 10.11772/j.issn.1001-9081.2019030413
	GAO Y， WANG X C， QIN P L， et al. Medical image super-resolution reconstruction based on deep separable convolution and wide residual network［J］. Journal of Computer Applications， 2019， 39（9）： 2731-2737. 10.11772/j.issn.1001-9081.2019030413
8	LEDIG C， THEIS L， HUSZAR F， et al. Photo-realistic single image super-resolution using a generative adversarial network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway： IEEE， 2017：105-144. 10.1109/cvpr.2017.19
9	高媛，刘志，秦品乐，等.基于深度残差生成对抗网络的医学影像超分辨率算法［J］. 计算机应用， 2018， 38（9）： 2689-2695. 10.11772/j.issn.1001-9081.2018030574
	GAO Y， LIU Z， QIN P L， et al. Medical image super-resolution algorithm based on deep residual generative adversarial network［J］. Journal of Computer Applications， 2018， 38（9）： 2689-2695. 10.11772/j.issn.1001-9081.2018030574
10	龚明杰. 基于级联GAN网络的医学图像超分辨率重建及图像数据集增广［D］. 杭州：浙江工业大学， 2020：21-42.
	GONG M J. Medical image super-resolution reconstruction and image dataset enlargement based on cascaded GAN network ［D］. Hangzhou： Zhejiang University of Technology， 2020：21-42.
11	WANG X， YU K， WU S， et al. ESRGAN： Enhanced super-resolution generative adversarial networks［C］// Proceedings of the 2018 European Conference on Computer Vision. Cham： Springer， 2018： 63-79. 10.1007/978-3-030-11021-5_5
12	JUSTIN J， ALEXANDRE A. Perceptual losses for real-time style transfer and super-resolution［C］// Proceedings of the 14th European Conference on Computer Vision. Cham： Springer， 2016： 694-711. 10.1007/978-3-319-46475-6_43
13	GOODFELLOW I， MIRZA M， XU B， et al. Generative adversarial nets［EB/OL］.［2020-06-22］. . 10.1145/3422622
14	JOLICOEUR M A. The relativistic discriminator： a key element missing from standard GAN ［EB/OL］. ［2021-03-30］. .
15	SHANG T， DAI Q， ZHU S. Perceptual extreme super resolution network with receptive field block［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Piscataway： IEEE， 2020：1778-1787. 10.1109/cvprw50498.2020.00228
16	FATTAL R. Image upsampling via imposed edge statistics ［J］. ACM Transactions on Graphics， 2007， 26（3）： 95. 10.1145/1276377.1276496
17	SUN J， XU Z B， SHUM H Y. Gradient profile prior and its applications in image super-resolution and enhancement［J］. IEEE Transactions on Image Processing， 2011， 20（6）：1529-1542. 10.1109/tip.2010.2095871
18	YAN Q， XU Y， YANG X， et al. Single image super-resolution based on gradient profile sharpness［J］. IEEE Transactions on Image Processing， 2015， 24（10）：3187-3202. 10.1109/tip.2015.2414877
19	MA C， RAO Y， CHENG Y， et al. Structure-preserving super resolution with gradient guidance［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 7656-7775. 10.1109/cvpr42600.2020.00779
20	PARK J， WOO S， LEE J Y， et al. BAM： Bottleneck Attention Module［EB/OL］. ［2021-03-30］. . 10.1007/s11263-019-01283-0
21	SHRIVASTAVA A， PFISTER T， TUZEL O， et al. Learning from simulated and unsupervised images through adversarial training［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2242-2251. 10.1109/cvpr.2017.241

残差块	PSNR/dB			SSIM
残差块	×2	×3	×4	×2	×3		×4
RB	36.122	31.314	30.053	0.9223	0.8547	0.8343
RDB	37.857	32.412	32.759	0.9522	0.8670	0.8461
MFEB	38.387	35.146	33.592	0.9604	0.9339	0.9113
MFEB+ DESA	39.719	37.927	35.760	0.9856	0.9527	0.9241
MFGAN	41.736	38.937	36.841	0.9893	0.9647	0.9567

残差块	PSNR/dB			SSIM
残差块	×2	×3	×4	×2	×3		×4
RB	36.122	31.314	30.053	0.9223	0.8547	0.8343
RDB	37.857	32.412	32.759	0.9522	0.8670	0.8461
MFEB	38.387	35.146	33.592	0.9604	0.9339	0.9113
MFEB+ DESA	39.719	37.927	35.760	0.9856	0.9527	0.9241
MFGAN	41.736	38.937	36.841	0.9893	0.9647	0.9567

算法	PSNR/dB			SSIM			PI
算法	×2	×3	×4	×2	×3	×4	×2	×3	×4
SRCNN	30.516	28.044	26.680	0.8395	0.835	0.8182	5.89	7.56	9.06
SRGAN	32.647	30.809	27.211	0.9160	0.8519	0.8568	5.34	6.74	8.41
ESRGAN	34.580	32.353	30.408	0.9395	0.9043	0.9154	4.13	5.26	7.59
DR-GAN	36.936	34.481	32.060	0.9519	0.9325	0.9224	3.27	4.13	6.47
RFB-ESRGAN	38.543	35.597	34.819	0.9686	0.9384	0.9307	2.93	3.43	5.35
SPSR	39.829	37.897	35.601	0.9815	0.9569	0.9485	2.47	3.98	4.27
MFGAN（本文算法）	41.736	38. 937	36.841	0.9893	0.9647	0.9567	1.89	2.53	3.76

算法	PSNR/dB			SSIM			PI
算法	×2	×3	×4	×2	×3	×4	×2	×3	×4
SRCNN	30.516	28.044	26.680	0.8395	0.835	0.8182	5.89	7.56	9.06
SRGAN	32.647	30.809	27.211	0.9160	0.8519	0.8568	5.34	6.74	8.41
ESRGAN	34.580	32.353	30.408	0.9395	0.9043	0.9154	4.13	5.26	7.59
DR-GAN	36.936	34.481	32.060	0.9519	0.9325	0.9224	3.27	4.13	6.47
RFB-ESRGAN	38.543	35.597	34.819	0.9686	0.9384	0.9307	2.93	3.43	5.35
SPSR	39.829	37.897	35.601	0.9815	0.9569	0.9485	2.47	3.98	4.27
MFGAN（本文算法）	41.736	38. 937	36.841	0.9893	0.9647	0.9567	1.89	2.53	3.76

网络模型	参数数量/M	PSNR/dB
网络模型	参数数量/M	×2	×3	×4
SPSR	53.8	39.726	36.897	35.137
MFGAN（5MSRG）	53.6	39.135	34.983	34.734
MFGAN（本文模型）	54.9	41.736	38.937	36.841
MFGAN（7MSRG）	55.4	41.891	37.258	35.439

Medical MRI image super-resolution reconstruction based on multi-receptive field generative adversarial network

基于多感受野的生成对抗网络医学MRI影像超分辨率重建

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 21

Related Articles 15

Recommended Articles

Metrics

[1]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[2]	Zihan LIU, Dengwen ZHOU, Yukai LIU. Image super-resolution network based on global dependency Transformer [J]. Journal of Computer Applications, 2024, 44(5): 1588-1596.
[3]	Lin GUO, Kunhu LIU, Chenyang MA, Youxue LAI, Yingfen XU. Image super-resolution reconstruction based on residual attention network with receptive field expansion [J]. Journal of Computer Applications, 2024, 44(5): 1579-1587.
[4]	Jun FENG, Jiankang BI, Yiru HUO, Jiakuan LI. PIPNet： lightweight asphalt pavement crack image segmentation network [J]. Journal of Computer Applications, 2024, 44(5): 1520-1526.
[5]	Xinyuan YOU, Heng WANG. Monaural speech enhancement based on gated dilated convolutional recurrent network [J]. Journal of Computer Applications, 2024, 44(4): 1317-1324.
[6]	Yang LIU, Rong LIU, Ke FANG, Xinyue ZHANG, Guangxu WANG. Video super-resolution reconstruction network based on frame straddling optical flow [J]. Journal of Computer Applications, 2024, 44(4): 1277-1284.
[7]	Lin WANG, Jingliang LIU, Wuwei WANG. Small target detection method in UAV images based on fusion of dilated convolution and Transformer [J]. Journal of Computer Applications, 2024, 44(11): 3595-3602.
[8]	Yu ZENG, Yang ZHANG, Shang ZENG, Maoli FU, Qixue HE, Linlong ZENG. Time series prediction algorithm based on multi-scale gated dilated convolutional network [J]. Journal of Computer Applications, 2024, 44(11): 3427-3434.
[9]	Feng CAO, Xiaoling YANG, Jianbing YI, Jun LI. Contradiction separation super-deduction method and application [J]. Journal of Computer Applications, 2024, 44(10): 3074-3080.
[10]	Hao CHEN, Zhenping XIA, Cheng CHENG, Xing LIN-LI, Bowen ZHANG. Lightweight image super-resolution reconstruction network based on Transformer-CNN [J]. Journal of Computer Applications, 2024, 44(1): 292-299.
[11]	Meijia LIANG, Xinwu LIU, Xiaopeng HU. Small target detection algorithm for train operating environment image based on improved YOLOv3 [J]. Journal of Computer Applications, 2023, 43(8): 2611-2618.
[12]	Min LIANG, Jiayi LIU, Jie LI. Image super-resolution reconstruction method based on iterative feedback and attention mechanism [J]. Journal of Computer Applications, 2023, 43(7): 2280-2287.
[13]	Lihua SHEN, Bo LI. Super-resolution reconstruction of lung CT images based on feature pyramid network and dense network [J]. Journal of Computer Applications, 2023, 43(5): 1612-1619.
[14]	Hui LIU, Linyu ZHANG, Fugang WANG, Rujin HE. Object detection algorithm based on attention mechanism and context information [J]. Journal of Computer Applications, 2023, 43(5): 1557-1564.
[15]	Jiadong LI, Danpu ZHANG, Yaqiong FAN, Jianfeng YANG. Lightweight ship target detection algorithm based on improved YOLOv5 [J]. Journal of Computer Applications, 2023, 43(3): 923-929.