Super-resolution and multi-view fusion based on magnetic resonance image inter-layer interpolation

doi:10.11772/j.issn.1001-9081.2020122065

Journal of Computer Applications ›› 2021, Vol. 41 ›› Issue (11): 3362-3367.DOI: 10.11772/j.issn.1001-9081.2020122065

• Multimedia computing and computer simulation • Previous Articles Next Articles

Super-resolution and multi-view fusion based on magnetic resonance image inter-layer interpolation

Meng LI¹^,²^,³, Pinle QIN¹^,²^,³, Jianchao ZENG¹^,²^,³(), Junbo LI¹^,²^,³

^1.Shanxi Medical Imaging and Data Analysis Engineering Research Center （North University of China），Taiyuan Shanxi 030051，China
^2.School of Data Science and Technology，North University of China，Taiyuan Shanxi 030051，China
^3.Shanxi Medical Imaging Artificial Intelligence Engineering Technology Research Center （North University of China），Taiyuan Shanxi 030051，China

Received:2020-12-31 Revised:2021-04-23 Accepted:2021-05-18 Online:2021-04-23 Published:2021-11-10
Contact: Jianchao ZENG
About author:LI Meng，born in 1995，M. S. candidate. His research interests include machine learning，computer vision，medical image analysis
QIN Pinle，born in 1978，Ph. D.，professor. His research interests include medical image analysis，big data，machine vision
ZENG Jianchao，born in 1963，Ph. D.，professor. His research interests include maintenance decision and health management for complex systems
LI Junbo，born in 1996，M. S. candidate. His research interests include machine learning，computer vision，medical image analysis.
Supported by:
the Innovation Project of Graduate Education in Shanxi Province(2020SY381)

基于磁共振影像层间插值的超分辨率及多视角融合

李萌¹^,²^,³, 秦品乐¹^,²^,³, 曾建潮¹^,²^,³(), 李俊伯¹^,²^,³

^1.山西省医学影像与数据分析工程研究中心（中北大学），太原 030051
^2.中北大学大数据学院，太原 030051
^3.山西省医学影像人工智能工程技术研究中心（中北大学），太原 030051

通讯作者: 曾建潮
作者简介:李萌（1995—），男，山东济宁人，硕士研究生，主要研究方向：机器学习、计算机视觉、医学影像分析
秦品乐（1978—），男，山西长治人，教授，博士，CCF会员，主要研究方向：医学影像分析、大数据、机器视觉
曾建潮（1963—），男，山西太原人，教授，博士生导师，博士， CCF会员，主要研究方向：复杂系统的维护决策和健康管理
李俊伯（1996—），男，山西运城人，硕士研究生，主要研究方向：机器学习、计算机视觉、医学影像分析。
基金资助:
山西省研究生教育创新项目(2020SY381)

Abstract

Abstract:

The high resolution in Magnetic Resonance （MR） image slices and low resolution between the slices lead to the lack of medical diagnostic significance of MR in the coronal and sagittal planes. In order to solve the problem， a medical image processing algorithm based on inter-layer interpolation and multi-view fusion network was proposed. Firstly， the inter-layer interpolation module was introduced to cut the MR volume data from three-dimensional data into two-dimensional images along the coronal and sagittal directions. Then， after the feature extraction on the coronal and sagittal planes， the weights were dynamically calculated by the spatial matrix filter and used for upsampling factor with any size to magnify the image. Finally， the results of the coronal and sagittal images obtained in the inter-layer interpolation module were aggregated into three-dimensional data and then cut into two-dimensional images along the axial direction. The obtained two-dimensional images were fused in pairs and corrected by the axial direction data. Experimental results show that， compared with other super-resolution algorithms， the proposed algorithm has improved the Peak Signal-to-Noise Ratio （PSNR） by about 1 dB in ×2， ×3， and ×4 scales. It can be seen that the proposed algorithm can effectively improve the quality of image reconstruction.

Key words: super-resolution, neural network, inter-layer interpolation, brain Magnetic Resonance (MR) image, multi-view fusion

摘要：

针对磁共振（MR）图像切片内分辨率高而切片间分辨率低，导致MR在冠状面和矢状面上缺乏医学诊断意义的问题，提出了一种基于层间插值及多视角融合网络的医学图像处理算法。首先，引入了层间插值模块，用来将MR体数据沿冠状和矢状方向从三维数据切割成二维图像；然后，在分别对冠状面和矢状面进行特征提取之后，通过空间矩阵滤波器动态计算权重用于任意大小的上采样因子放大图像；最后，将冠状图和矢状图在层间插值模块中得到的结果聚合成三维数据后再次沿轴状方向切割成二维图像，对得到的二维图像两两进行融合并通过轴状方向数据进行修正。实验结果表明，所提算法相较于其他超分辨率算法在×2、×3、×4尺度下的峰值信噪比（PSNR）均有1 dB左右的提升，可见所提算法有效提升了图像的重建质量。

关键词: 超分辨率, 神经网络, 层间插值, 脑部磁共振影像, 多视角融合

CLC Number:

TP389.1

Meng LI, Pinle QIN, Jianchao ZENG, Junbo LI. Super-resolution and multi-view fusion based on magnetic resonance image inter-layer interpolation[J]. Journal of Computer Applications, 2021, 41(11): 3362-3367.

李萌, 秦品乐, 曾建潮, 李俊伯. 基于磁共振影像层间插值的超分辨率及多视角融合[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3362-3367.

Figures/Tables 8

References 25

1	RIVIERE P J LA， VARGAS P， FU G， et al. Accelerating X-ray fluorescence computed tomography ［C］// Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway： IEEE， 2009： 1000-1003. 10.1109/iembs.2009.5333568
2	PELED S， YESHURUN Y. Super resolution in MRI： application to human white matter fiber tract visualization by diffusion tensor imaging ［J］. Magnetic Resonance in Medicine， 2001， 45（1）： 29-35. 10.1002/1522-2594(200101)45:1<29::aid-mrm1005>3.0.co;2-z
3	GOSHTASBY A， TURNER D A， ACKERMAN L V. Matching of tomographic slices for interpolation ［J］. IEEE Transactions on Medical Imaging， 1992， 11（4）： 507-516. 10.1109/42.192686
4	GREVERA G J， UDUPA J K. Shape-based interpolation of multidimensional grey-level images ［J］. IEEE Transactions on Medical Imaging， 1996， 15（6）： 881-892. 10.1109/42.544506
5	LEE T Y， WANG W H. Morphology-based three-dimensional interpolation ［J］. IEEE Transactions on Medical Imaging， 2000， 19（7）： 711-721. 10.1109/42.875193
6	朱杨兴，鲍旭东.基于非刚体配准的体数据插值重建方法［J］.中国医学物理学杂志，2006，23（6）：412-415，457. 10.3969/j.issn.1005-202X.2006.06.005
	ZHU Y X， BAO X D. Nonrigid registration based interpolation approach ［J］. Chinese Journal of Medical Physics， 2006， 23（6）： 412-415， 457. 10.3969/j.issn.1005-202X.2006.06.005
7	LENG J L， XU G L， ZHANG Y J. Medical image interpolation based on multi-resolution registration ［J］. Computers and Mathematics with Applications， 2013， 66（1）： 1-18. 10.1016/j.camwa.2013.04.026
8	HERMAN G T， ROWLAND S W， YAU M M. A comparative study of the use of linear and modified cubic spline interpolation for image reconstruction ［J］. IEEE Transactions on Nuclear Science， 1979， 26（2）： 2879-2894. 10.1109/tns.1979.4330555
9	LEHMANN T M， GONNER C， SPITZER K. Survey： interpolation methods in medical image processing ［J］. IEEE Transactions on Medical Imaging， 1999， 18（11）： 1049-1075. 10.1109/42.816070
10	ZHANG C， LI W， TRAVIS D. Gaps-fill of SLC-off Landsat ETM+ satellite image using a geostatistical approach ［J］. International Journal of Remote Sensing， 2007， 28（22）： 5103-5122. 10.1080/01431160701250416
11	FERRAND G， LUONG M， CLOOS M A， et al. Accelerating parallel transmit array B1 mapping in high field MRI with slice undersampling and interpolation by kriging ［J］. IEEE Transactions on Medical Imaging， 2014， 33（8）： 1726-1734. 10.1109/tmi.2014.2322440
12	CARR J C， FRIGHT W R， BEATSON R K. Surface interpolation with radial basis functions for medical imaging ［J］. IEEE Transactions on Medical Imaging， 1997， 16（1）： 96-107. 10.1109/42.552059
13	PENNEY G P， SCHNABEL J A， RUECKERT D， et al. Registration-based interpolation ［J］. IEEE Transactions on Medical Imaging， 2004， 23（7）： 922-926. 10.1109/tmi.2004.828352
14	FRAKES D H， DASI L P， PEKKAN K， et al. A new method for registration-based medical image interpolation ［J］. IEEE Transactions on Medical Imaging， 2008， 27（3）： 370-377. 10.1109/tmi.2007.907324
15	SCHLEMPER J， CABALLERO J， HAJNAL J V， et al. A deep cascade of convolutional neural networks for dynamic MR image reconstruction ［J］. IEEE Transactions on Medical Imaging， 2018， 37（2）： 491-503. 10.1109/tmi.2017.2760978
16	RONNEBERGER O， FISCHER P， BROX T. U-net： convolutional networks for biomedical image segmentation ［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS9351. Cham： Springer， 2015： 234-241.
17	LIU S Q， XU D G， ZHOU S K， et al. 3D anisotropic hybrid network： transferring convolutional features from 2D images to 3D anisotropic volumes ［C］// Proceedings of the 2018 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS11071. Cham： Springer， 2018： 851-858.
18	DONG C， LOY C C， HE K M， et al. Image super-resolution using deep convolutional networks ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2016， 38（2）： 295-307. 10.1109/tpami.2015.2439281
19	KIM J， LEE J K， LEE K M. Accurate image super-resolution using very deep convolutional networks ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 1646-1654. 10.1109/cvpr.2016.182
20	DONG C， LOY C C， TANG X O. Accelerating the super-resolution convolutional neural network ［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS9906. Cham： Springer， 2016： 391-407.
21	ZHANG Y L， TIAN Y P， KONG Y， et al. Residual dense network for image super-resolution ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 2472-2481. 10.1109/cvpr.2018.00262
22	晋银峰，朱金秀，吴文霞，等.基于GAN和TV正则化的MRI超分辨率重建算法［J］.计算机工程与设计，2019，40（3）：767-773.
	JIN Y F， ZHU J X， WU W X， et al. Total variation regularized MRI super-resolution algorithm with generative adversarial networks ［J］. Computer Engineering and Design， 2019， 40（3）： 767-773.
23	LIM B， SON S， KIM H， et al. Enhanced deep residual networks for single image super-resolution ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2017： 1132-1440. 10.1109/cvprw.2017.151
24	HU X C， MU H Y， ZHANG X Y， et al. Meta-SR： a magnification-arbitrary network for super-resolution ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 1575-1584. 10.1109/cvpr.2019.00167
25	PENG C， LIN W A， LIAO H F， et al. Deep slice interpolation via marginal super-resolution， fusion and refinement ［EB/OL］. （2019-08-15）［2020-09-15］. . 10.1109/cvpr42600.2020.00777

算法	PSNR/dB					SSIM
算法	×2	×3	×4	均值	标准差	×2	×3	×4	均值	标准差
Bilinear	30.26	28.71	26.98	28.65	1.64	0.876 8	0.782 4	0.747 5	0.802 2	0.066 9
Bicubic	30.54	29.27	27.36	29.06	1.60	0.884 1	0.793 7	0.759 8	0.812 5	0.064 3
RDN	33.45	32.17	31.52	32.38	0.98	0.952 4	0.944 8	0.937 2	0.944 8	0.007 6
Meta-SR	34.08	32.49	31.67	32.75	1.22	0.960 9	0.949 4	0.940 5	0.950 3	0.010 2
DSIM-SR	34.53	33.31	32.22	33.35	1.15	0.963 5	0.950 7	0.944 1	0.952 8	0.009 9
本文算法	35.64	34.08	33.49	34.40	1.11	0.980 2	0.969 6	0.958 3	0.969 4	0.011 0

算法	PSNR/dB					SSIM
算法	×2	×3	×4	均值	标准差	×2	×3	×4	均值	标准差
Bilinear	30.26	28.71	26.98	28.65	1.64	0.876 8	0.782 4	0.747 5	0.802 2	0.066 9
Bicubic	30.54	29.27	27.36	29.06	1.60	0.884 1	0.793 7	0.759 8	0.812 5	0.064 3
RDN	33.45	32.17	31.52	32.38	0.98	0.952 4	0.944 8	0.937 2	0.944 8	0.007 6
Meta-SR	34.08	32.49	31.67	32.75	1.22	0.960 9	0.949 4	0.940 5	0.950 3	0.010 2
DSIM-SR	34.53	33.31	32.22	33.35	1.15	0.963 5	0.950 7	0.944 1	0.952 8	0.009 9
本文算法	35.64	34.08	33.49	34.40	1.11	0.980 2	0.969 6	0.958 3	0.969 4	0.011 0

算法	PSNR/dB					SSIM
算法	×2	×3	×4	均值	标准差	×2	×3	×4	均值	标准差
仅对冠状面	34.26	32.91	31.98	33.05	1.15	0.962 1	0.953 0	0.943 1	0.952 7	0.009 5
仅对矢状面	34.54	33.07	31.83	33.15	1.36	0.961 5	0.952 5	0.943 3	0.952 4	0.009 1
多视角融合后	35.64	34.08	33.49	34.40	1.11	0.980 2	0.969 6	0.958 3	0.969 4	0.011 0

算法	PSNR/dB					SSIM
算法	×2	×3	×4	均值	标准差	×2	×3	×4	均值	标准差
仅对冠状面	34.26	32.91	31.98	33.05	1.15	0.962 1	0.953 0	0.943 1	0.952 7	0.009 5
仅对矢状面	34.54	33.07	31.83	33.15	1.36	0.961 5	0.952 5	0.943 3	0.952 4	0.009 1
多视角融合后	35.64	34.08	33.49	34.40	1.11	0.980 2	0.969 6	0.958 3	0.969 4	0.011 0

[1]	WANG Hebing, ZHANG Chunmei. Facial landmark detection based on ResNeXt with asymmetric convolution and squeeze excitation [J]. Journal of Computer Applications, 2021, 41(9): 2741-2747.
[2]	LIU Zichen, LI Xiaojuan, WEI Wei. Automatic patent price evaluation based on recurrent neural network [J]. Journal of Computer Applications, 2021, 41(9): 2532-2538.
[3]	SONG Zhongshan, LIANG Jiarui, ZHENG Lu, LIU Zhenyu, TIE Jun. Remote sensing scene classification based on bidirectional gated scale feature fusion [J]. Journal of Computer Applications, 2021, 41(9): 2726-2735.
[4]	LI Kangkang, ZHANG Jing. Multi-layer encoding and decoding model for image captioning based on attention mechanism [J]. Journal of Computer Applications, 2021, 41(9): 2504-2509.
[5]	ZHANG Yongbin, CHANG Wenxin, SUN Lianshan, ZHANG Hang. Detection method of domains generated by dictionary-based domain generation algorithm [J]. Journal of Computer Applications, 2021, 41(9): 2609-2614.
[6]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[7]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[8]	MOU Changning, WANG Haipeng, ZHOU Piyu, HOU Xinhang. De novo peptide sequencing by tandem mass spectrometry based on graph convolutional neural network [J]. Journal of Computer Applications, 2021, 41(9): 2773-2779.
[9]	DING Yin, SANG Nan, LI Xiaoyu, WU Feizhou. Prediction method of capacity data in telecom industry based on recurrent neural network [J]. Journal of Computer Applications, 2021, 41(8): 2373-2378.
[10]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[11]	HUANG Chengcheng, DONG Xiaoxiao, LI Zhao. Deep pipeline 5×5 convolution method based on two-dimensional Winograd algorithm [J]. Journal of Computer Applications, 2021, 41(8): 2258-2264.
[12]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[13]	ZENG Xiangyin, ZHENG Bochuan, LIU Dan. Detection of left and right railway tracks based on deep convolutional neural network and clustering [J]. Journal of Computer Applications, 2021, 41(8): 2324-2329.
[14]	LIU Huan, LI Xiaoge, HU Likun, HU Feixiong, WANG Penghua. Knowledge graph driven recommendation model of graph neural network [J]. Journal of Computer Applications, 2021, 41(7): 1865-1870.
[15]	WU Zeju, JIAO Cuijuan, CHEN Liang. Tire defect detection method based on improved Faster R-CNN [J]. Journal of Computer Applications, 2021, 41(7): 1939-1946.

Super-resolution and multi-view fusion based on magnetic resonance image inter-layer interpolation

基于磁共振影像层间插值的超分辨率及多视角融合

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 25

Related Articles 15

Recommended Articles

Metrics