生成对抗残差网络的医学图像融合算法

doi:10.11772/j.issn.1001-9081.2019050937

摘要/Abstract

摘要： 针对传统医学图像融合中需要依靠先验知识手动设置融合规则和参数，导致融合效果存在不确定性、细节表现力不足的问题，提出了一种基于改进生成对抗网络（GAN）的脑部计算机断层扫描（CT）/磁共振（MR）图像融合算法。首先，对生成器和判别器两个部分的网络结构进行改进，在生成器网络的设计中采用残差块和快捷连接以加深网络结构，更好地捕获深层次的图像信息；然后，去掉常规网络中的下采样层，以减少图像传输过程中的信息损失，并将批量归一化改为层归一化，以更好地保留源图像信息，增加判别器网络的深度以提高网络性能；最后，连接CT图像和MR图像，将其输入到生成器网络中得到融合图像，通过损失函数不断优化网络参数，训练出最适合医学图像融合的模型来生成高质量的图像。实验结果表明，与当前表现优良的基于离散小波变换（DWT）算法、基于非下采样剪切波变换（NSCT）算法、基于稀疏表示（SR）算法和基于图像分类块稀疏表示（PSR）算法对比，所提算法在互信息（MI）、信息熵（IE）、结构相似性（SSIM）上均表现良好，最终的融合图像纹理和细节丰富，同时避免了人为因素对融合效果稳定性的影响。

关键词: 医学图像融合, 生成对抗网络, 计算机断层成像(CT)/磁共振(MR), 层归一化, 残差块

Abstract: In the traditional medical image fusion, it is necessary to manually set the fusion rules and parameters by using prior knowledge, which leads to the uncertainty of fusion effect and the lack of detail expression. In order to solve the problems, a Computed Tomography (CT)/Magnetic Resonance (MR) image fusion algorithm based on improved Generative Adversarial Network (GAN) was proposed. Firstly, the network structures of generator and discriminator were improved. In the design of generator network, residual block and fast connection were used to deepen the network structure, so as to better capture the deep image information. Then, the down-sampling layer of the traditional network was removed to reduce the information loss during image transmission, and the batch normalization was changed to the layer normalization to better retain the source image information, and the depth of the discriminator network was increased to improve the network performance. Finally, the CT image and the MR image were connected and input into the generator network to obtain the fused image, and the network parameters were continuously optimized through the loss function, and the model most suitable for medical image fusion was trained to generate the high-quality image. The experimental results show that, the proposed algorithm is superior to Discrete Wavelet Transformation (DWT) algorithm, NonSubsampled Contourlet Transform (NSCT) algorithm, Sparse Representation (SR) algorithm and Sparse Representation of classified image Patches (PSR) algorithm on Mutual Information (MI), Information Entropy (IE) and Structural SIMilarity (SSIM). The final fused image has rich texture and details. At the same time, the influence of human factors on the stability of the fusion effect is avoided.

Key words: medical image fusion, Generative Adversarial Network (GAN), Computed Tomography (CT)/ Magnetic Resonance (MRI), layer normalization, residual block

中图分类号:

TP391.4

高媛, 吴帆, 秦品乐, 王丽芳. 生成对抗残差网络的医学图像融合算法[J]. 计算机应用, 2019, 39(12): 3528-3534.

GAO Yuan, WU Fan, QIN Pinle, WANG Lifang. Medical image fusion algorithm based on generative adversarial residual network[J]. Journal of Computer Applications, 2019, 39(12): 3528-3534.

参考文献

[1] SHEN R, CHENG I, BASU A. Cross-scale coefficient selection for volumetric medical image fusion[J]. IEEE Transactions on Biomedical Engineering, 2013, 60(4):1069-1079.
[2] TAO L, QIAN Z. An improved medical image fusion algorithm based on wavelet transform[C]//Proceedings of the 7th International Conference on Natural Computation. Piscataway:IEEE, 2011:76-78.
[3] 江泽涛,杨阳,郭川.基于提升小波变换的图像融合改进算法的研究[J].图像与信号处理,2015,4:11-19.(JANG Z T, YANG Y, GUO C. Study on the improvement of image fusion algorithm based on lifting wavelet transform[J]. Journal of Image and Signal Processing, 2015, 4:11-19.)
[4] 曹义亲,雷章明,黄晓生.基于区域的非下采样形态小波医学图像融合算法[J].计算机应用研究,2012,29(6):2379-2381.(CAO Y Q, LEI Z M, HUANG X S. Region-based algorithm for non-sampling morphological wavelet medical image fusion[J]. Application Research of Computers, 2012, 29(6):2379-2381.)
[5] LI S, YANG B, HU J. Performance comparison of different multi-resolution transforms for image fusion[J]. Information Fusion, 2011, 12(2):74-84.
[6] WANG J, PENG J, FENG X, et al. Fusion method for infrared and visible images by using non-negative sparse representation[J]. Infrared Physics and Technology, 2014, 67:477-489.
[7] XIANG T, YAN L, GAO R. A fusion algorithm for infrared and visible images based on adaptive dual-channel unit-linking PCNN in NSCT domain[J]. Infrared Physics and Technology, 2015, 69:53-61.
[8] MA J, ZHOU Z, WANG B, et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics and Technology, 2017, 82:8-17.
[9] HE G, XING S, HE X, et al. Image fusion method based on simultaneous sparse representation with non-subsampled contourlet transform[J]. IET Computer Vision, 2019, 13(2):240-248.
[10] DING S, ZHAO X, XU H, et al. NSCT-PCNN image fusion based on image gradient motivation[J]. IET Computer Vision, 2018, 12(4):377-383.
[11] 蔺素珍,韩泽.基于深度堆叠卷积神经网络的图像融合[J].计算机学报,2017,40(11):2506-2518.(LIN S Z, HAN Z. Images fusion based on deep stack convolutional neural network[J]. Chinese Journal of Computers, 2017, 40(11):2506-2518.)
[12] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Red Hook:Curran Associates Inc., 2012:1097-1105.
[13] RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL].[2019-01-20]. https://arxiv.org/pdf/1511.06434.pdf.
[14] PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context encoders:feature learning by inpainting[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:2536-2544.
[15] ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:5967-5976.
[16] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge, MA:MIT Press, 2014:2672-2680.
[17] 黄宣珲.基于深度学习的医学图像处理[J].中国新通信,2019,21(3):103-105.(HUANG X H. Medical image processing based on deep learning[J]. China New Telecommunications, 2019, 21(3):103-105.)
[18] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:770-778.
[19] LIU Y, CHEN X, WARD R K, et al. Medical image fusion via convolutional sparsity based morphological component analysis[J]. IEEE Signal Processing Letters, 2019, 26(3):485-489.
[20] XU X, WANG Y, CHEN S. Medical image fusion using discrete fractional wavelet transform[J]. Biomedical Signal Processing and Control, 2016, 27:103-111.
[21] LI H, MANJUNATH B S, MITRA S K. Multisensor image fusion using the wavelet transform[J]. Graphical Models and Image Processing, 1995, 57(3):235-245.
[22] DAS S, KUNDU M K. NSCT-based multimodal medical image fusion using pulse-coupled neural network and modified spatial frequency[J]. Medical and Biological Engineering and Computing, 2012, 50(10):1105-1114.
[23] YANG B, LI S. Multifocus image fusion and restoration with sparse representation[J]. IEEE Transactions on Instrumentation and Measurement, 2010, 59(4):884-892.
[24] ZONG J, QIU T. Medical image fusion based on sparse representation of classified image patches[J]. Biomedical Signal Processing and Control, 2017, 34:195-205.
[25] VEIT A, WILBER M J, BELONGIE S. Residual networks behave like ensembles of relatively shallow networks[C]//Proceedings of the 29th International Conference on Neural Information Processing Systems. Cambridge, MA:MIT Press, 2016:550-558.
[26] HAN D, KIM J, KIM J. Deep pyramidal residual networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:6307-6315.
[27] IOFFE S, SZEGEDY C. Batch normalization:accelerating deep network training by reducing internal covariate shift[EB/OL].[2019-01-20]. https://arxiv.org/pdf/1502.03167.pdf.
[28] BA J L, KIROS J R, HINTON G E. Layer normalization[EB/OL].[2019-01-20]. https://arxiv.org/pdf/1607.06450.pdf.
[29] LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:1132-1140.