Super-resolution reconstruction of lung CT images based on feature pyramid network and dense network

doi:10.11772/j.issn.1001-9081.2022040620

Abstract

Abstract:

To pay more attention to pulmonary nodules and satisfy the objective existence of reconstructed features in lung Computed Tomography （CT） image Super-Resolution （SR） reconstruction， a lung image SR reconstruction method based on Feature Pyramid Network （FPN） and dense network was proposed. Firstly， at the feature extraction layer， FPN was used to extract features. Secondly， the local structure based on residual network was designed at the feature mapping layer， and then the special dense network was used to connect the local structure. Thirdly， at the feature reconstruction layer， Convolution Neural Network （CNN） was used to gradually reduce the convolution layers with different depths to the image size. Finally， the residual network was used to integrate the initial Low-Resolution （LR） features and the reconstructed High-Resolution （HR） features to form the final SR image. In comparison experiments， the deep learning network with two feature fusion in FPN and five local structure connections in feature mapping has better effect. Compared with classic networks such as Super-Resolution Convolutional Neural Network （SRCNN）， the proposed network has higher Peak Signal-to-Noise Ratio （PSNR） and better visual quality of the reconstructed SR images.

Key words: lung Computed Tomography (CT) image, Super-Resolution (SR) reconstruction, Feature Pyramid Network (FPN), dense network, residual network

摘要：

针对肺部计算机断层扫描（CT）图像的超分辨率（SR）重建中需要加大对肺结节的关注度、满足重建后的特征具有客观存在性等问题，提出一种基于特征金字塔网络（FPN）和密集网络的肺部图像SR重建方法。首先，在特征提取层利用FPN提取特征；其次，在特征映射层设计基于残差网络的局部结构，再用特殊的密集网络连接此类局部结构；再次，在特征重建层利用卷积神经网络（CNN）将不同深度的卷积层逐渐降为图像大小；最后，利用残差网络融合初始低分辨率（LR）特征与重建的高分辨率（HR）特征，形成最终的SR图像。对比实验显示，FPN中2次特征融合和特征映射中5个局部结构连接的深度学习网络效果更佳。所提出的网络相较于超分辨率卷积神经网络（SRCNN）等经典网络重建SR图像的峰值信噪比（PSNR）更高，并且可以获得更好的视觉质量。

关键词: 肺部计算机断层扫描图像, 超分辨率重建, 特征金字塔网络, 密集网络, 残差网络

CLC Number:

TP183

Lihua SHEN, Bo LI. Super-resolution reconstruction of lung CT images based on feature pyramid network and dense network[J]. Journal of Computer Applications, 2023, 43(5): 1612-1619.

申利华, 李波. 基于特征金字塔网络和密集网络的肺部CT图像超分辨率重建[J]. 《计算机应用》唯一官方网站, 2023, 43(5): 1612-1619.

Figures/Tables 16

Fig. 1 FDSR network

Fig. 2 Feature pyramid network

Fig. 3 Feature extraction structure of FDSR

Fig. 4 RCN

Fig. 5 Dense connection between RCNs

Fig. 6 Feature reconstruction

Fig. 7 Training and evaluation data preprocessing

Fig. 8 HR image and processed LR image

Fig. 9 Relationship between PSNR and epochs

Tab. 1 PSNR and SSIM of test images under different fusion times

图像	比例	1次融合（m1）		2次融合（m2）		3次融合（m3）
图像	比例	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
1	$×$ 2	36.91	0.957	37.17	0.958	36.74	0.956
	$×$ 3	27.52	0.897	28.14	0.903	27.92	0.893
	$×$ 4	24.08	0.762	24.11	0.762	24.09	0.762
2	$×$ 2	26.28	0.752	26.41	0.756	26.12	0.743
	$×$ 3	21.59	0.602	21.62	0.603	21.66	0.602
	$×$ 4	20.28	0.523	20.27	0.523	20.27	0.523
3	$×$ 2	35.48	0.930	35.42	0.929	35.35	0.928
	$×$ 3	26.42	0.813	26.46	0.814	26.48	0.813
	$×$ 4	24.63	0.738	24.64	0.739	24.63	0.738

Tab. 1 PSNR and SSIM of test images under different fusion times

图像	比例	1次融合（m1）		2次融合（m2）		3次融合（m3）
图像	比例	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
1	$×$ 2	36.91	0.957	37.17	0.958	36.74	0.956
	$×$ 3	27.52	0.897	28.14	0.903	27.92	0.893
	$×$ 4	24.08	0.762	24.11	0.762	24.09	0.762
2	$×$ 2	26.28	0.752	26.41	0.756	26.12	0.743
	$×$ 3	21.59	0.602	21.62	0.603	21.66	0.602
	$×$ 4	20.28	0.523	20.27	0.523	20.27	0.523
3	$×$ 2	35.48	0.930	35.42	0.929	35.35	0.928
	$×$ 3	26.42	0.813	26.46	0.814	26.48	0.813
	$×$ 4	24.63	0.738	24.64	0.739	24.63	0.738

Fig. 10 Comparison of images with different fusion times

Fig. 11 VDSR_RCN

Tab. 2 Experimental results of VDSR， VDSR_RDB and VDSR_RCN

图像	比例	VDSR		VDSR_RDB		VDSR_RCN
图像	比例	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
1	$×$ 2	34.89	0.949	34.74	0.949	36.59	0.957
	$×$ 3	29.78	0.836	29.06	0.842	30.64	0.872
	$×$ 4	24.03	0.761	24.04	0.761	24.07	0.762
2	$×$ 2	25.91	0.761	25.87	0.761	26.81	0.777
	$×$ 3	21.64	0.604	21.74	0.606	21.61	0.603
	$×$ 4	20.19	0.521	20.18	0.521	20.24	0.522
3	$×$ 2	34.45	0.924	34.25	0.924	35.52	0.931
	$×$ 3	26.49	0.814	26.62	0.815	26.38	0.814
	$×$ 4	24.58	0.737	24.56	0.737	24.61	0.738

Tab. 2 Experimental results of VDSR， VDSR_RDB and VDSR_RCN

图像	比例	VDSR		VDSR_RDB		VDSR_RCN
图像	比例	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
1	$×$ 2	34.89	0.949	34.74	0.949	36.59	0.957
	$×$ 3	29.78	0.836	29.06	0.842	30.64	0.872
	$×$ 4	24.03	0.761	24.04	0.761	24.07	0.762
2	$×$ 2	25.91	0.761	25.87	0.761	26.81	0.777
	$×$ 3	21.64	0.604	21.74	0.606	21.61	0.603
	$×$ 4	20.19	0.521	20.18	0.521	20.24	0.522
3	$×$ 2	34.45	0.924	34.25	0.924	35.52	0.931
	$×$ 3	26.49	0.814	26.62	0.815	26.38	0.814
	$×$ 4	24.58	0.737	24.56	0.737	24.61	0.738

Tab. 3 PSNR and SSIM of test images under different RCN numbers

图像	比例	n₁		n₂		n₃		n₄		n₅		n₆
图像	比例	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
1	$×$ 2	36.62	0.956	37.09	0.957	37.07	0.957	37.04	0.957	37.17	0.958	36.82	0.956
	$×$ 3	27.51	0.887	28.06	0.902	28.04	0.902	27.85	0.898	28.14	0.903	27.63	0.892
	$×$ 4	24.08	0.762	24.09	0.761	24.09	0.762	24.09	0.762	24.11	0.762	24.08	0.762
2	$×$ 2	26.08	0.743	26.27	0.749	26.33	0.749	26.33	0.751	26.41	0.756	26.09	0.742
	$×$ 3	21.61	0.601	21.57	0.601	21.59	0.601	21.61	0.601	21.62	0.603	21.61	0.602
	$×$ 4	20.28	0.523	20.26	0.522	20.27	0.522	20.26	0.522	20.27	0.523	20.25	0.521
3	$×$ 2	35.38	0.928	35.41	0.929	35.42	0.929	35.41	0.929	35.42	0.929	35.37	0.928
	$×$ 3	26.51	0.813	26.43	0.812	26.45	0.812	26.46	0.812	26.46	0.814	26.45	0.813
	$×$ 4	24.63	0.738	24.62	0.737	24.63	0.737	24.61	0.736	24.64	0.739	24.62	0.738

Tab. 3 PSNR and SSIM of test images under different RCN numbers

图像	比例	n₁		n₂		n₃		n₄		n₅		n₆
图像	比例	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
1	$×$ 2	36.62	0.956	37.09	0.957	37.07	0.957	37.04	0.957	37.17	0.958	36.82	0.956
	$×$ 3	27.51	0.887	28.06	0.902	28.04	0.902	27.85	0.898	28.14	0.903	27.63	0.892
	$×$ 4	24.08	0.762	24.09	0.761	24.09	0.762	24.09	0.762	24.11	0.762	24.08	0.762
2	$×$ 2	26.08	0.743	26.27	0.749	26.33	0.749	26.33	0.751	26.41	0.756	26.09	0.742
	$×$ 3	21.61	0.601	21.57	0.601	21.59	0.601	21.61	0.601	21.62	0.603	21.61	0.602
	$×$ 4	20.28	0.523	20.26	0.522	20.27	0.522	20.26	0.522	20.27	0.523	20.25	0.521
3	$×$ 2	35.38	0.928	35.41	0.929	35.42	0.929	35.41	0.929	35.42	0.929	35.37	0.928
	$×$ 3	26.51	0.813	26.43	0.812	26.45	0.812	26.46	0.812	26.46	0.814	26.45	0.813
	$×$ 4	24.63	0.738	24.62	0.737	24.63	0.737	24.61	0.736	24.64	0.739	24.62	0.738

Tab. 4 Comparison of FDSR and different deep learning methods on PSNR and SSIM

图像	比例	Bicubic		SRCNN		FSRCNN		VDSR		LapSRN		FDSR（RCN=5）
图像	比例	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
1	$×$ 2	29.78	0.915	36.08	0.955	36.28	0.956	36.81	0.956	37.02	0.956	37.17	0.958
	$×$ 3	25.56	0.823	27.23	0.872	27.26	0.876	27.54	0.881	27.86	0.893	28.14	0.903
	$×$ 4	24.07	0.758	24.02	0.761	24.22	0.762	24.32	0.763	24.40	0.764	24.11	0.762
2	$×$ 2	23.67	0.696	26.36	0.769	26.39	0.769	26.41	0.756	26.41	0.762	26.41	0.756
	$×$ 3	21.66	0.592	21.69	0.605	21.70	0.605	21.71	0.606	22.83	0.609	21.62	0.603
	$×$ 4	20.17	0.521	20.18	0.521	20.21	0.520	20.21	0.522	20.25	0.523	20.27	0.523
3	$×$ 2	29.96	0.888	35.20	0.928	35.34	0.927	35.41	0.928	35.42	0.928	35.42	0.929
	$×$ 3	26.42	0.803	26.53	0.815	26.54	0.815	26.55	0.816	26.53	0.814	26.46	0.814
	$×$ 4	24.58	0.735	24.54	0.736	24.59	0.737	24.61	0.737	24.62	0.738	24.64	0.739

Tab. 4 Comparison of FDSR and different deep learning methods on PSNR and SSIM

图像	比例	Bicubic		SRCNN		FSRCNN		VDSR		LapSRN		FDSR（RCN=5）
图像	比例	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
1	$×$ 2	29.78	0.915	36.08	0.955	36.28	0.956	36.81	0.956	37.02	0.956	37.17	0.958
	$×$ 3	25.56	0.823	27.23	0.872	27.26	0.876	27.54	0.881	27.86	0.893	28.14	0.903
	$×$ 4	24.07	0.758	24.02	0.761	24.22	0.762	24.32	0.763	24.40	0.764	24.11	0.762
2	$×$ 2	23.67	0.696	26.36	0.769	26.39	0.769	26.41	0.756	26.41	0.762	26.41	0.756
	$×$ 3	21.66	0.592	21.69	0.605	21.70	0.605	21.71	0.606	22.83	0.609	21.62	0.603
	$×$ 4	20.17	0.521	20.18	0.521	20.21	0.520	20.21	0.522	20.25	0.523	20.27	0.523
3	$×$ 2	29.96	0.888	35.20	0.928	35.34	0.927	35.41	0.928	35.42	0.928	35.42	0.929
	$×$ 3	26.42	0.803	26.53	0.815	26.54	0.815	26.55	0.816	26.53	0.814	26.46	0.814
	$×$ 4	24.58	0.735	24.54	0.736	24.59	0.737	24.61	0.737	24.62	0.738	24.64	0.739

Fig. 12 Visual comparison of FDSR and different deep learning methods

References 29

1	WANG C P， WANG S M， XIA Z Q， et al. Medical image super-resolution via deep residual neural network in the shearlet domain［J］. Multimedia Tools and Applications， 2021， 80（17）：26637-26655.
2	YU J F， GAO X B， TAO D C， et al. A unified learning framework for single image super-resolution［J］. IEEE Transactions on Neural Networks and Learning Systems， 2014， 25（4）：780-792. 10.1109/tnnls.2013.2281313
3	段友祥，张含笑，孙歧峰，等. 基于拉普拉斯金字塔生成对抗网络的图像超分辨率重建算法［J］. 计算机应用， 2021， 41（4）： 1020-1026. 10.11772/j.issn.1001-9081.2020081299
	DUAN Y X， ZHANG H X， SUN Q F， et al. Image super-resolution reconstruction algorithm based on Laplacian pyramid generative adversarial network［J］. Journal of Computer Applications， 2021， 41（4）： 1020-1026. 10.11772/j.issn.1001-9081.2020081299
4	DONG C， LOY C C， HE K M， et al. Learning a deep convolutional network for image super-resolution［C］// Proceedings of the 2014 European Conference on Computer Vision， LNCS 8692. Cham： Springer， 2014：184-199.
5	UMEHARA K， OTA J， ISHIDA T. Application of super-resolution convolutional neural network for enhancing image resolution in chest CT［J］. Journal of Digital Imaging， 2018， 31（4）：441-450. 10.1007/s10278-017-0033-z
6	ABDEL-ZAHER A M， ELDEIB A M. Breast cancer classification using deep belief networks［J］. Expert Systems with Applications， 2016， 46：139-144. 10.1016/j.eswa.2015.10.015
7	PRIYA C L， GOWTHAMI D， POONGUZHALI S. Lung pattern classification for interstitial lung diseases using an ANN-back propagation network ［C］// Proceedings of the 2017 International Conference on Communication and Signal Processing. Piscataway： IEEE， 2017： 1917-1922. 10.1109/iccsp.2017.8286732
8	DONG C， LOY C C， TANG X O. Accelerating the super-resolution convolutional neural network［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9906. Cham： Springer， 2016：391-407.
9	SHI W Z， CABALLERO J， HUSZÁR F， et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 1874-1883. 10.1109/cvpr.2016.207
10	ZHANG D， ZHU B L， ZHONG Y H. MBMR-Net： multi-branches multi-resolution cross-projection network for single image super-resolution［J］. Applied Intelligence， 2022， 52（11）：12975-12989. 10.1007/s10489-022-03322-9
11	HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016：770-778. 10.1109/cvpr.2016.90
12	KIM J， LEE J K， LEE K M. Deeply-recursive convolutional network for image super-resolution［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016：1637-1645. 10.1109/cvpr.2016.181
13	DU Y B， JIA R S， CUI Z， et al. X-ray image super-resolution reconstruction based on a multiple distillation feedback network［J］. Applied Intelligence， 2021， 51（7）： 5081-5094. 10.1007/s10489-020-02123-2
14	LEDING C， THEIS L， HUSZÁR F， et al. Photo-realistic single image super-resolution using a generative adversarial network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 105-114. 10.1109/cvpr.2017.19
15	WANG Y Q， LI X， NAN F Z， et al. Image super-resolution reconstruction based on generative adversarial network model with feedback and attention mechanisms［J］. Multimedia Tools and Applications， 2022， 81（5）：6633-6652. 10.1007/s11042-021-11679-1
16	YANG X， LI X C， LI Z Q， et al. Image super-resolution based on deep neural network of multiple attention mechanism［J］. Journal of Visual Communication and Image Representation， 2021， 75： No.103019. 10.1016/j.jvcir.2021.103019
17	LAI W S， HUANG J B， AHUJA N， et al. Deep Laplacian pyramid networks for fast and accurate super-resolution［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017：5835-5843. 10.1109/cvpr.2017.618
18	TANG R， CHEN L H， ZHANG R Z， et al. Medical image super-resolution with Laplacian dense network［J］. Multimedia Tools and Applications， 2020， 81（3）：3131-3144.
19	常志国，郭茹侠，李晶，等. 基于稀疏表示和近邻嵌入的图像超分辨率重构［J］. 计算机测量与控制， 2016， 24（5）：173-177. 10.16526/j.cnki.11-4762/tp.2016.05.050
	CHANG Z G， GUO R X， LI J， et al. Image super-resolution reconstruction based on sparse representation and neighbor embedding［J］. Computer Measurement and Control， 2016， 24（5）：173-177. 10.16526/j.cnki.11-4762/tp.2016.05.050
20	KIM J， LEE J K， LEE K M. Accurate image super-resolution using very deep convolutional networks［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016：1646-1654. 10.1109/cvpr.2016.182
21	YANG X， ZHANG Y F， LI Tet al. Image super-resolution based on the down-sampling iterative module and deep CNN［J］. Circuits， Systems， and Signal Processing， 2021， 40（7）：3437-3455. 10.1007/s00034-020-01630-4
22	LI Q Q， GUO X M， HAN J X， et al. Single image super-resolution reconstruction algorithm based on deep learning［J］. Journal of Physics： Conference Series， 2021， 1995： No.012034. 10.1088/1742-6596/1995/1/012034
23	QIU D F， ZHENG L X， ZHU J Q， et al. Multiple improved residual networks for medical image super-resolution［J］. Future Generation Computer Systems， 2021， 116：200-208. 10.1016/j.future.2020.11.001
24	刘朋伟，高媛，秦品乐，等. 基于多感受野的生成对抗网络医学MRI影像超分辨率重建［J］. 计算机应用， 2022， 42（3）： 938-945.
	LIU P W， GAO Y， QIN P L， et al. Medical MRI image super-resolution reconstruction based on multi-receptive field generative adversarial network［J］. Journal of Computer Applications， 2022， 42（3）： 938-945.
25	ZHANG Y B， ZHANG Z H， LI T L. Research on image super-resolution reconstruction based on deep learning［J］. Journal of Physics Conference Series， 2021， 1802（4）： No.042034. 10.1088/1742-6596/1802/4/042034
26	LIU F Q， YU Q， CHEN L H， et al. Aerial image super-resolution based on deep recursive dense network for disaster area surveillance［J］. Personal and Ubiquitous Computing， 2022， 26（4）：1205-1214. 10.1007/s00779-020-01516-x
27	ZHANG Y L， LI K P， LI K， et al. Image super-resolution using very deep residual channel attention networks［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11211. Cham： Springer， 2018： 294-310.
28	WANG X T， YU K， WU S X， et al. ESRGAN： enhanced super-resolution generative adversarial networks［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11133. Cham： Springer， 2019： 63-79.
29	YANG X， ZHANG Y F， GUO Y Q， et al. An image super-resolution deep learning network based on multi-level feature extraction module［J］. Multimedia Tools and Applications， 2021， 80（5）：7063-7075. 10.1007/s11042-020-09958-4

[1]	Yuanjiong LIU, Maozheng HE, Yibin HUANG, Cheng QIAN. Ship identification model based on ResNet50 and improved attention mechanism [J]. Journal of Computer Applications, 2024, 44(6): 1935-1941.
[2]	Lin GUO, Kunhu LIU, Chenyang MA, Youxue LAI, Yingfen XU. Image super-resolution reconstruction based on residual attention network with receptive field expansion [J]. Journal of Computer Applications, 2024, 44(5): 1579-1587.
[3]	Boyue WANG, Yingxiang LI, Jiandan ZHONG. Segmentation network for day and night ground-based cloud images based on improved Res-UNet [J]. Journal of Computer Applications, 2024, 44(4): 1310-1316.
[4]	Shunwang FU, Qian CHEN, Zhi LI, Guomei WANG, Yu LU. Two-channel progressive feature filtering network for tampered image detection and localization [J]. Journal of Computer Applications, 2024, 44(4): 1303-1309.
[5]	Jingxian ZHOU, Xina LI. UAV detection and recognition based on improved convolutional neural network and radio frequency fingerprint [J]. Journal of Computer Applications, 2024, 44(3): 876-882.
[6]	Xueyu HUANG, Huaiyu HE, Huimin LIN, Jinshui CHEN. Classification and recognition method of copper alloy metallograph based on feature aggregation [J]. Journal of Computer Applications, 2023, 43(8): 2593-2601.
[7]	Zhangjian JI, Ming ZHANG, Zilong WANG. High-precision object detection algorithm based on improved VarifocalNet [J]. Journal of Computer Applications, 2023, 43(7): 2147-2154.
[8]	Yuanyuan QIN, Hong ZHANG. Pulmonary nodule detection algorithm based on attention feature pyramid networks [J]. Journal of Computer Applications, 2023, 43(7): 2311-2318.
[9]	Huibin ZHANG, Liping FENG, Yaojun HAO, Yining WANG. Ancient mural dynasty identification based on attention mechanism and transfer learning [J]. Journal of Computer Applications, 2023, 43(6): 1826-1832.
[10]	Wanli SHEN, Yujin ZHANG, Wan HU. U-shaped feature pyramid network for image inpainting forensics [J]. Journal of Computer Applications, 2023, 43(2): 545-551.
[11]	Qihong SONG, Jianxun LIU, Haize HU, Xiangping ZHANG. Code search model based on collaborative fusion network [J]. Journal of Computer Applications, 2023, 43(12): 3896-3902.
[12]	Zhiang ZHANG, Guangzhong LIAO. Multi-scale feature enhanced retinal vessel segmentation algorithm based on U-Net [J]. Journal of Computer Applications, 2023, 43(10): 3275-3281.
[13]	GUO Zihao, DONG Lele, QU Zhijian. Arthropod object detection method based on improved Faster RCNN [J]. Journal of Computer Applications, 2023, 43(1): 88-97.
[14]	Liefa LIAO, Zhiming LI, Saisai ZHANG. Image retrieval method based on deep residual network and iterative quantization hashing [J]. Journal of Computer Applications, 2022, 42(9): 2845-2852.
[15]	Huaiqing HE, Jianqing YAN, Kanghua HUI. Lightweight face recognition method based on deep residual network [J]. Journal of Computer Applications, 2022, 42(7): 2030-2036.