Dual-branch residual low-light image enhancement combined with attention

doi:10.11772/j.issn.1001-9081.2022030479

Abstract

Abstract:

Photos taken in low-light conditions will suffer from a series of visual problems due to underexposure， such as low brightness， loss of information， noise and color distortion. In order to solve the above problems， a dual-branch residual low-light image enhancement network combined with attention was proposed. Firstly， the improved InceptionV2 was used to extract shallow features. Secondly， Residual Feature extraction Block （RFB） and Dense RFB （DRFB） were used to extract deep features. Thirdly， the shallow and deep features were fused， and the fusion result was input into Brightness Adjustment Module （BAM） to adjust the brightness， and finally the enhanced image was obtained. At the same time， a Feature Fusion Module （FFM） was designed in combination with attention mechanism to capture important feature information， which helped to restore dark areas in low-light images. In addition， a joint loss function was introduced to measure the network training loss from multiple aspects. Experimental results show that， compared with RRM （Robust Retinex Model）， Zero-DCE （Zero-reference Deep Curve Estimation） and EnlightenGAN （Enlighten Generative Adversarial Network）， on LOL （LOw-Light） dataset， the Peak Signal-to-Noise Ratio （PSNR） indicator of the proposed network is increased by 49.9%， 40.0% and 18.5% respectively. Meanwhile， the Structural Similarity Index Measure （SSIM） indicator of the proposed network is increased by 20.3%， 50.0% and 34.5% compared with those of the above three on LOL?V2 dataset. The proposed network improves the brightness of low-light images while reducing noise， color distortion and artifacts， resulting in sharper and more natural enhanced images.

Key words: low-light, image enhancement, attention mechanism, dual-branch, joint loss function

摘要：

在低光条件下拍摄的照片会因曝光不足而产生一系列的视觉问题，如亮度低、信息丢失、噪声和颜色失真等。为了解决上述问题，提出一个结合注意力的双分支残差低光照图像增强网络。首先，采用改进InceptionV2提取浅层特征；其次，使用残差特征提取块（RFB）和稠密残差特征提取块（DRFB）提取深层特征；然后，融合浅层和深层特征，并将融合结果输入亮度调整块（BAM）调整亮度，最终得到增强图像。同时，结合注意力机制设计特征融合块（FFM）捕获重要的特征信息，以帮助恢复低光照图像的暗部区域。此外，引入一个联合损失函数从多方面衡量网络训练损失。实验结果表明，相较于鲁棒的视网膜大脑皮层模型（RRM）、Zero-DCE（Zero-Reference Deep Curve Estimation）和EnlightenGAN（Enlighten Generative Adversarial Network），在LOL（LOw-Light）数据集上，所提网络的峰值信噪比（PSNR）指标分别提高了49.9%、40.0%和18.5%；在LOL-V2数据集上，结构相似性（SSIM）指标分别提高了20.3%、50.0%和34.5%。所提网络在提高低光照图像亮度的同时降低了噪声，减少了颜色失真和伪影，得到的增强图像更加清晰自然。

关键词: 低光照, 图像增强, 注意力机制, 双分支, 联合损失函数

CLC Number:

TP391.4

Jiazhen ZU, Yongxia ZHOU, Le CHEN. Dual-branch residual low-light image enhancement combined with attention[J]. Journal of Computer Applications, 2023, 43(4): 1240-1247.

祖佳贞, 周永霞, 陈乐. 结合注意力的双分支残差低光照图像增强[J]. 《计算机应用》唯一官方网站, 2023, 43(4): 1240-1247.

Figures/Tables 17

References 30

1	欧嘉敏，胡晓，杨佳信. 改进Retinex-Net的低光照图像增强算法［J］. 模式识别与人工智能， 2021， 34（1）： 77-86.
	OU J M， HU X， YANG J X. Low-light image enhancement algorithm based on improved Retinex-Net［J］. Pattern Recognition and Artificial Intelligence， 2021， 34（1）： 77-86.
2	LAND E H. The retinex theory of color vision［J］. Scientific American， 1977， 237（6）： 108-128. 10.1038/scientificamerican1277-108
3	JOBSON D J， RAHMAN Z， WOODELL G A. Properties and performance of a center/surround retinex［J］. IEEE Transactions on Image Processing， 1997， 6（3）： 451-462. 10.1109/83.557356
4	JOBSON D J， RAHMAN Z， WOODELL G A. A multiscale retinex for bridging the gap between color images and the human observation of scenes［J］. IEEE Transactions on Image Processing， 1997， 6（7）： 965-976. 10.1109/83.597272
5	卫依雪，周冬明，王长城，等. 结合多分支结构和U-net的低照度图像增强［J］. 计算机工程与应用， 2022， 58（12）： 199-208. 10.3778/j.issn.1002-8331.2011-0097
	WEI Y X， ZHOU D M， WANG C C， et al. Low-light image enhancement using multi-branch structure and U-net［J］. Computer Engineering and Applications， 2022， 58（12）： 199-208. 10.3778/j.issn.1002-8331.2011-0097
6	WANG S H， ZHENG J， HU H M， et al. Naturalness preserved enhancement algorithm for non-uniform illumination images［J］. IEEE Transactions on Image Processing， 2013， 22（9）： 3538-3548. 10.1109/tip.2013.2261309
7	FU X Y， LIAO Y H， ZENG D L， et al. A probabilistic method for image enhancement with simultaneous illumination and reflectance estimation［J］. IEEE Transactions on Image Processing， 2015， 24（12）： 4965-4977. 10.1109/tip.2015.2474701
8	GUO X J， LI Y， LING H B. LIME： low-light image enhancement via illumination map estimation［J］. IEEE Transactions on Image Processing， 2017， 26（2）： 982-993. 10.1109/tip.2016.2639450
9	FU X Y， ZENG D L， HUANG Y， et al. A fusion-based enhancing method for weakly illuminated images［J］. Signal Processing， 2016， 129： 82-96. 10.1016/j.sigpro.2016.05.031
10	王萌萌，彭敦陆. Retinex-ADNet：一个低光照图像增强系统［J］. 小型微型计算机系统， 2022， 43（2）： 367-371.
	WANG M M， PENG D L. Retinex-ADNet： a low-light image enhancement system［J］. Journal of Chinese Computer Systems， 2022， 43（2）： 367-371.
11	WEI C， WANG W J， YANG W H， et al. Deep Retinex decomposition for low-light enhancement［C］// Proceedings of the 2018 British Machine Vision Conference. Durham： BMVA Press， 2018： No.155. 10.48550/arXiv.1808.04560
12	WANG W J， WEI C， YANG W H， et al. GLADNet： low-light enhancement network with global awareness［C］// Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway： IEEE， 2018： 751-755. 10.1109/fg.2018.00118
13	LV F F， LU F， WU J H， et al. MBLLEN： low-light image/video enhancement using CNNs［C］// Proceedings of the 2018 British Machine Vision Conference. Durham： BMVA Press， 2018： No.220.
14	WANG R X， ZHANG Q， FU C W， et al. Underexposed photo enhancement using deep illumination estimation［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 6842-6850. 10.1109/cvpr.2019.00701
15	JIANG Y F， GONG X Y， LIU D， et al. EnlightenGAN： deep light enhancement without paired supervision［J］. IEEE Transactions on Image Processing， 2021， 30： 2340-2349. 10.1109/tip.2021.3051462
16	GUO C L， LI C Y， GUO J C， et al. Zero-reference deep curve estimation for low-light image enhancement［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 1777-1786. 10.1109/cvpr42600.2020.00185
17	RONNEBERGER O， FISCHER P， BROX T. U-net： convolutional networks for biomedical image segmentation［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9351. Cham： Springer， 2015： 234-241.
18	WANG L W， LIU Z S， SIU W C， et al. Lightening network for low-light image enhancement［J］. IEEE Transactions on Image Processing， 2020， 29： 7984-7996. 10.1109/tip.2020.3008396
19	SZEGEDY C， VANHOUCKE V， IOFFE S， et al. Rethinking the inception architecture for computer vision［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 2818-2826. 10.1109/cvpr.2016.308
20	WANG Q L， WU B G， ZHU P F， et al. ECA-Net： efficient channel attention for deep convolutional neural networks［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 11531-11539. 10.1109/cvpr42600.2020.01155
21	HU J， SHEN L， SUN G. Squeeze-and-excitation networks［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7132-7141. 10.1109/cvpr.2018.00745
22	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［EB/OL］. （2015-04-10）［2022-06-07］..
23	YANG W H， WANG W J， HUANG H F， et al. Sparse gradient regularized deep Retinex network for robust low-light image enhancement［J］. IEEE Transactions on Image Processing， 2021， 30： 2072-2086. 10.1109/tip.2021.3050850
24	DANG-NGUYEN D T， PASQUINI C， CONOTTER V， et al. RAISE： a raw images dataset for digital image forensics［C］// Proceedings of the 6th ACM Multimedia Systems Conference. New York： ACM， 2015： 219-224. 10.1145/2713168.2713194
25	LEE C， LEE C， KIM C S. Contrast enhancement based on layered difference representation［C］// Proceedings of the 2012 19th IEEE International Conference on Image Processing. Piscataway： IEEE， 2012： 965-968. 10.1109/icip.2012.6467022
26	MA K， ZENG K， WANG Z. Perceptual quality assessment for multi-exposure image fusion［J］. IEEE Transactions on Image Processing， 2015， 24（11）： 3345-3356. 10.1109/tip.2015.2442920
27	ZHANG L， ZHANG L， MOU X Q， et al. FSIM： a feature similarity index for image quality assessment［J］. IEEE Transactions on Image Processing， 2011， 20（8）： 2378-2386. 10.1109/tip.2011.2109730
28	MITTAL A， SOUNDARARAJAN R， BOVIK A C. Making a “completely blind” image quality analyzer［J］. IEEE Signal Processing Letters， 2012， 20（3）： 209-212. 10.1109/lsp.2012.2227726
29	LI M D， LIU J Y， YANG W H， et al. Structure-revealing low-light image enhancement via robust Retinex model［J］. IEEE Transactions on Image Processing， 2018， 27（6）： 2828-2841. 10.1109/tip.2018.2810539
30	YANG W H， WANG S Q， FANG Y M， et al. From fidelity to perceptual quality： a semi-supervised approach for low-light image enhancement［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 3060-3069. 10.1109/cvpr42600.2020.00313

模块组合	PSNR/dB	SSIM	模块组合	PSNR/dB	SSIM
M1	22.61	0.854	M5	18.52	0.803
M2	22.37	0.864	M6	21.23	0.825
M3	21.55	0.862	M7	22.48	0.865
M4	21.47	0.860	本文算法	22.73	0.870

模块组合	PSNR/dB	SSIM	模块组合	PSNR/dB	SSIM
M1	22.61	0.854	M5	18.52	0.803
M2	22.37	0.864	M6	21.23	0.825
M3	21.55	0.862	M7	22.48	0.865
M4	21.47	0.860	本文算法	22.73	0.870

损失组合名称	PSNR/dB	SSIM	损失组合名称	PSNR/dB	SSIM
L1	19.88	0.842	L3	22.11	0.861
L2	21.37	0.858	本文算法	22.73	0.870

损失组合名称	PSNR/dB	SSIM	损失组合名称	PSNR/dB	SSIM
L1	19.88	0.842	L3	22.11	0.861
L2	21.37	0.858	本文算法	22.73	0.870

算法	LOL			DICM	LIME	MEF
算法	PSNR/dB	SSIM	NIQE	NIQE
LIME	16.92	0.504	8.795	3.783	4.347	3.827
SRIE	12.28	0.509	7.716	3.173	3.469	3.204
MF	16.69	0.508	9.713	3.544	4.103	3.400
NPE	16.97	0.484	9.135	3.521	3.840	3.547
RRM	13.88	0.664	3.952	3.406	4.033	3.883
Retinex-Net	14.98	0.328	9.729	4.320	4.908	4.905
GLADNet	16.94	0.552	6.797	3.114	3.404	3.180
MBLLEN	17.90	0.694	4.266	2.665	3.647	3.170
DeepUPE	12.90	0.465	7.994	2.954	3.462	3.178
EnlightenGAN	17.56	0.666	4.815	2.530	3.295	3.053
Zero-DCE	14.86	0.562	8.223	2.828	3.789	3.309
DLN	21.95	0.807	3.656	2.734	3.601	3.523
本文算法	20.80	0.822	3.638	2.523	3.255	2.990