Dual-branch network guided by local entropy for dynamic scene high dynamic range imaging

doi:10.11772/j.issn.1001-9081.2023121726

Abstract

Abstract:

For addressing the issues of motion artifacts and exposure distortion in High Dynamic Range （HDR） imaging tasks based on a sequence of multiple exposed images when there is camera shake or subject movement， a dual-branch network guided by local entropy for dynamic scene HDR imaging was proposed. Firstly， the Discrete Wavelet Transform （DWT） was employed to separate the low-frequency illumination-related information and high-frequency motion-related information from the input images， enabling the network to address exposure and subject movement purposefully. Secondly， for the low-frequency illumination-related information branch， a module was designed to calculate attention using image local entropy， thereby guiding the network to reduce the extraction of exposure features lacking details. For the high-frequency motion-related information branch， a lightweight feature alignment module was introduced for consistent alignment of scene， thereby reducing the extraction of motion features. Finally， a time-domain self-attention module was constructed by integrating channel attention， thereby enhancing the mutual dependence of exposure image sequence in temporal domain， so as to further improve the quality of the results. Evaluation was performed on public datasets Kalantari， Sen， and Tursun. Experimental results on Kalantari dataset show that the proposed network achieves the first place in PSNR-l （42.20 dB） and the third place in SSIM-l （0.988 9） compared to some latest methods. By integrating experimental results on the remaining datasets， it can be seen that the proposed network can reduce exposure distortion and motion artifacts effectively， and generate images with abundant details and excellent visual effect.

Key words: High Dynamic Range (HDR) imaging, local entropy, attention mechanism, Discrete Wavelet Transform (DWT), image information separation

摘要：

针对基于多张曝光图像序列的高动态范围（HDR）成像任务在相机抖动或拍摄主体移动时出现运动伪影以及曝光失真的问题，提出一个用于动态场景HDR成像的局部熵引导的双分支网络。首先，利用离散小波变换（DWT）分离出输入图像的低频光照相关信息以及高频运动相关信息，以便于网络有针对性地处理曝光以及主体移动；其次，对于低频光照相关信息分支，设计一个利用图像局部熵计算注意力的模块来引导网络减少细节不足的曝光特征的提取；对于高频运动相关信息分支，引入一个轻量级的特征对齐模块来进行场景的一致性对齐，从而减少运动特征的提取；最后，结合通道注意力构建时域自注意力模块，从而加强曝光图像序列在时间域之间的相互依赖关系，以进一步提高结果质量。在公开数据集Kalantari、Sen、Tursun上进行评估。在Kalantari数据集上的实验结果表明，与最新的一些方法对比，所提网络以PSNR-l为42.20 dB的成绩取得第一，SSIM-l为0.988 9的成绩取得第三。结合其余数据集上的实验结果可知，所提网络可以有效减少曝光失真以及运动伪影，并生成细节多、视觉效果佳的图像。

关键词: 高动态范围成像, 局部熵, 注意力机制, 离散小波变换, 图像信息分离

CLC Number:

TP391.413

Ying HUANG, Changsheng LI, Hui PENG, Su LIU. Dual-branch network guided by local entropy for dynamic scene high dynamic range imaging[J]. Journal of Computer Applications, 2025, 45(1): 204-213.

黄颖, 李昌盛, 彭慧, 刘苏. 用于动态场景高动态范围成像的局部熵引导的双分支网络[J]. 《计算机应用》唯一官方网站, 2025, 45(1): 204-213.

Figures/Tables 16

References 45

1	FROEHLICH J， GRANDINETTI S， EBERHARDT B， et al. Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays ［C］// Proceedings of the SPIE 9023， Digital Photography X. Bellingham， WA： SPIE， 2014： No.90230X.
2	TOCCI M D， KISER C， TOCCI N， et al. A versatile HDR video production system ［J］. ACM Transactions on Graphics， 2011， 30（4）： No.41.
3	刘颖，王凤伟，刘卫华，等.基于亮度分区模糊融合的高动态范围成像算法［J］.计算机应用， 2020， 40（1）： 233-238.
	LIU Y， WANG F W， LIU W H， et al. High dynamic range imaging algorithm based on luminance partition fuzzy fusion ［J］. Journal of Computer Applications， 2020， 40（1）： 233-238.
4	MA K， DUANMU Z， ZHU H， et al. Deep guided learning for fast multi-exposure image fusion ［J］. IEEE Transactions on Image Processing， 2020， 29： 2808-2819.
5	MA K， LI H， YONG H， et al. Robust multi-exposure image fusion： a structural patch decomposition approach ［J］. IEEE Transactions on Image Processing， 2017， 26（5）： 2519-2532.
6	PRABHAKAR K R， SRIKAR V S， BABU R V. DeepFuse： a deep unsupervised approach for exposure fusion with extreme exposure image pairs ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 4724-4732.
7	WANG L， YOON K J. Deep learning for HDR imaging： state-of-the-art and future trends ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 44（12）： 8874-8895.
8	KALANTARI N K， RAMAMOORTHI R. Deep high dynamic range imaging of dynamic scenes ［J］. ACM Transactions on Graphics， 2017， 36（4）： No.144.
9	PRABHAKAR K R， ARORA R， SWAMINATHAN A， et al. A fast， scalable， and reliable deghosting method for extreme exposure fusion ［C］// Proceedings of the 2019 IEEE International Conference on Computational Photography. Piscataway： IEEE， 2019： 1-8.
10	DAI J， QI H， XIONG Y， et al. Deformable convolutional networks ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 764-773.
11	TAN X， CHEN H， XU K， et al. High dynamic range imaging for dynamic scenes with large-scale motions and severe saturation ［J］. IEEE Transactions on Instrumentation and Measurement， 2022， 71： No.5003415.
12	MARÍN-VEGA J， SLOTH M， SCHNEIDER-KAMP P， et al. DRHDR： a dual branch residual network for multi-bracket high dynamic range imaging ［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2022： 843-851.
13	HASINOFF S W， SHARLET D， GEISS R， et al. Burst photography for high dynamic range and low-light imaging on mobile cameras ［J］. ACM Transactions on Graphics， 2016， 35（6）： No.192.
14	POULI T， BOITARD R， CHAMARET C， et al. HDR in the living room ［C］// ACM SIGGRAPH 2014 Studio. New York： ACM， 2014： No.5.
15	WANG X， YU K， DONG C， et al. Recovering realistic texture in image super-resolution by deep spatial feature transform ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 606-615.
16	BOGONI L. Extending dynamic range of monochrome and color images through fusion ［C］// Proceedings of the 15th International Conference on Pattern Recognition， Volume 3. Piscataway： IEEE， 2000： 7-12.
17	SEN P， KALANTARI N K， YAESOUBI M， et al. Robust patch-based HDR reconstruction of dynamic scenes ［J］. ACM Transactions on Graphics， 2012， 31（6）： No.203.
18	ZIMMER H， BRUHN A， WEICKERT J. Freehand HDR imaging of moving scenes with simultaneous resolution enhancement ［J］. Computer Graphics Forum， 2011， 30（2）： 405-414.
19	HU J， GALLO O， PULLI K， et al. HDR deghosting： how to deal with saturation？［C］// Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2013： 1163-1170.
20	WU S， XU J， TAI Y W， et al. Deep high dynamic range imaging with large foreground motions ［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11206. Cham： Springer， 2018： 120-135.
21	WANG X， GIRSHICK R， GUPTA， et al. Non-local neural networks ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7794-7803.
22	YAN Q， ZHANG L， LIU Y， et al. Deep HDR imaging via a non-local network ［J］. IEEE Transactions on Image Processing， 2020， 29： 4308-4322.
23	NIU Y， WU J， LIU W， et al. HDR-GAN： HDR image reconstruction from multi-exposed LDR images with large motions ［J］. IEEE Transactions on Image Processing， 2021， 30： 3885-3896.
24	LAND E H， McCANN J J. Lightness and Retinex theory ［J］. Journal of the Optical Society of America， 1971， 61（1）： 1-11.
25	WEI C， WANG W， YANG W， et al. Deep Retinex decomposition for low-light enhancement ［C］// Proceedings of the 2018 British Machine Vision Conference. Durham： BMVA Press， 2018： No.451.
26	WU K， CHEN J， MA J. DMEF： multi-exposure image fusion based on a novel deep decomposition method ［J］. IEEE Transactions on Multimedia， 2023， 25： 5690-5703.
27	王克强，张雨帅，王保群.基于Retinex理论的多曝光图像融合算法［J］.计算机应用， 2019， 39（7）： 2087-2092.
	WANG K Q， ZHANG Y S， WANG B Q. Multi-exposure image fusion algorithm based on Retinex theory ［J］. Journal of Computer Applications， 2019， 39（7）： 2087-2092.
28	SINGH S， SINGH H， GEHLOT A， et al. IR and visible image fusion using DWT and bilateral filter ［J］. Microsystem Technologies， 2023， 29（4）： 457-467.
29	LI Q， SHEN L， GUO S， et al. WaveCNet： wavelet integrated CNNs to suppress aliasing effect for noise-robust image classification ［J］. IEEE Transactions on Image Processing， 2021， 30： 7074-7089.
30	XU J， YUAN M， YAN D M， et al. Illumination guided attentive wavelet network for low-light image enhancement ［J］. IEEE Transactions on Multimedia， 2023， 25： 6258-6271.
31	JI Z， JUNG C. Subband adaptive enhancement of low light images using wavelet-based convolutional neural networks ［C］// Proceedings of the 2021 IEEE International Conference on Image Processing. Piscataway： IEEE， 2021： 1669-1673.
32	FU J， LIU J， TIAN H， et al. Dual attention network for scene segmentation ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3141-3149.
33	ZHAO H， JIA J， KOLTUN V. Exploring self-attention for image recognition ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 10073-10082.
34	刘辉，张琳玉，王复港，等.基于注意力机制和上下文信息的目标检测算法［J］.计算机应用， 2023， 43（5）： 1557-1564.
	LIU H， ZHANG L Y， WANG F G， et al. Object detection algorithm based on attention mechanism and context information ［J］. Journal of Computer Applications， 2023， 43（5）： 1557-1564.
35	DAI W， WANG K. An image edge detection algorithm based on local entropy ［C］// Proceedings of the 2007 IEEE International Conference on Integration Technology. Piscataway： IEEE， 2007： 418-420.
36	YAN Q， GONG D， SHI Q， et al. Attention-guided network for ghost-free high dynamic range imaging ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 1751-1760.
37	LIU L， LIU J， YUAN S， et al. Wavelet-based dual-branch network for image demoiréing ［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12358. Cham： Springer， 2020： 86-102.
38	LIU Z， LIN Y， CAO Y， et al. Swin Transformer： hierarchical vision Transformer using shifted windows ［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2021： 9992-10002.
39	LIANG J， CAO J， SUN G， et al. SwinIR： image restoration using Swin Transformer ［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway： IEEE， 2021： 1833-1844.
40	YE Q， XIAO J， LAM K M， et al. Progressive and selective fusion network for high dynamic range imaging ［C］// Proceedings of the 29th ACM International Conference on Multimedia. New York： ACM， 2021： 5290-5297.
41	HUANG Y M， CHIANG J C， CHEN S G. HDR-AGAN： ghost-free high dynamic range imaging with attention guided adversarial network ［C］// Proceedings of the 2022 IEEE International Conference on Image Processing. Piscataway： IEEE， 2022： 3316-3320.
42	SONG J W， PARK Y I， KONG K， et al. Selective TransHDR： Transformer-based selective HDR imaging using ghost region mask ［C］// Proceedings of the 2022 European Conference on Computer Vision， LNCS 13677. Cham： Springer， 2022： 288-304.
43	LIU Z， WANG Y， ZENG B， et al. Ghost-free high dynamic range imaging with context-aware Transformer ［C］// Proceedings of the 2022 European Conference on Computer Vision， LNCS 13679. Cham： Springer， 2022： 344-360.
44	REN H， FAN Y， HUANG S. Robust real-world image enhancement based on multi-exposure LDR images ［C］// Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2023： 1715-1723.
45	TURSUN O T， AKYÜZ A O， ERDEM A， et al. An objective deghosting quality metric for HDR images ［J］. Computer Graphics Forum， 2016， 35（2）： 139-152.

网络	评价指标				计算时间/s	参数量/10⁶
网络	PSNR-μ/dB	PSNR-l/dB	SSIM-μ	SSIM-l	计算时间/s	参数量/10⁶
文献［17］网络	40.80	38.11	0.980 8	0.972 1	73.96	0.00
文献［19］网络	35.79	30.76	0.971 7	0.950 3	—	—
文献［8］网络	42.67	41.23	0.988 8	0.984 6	32.79	0.38
文献［20］网络	41.65	40.88	0.986 0	0.985 8	0.18	20.40
文献［22］网络	42.41	41.43	0.987 7	0.985 7	0.16	38.10
文献［36］网络	43.63	41.14	0.990 0	0.970 2	0.53	1.52
文献［23］网络	43.92	41.57	0.990 5	0.986 5	0.26	2.56
文献［40］网络	44.06	41.57	0.990 7	0.986 7	—	—
文献［41］网络	43.05	41.33	0.989 6	0.986 6	—	—
文献［11］网络	43.96	41.67	0.9929	0.9906	0.60	7.46
文献［42］网络	44.09	41.70	0.990 9	0.987 2	—	—
文献［43］网络	44.32	42.18	0.991 6	0.988 4	0.16	1.22
文献［44］网络	44.63	42.12	0.993 2	0.991 0	—	—
本文网络	43.43	42.20	0.991 4	0.988 9	1.13	2.35

网络	评价指标				计算时间/s	参数量/10⁶
网络	PSNR-μ/dB	PSNR-l/dB	SSIM-μ	SSIM-l	计算时间/s	参数量/10⁶
文献［17］网络	40.80	38.11	0.980 8	0.972 1	73.96	0.00
文献［19］网络	35.79	30.76	0.971 7	0.950 3	—	—
文献［8］网络	42.67	41.23	0.988 8	0.984 6	32.79	0.38
文献［20］网络	41.65	40.88	0.986 0	0.985 8	0.18	20.40
文献［22］网络	42.41	41.43	0.987 7	0.985 7	0.16	38.10
文献［36］网络	43.63	41.14	0.990 0	0.970 2	0.53	1.52
文献［23］网络	43.92	41.57	0.990 5	0.986 5	0.26	2.56
文献［40］网络	44.06	41.57	0.990 7	0.986 7	—	—
文献［41］网络	43.05	41.33	0.989 6	0.986 6	—	—
文献［11］网络	43.96	41.67	0.9929	0.9906	0.60	7.46
文献［42］网络	44.09	41.70	0.990 9	0.987 2	—	—
文献［43］网络	44.32	42.18	0.991 6	0.988 4	0.16	1.22
文献［44］网络	44.63	42.12	0.993 2	0.991 0	—	—
本文网络	43.43	42.20	0.991 4	0.988 9	1.13	2.35

变体	评价指标
变体	PSNR-μ/dB	PSNR-l/dB	SSIM-μ	SSIM-l
基线	42.98	41.59	0.991 1	0.987 6
变体1	43.18	41.99	0.990 8	0.987 6
变体2	43.38	42.06	0.991 4	0.988 0
变体3	43.14	41.94	0.991 3	0.987 5
变体4	43.40	41.90	0.991 4	0.988 7
变体5	43.43	42.20	0.991 4	0.988 9

变体	评价指标
变体	PSNR-μ/dB	PSNR-l/dB	SSIM-μ	SSIM-l
基线	42.98	41.59	0.991 1	0.987 6
变体1	43.18	41.99	0.990 8	0.987 6
变体2	43.38	42.06	0.991 4	0.988 0
变体3	43.14	41.94	0.991 3	0.987 5
变体4	43.40	41.90	0.991 4	0.988 7
变体5	43.43	42.20	0.991 4	0.988 9

[1]	Jie XU, Yong ZHONG, Yang WANG, Changfu ZHANG, Guanci YANG. Facial attribute estimation and expression recognition based on contextual channel attention mechanism [J]. Journal of Computer Applications, 2025, 45(1): 253-260.
[2]	Junying CHEN, Shijie GUO, Lingling CHEN. Lightweight human pose estimation based on decoupled attention and ghost convolution [J]. Journal of Computer Applications, 2025, 45(1): 223-233.
[3]	Lifang WANG, Jingshuang WU, Pengliang YIN, Lihua HU. Action recognition algorithm based on attention mechanism and energy function [J]. Journal of Computer Applications, 2025, 45(1): 234-239.
[4]	Jialin ZHANG, Qinghua REN, Qirong MAO. Speaker verification system utilizing global-local feature dependency for anti-spoofing [J]. Journal of Computer Applications, 2025, 45(1): 308-317.
[5]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[6]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[7]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[8]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.
[9]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[10]	Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594.
[11]	Shangbin MO, Wenjun WANG, Ling DONG, Shengxiang GAO, Zhengtao YU. Single-channel speech enhancement based on multi-channel information aggregation and collaborative decoding [J]. Journal of Computer Applications, 2024, 44(8): 2611-2617.
[12]	Wu XIONG, Congjun CAO, Xuefang SONG, Yunlong SHAO, Xusheng WANG. Handwriting identification method based on multi-scale mixed domain attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2225-2232.
[13]	Huanhuan LI, Tianqiang HUANG, Xuemei DING, Haifeng LUO, Liqing HUANG. Public traffic demand prediction based on multi-scale spatial-temporal graph convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2065-2072.
[14]	Dianhui MAO, Xuebo LI, Junling LIU, Denghui ZHANG, Wenjing YAN. Chinese entity and relation extraction model based on parallel heterogeneous graph and sequential attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2018-2025.
[15]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.