Hardware reconstruction acceleration method of convolutional neural network-based single image defogging model

doi:10.11772/j.issn.1001-9081.2021081475

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (10): 3184-3190.DOI: 10.11772/j.issn.1001-9081.2021081475

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Hardware reconstruction acceleration method of convolutional neural network-based single image defogging model

Guanjun WANG¹, Chunlian JIAN², Qiang XIANG¹

^1.College of Electronic and Information，Southwest Minzu University，Chengdu Sichuan 610041，China
^2.Chengdu Powerview Science and Technology Company Limited，Chengdu Sichuan 610046，China

Received:2021-08-18 Revised:2021-11-22 Accepted:2021-11-24 Online:2022-01-07 Published:2022-10-10
Contact: Qiang XIANG
About author:WANG Guanjun, born in 1990， M. S. candidate. His research interests include image processing， computer vision， embedded system.
JIAN Chunlian, born in 1988， engineer. Her research interests include image processing， computer vision.
XIANG Qiang, born in 1973， Ph. D. ， associate professor. His research interests include signal and information processing， embedded system.
Supported by:
Key Research and Development of Science and Technology Department of Sichuan Province(22ZDYF3125);Innovative Scientific Research Project for Graduates of Southwest Minzu University(CX2021SZ46)

基于卷积神经网络的单图像去雾模型硬件重构加速方法

王官军¹, 简春莲², 向强¹

^1.西南民族大学电子信息学院，成都 610041
^2.成都动力视讯科技股份有限公司，成都 610046

通讯作者: 向强
作者简介:第一联系人：王官军（1990—），男，四川内江人，硕士研究生，主要研究方向：图像处理、计算机视觉、嵌入式系统
简春莲（1988—），女，四川达州人，工程师，主要研究方向：图像处理、计算机视觉
向强（1973—），男，四川营山人，副教授，博士，主要研究方向：信号与信息处理、嵌入式系统。xiangqiang@swun.edu.cn
基金资助:
四川省科技厅重点研发项目(22ZDYF3125);西南民族大学研究生创新型科研项目(CX2021SZ46)

Abstract

Abstract:

Single image defogging model based on Convolutional Neural Network （CNN） was difficult to deploy on mobile/embedded system and used for real-time video defogging. To solve this problem， a method of hardware reconstruction and acceleration was proposed， based on Zynq System-on-Chip （SoC）. First， a quantization-dequantization algorithm was proposed to perform quantization on two representative defogging models； second， a quantized defogging model was reconstructed and a hardware IP core with Advanced eXtensible Interface 4 （AXI4） was generated， based on video stream memory architecture， hardware/software co-design， pipeline technology and High-Level Synthesis （HLS） tool. Experimental results show that the model parameters can be quantified from float32 to int5（5 bit） under premise of defogging performance， saving about 84.4% of storage space； the highest pixel clock frequency of the generated hardware IP core is 182 Mpixel/s， which can achieve 1080P@60 frame/s video defogging； the hardware IP core processes a single hazy image with the resolution of 640 pixel × 480 pixel only in 2.4 ms， and the on-chip power consumption is only 2.25 W. This hardware IP core with AXI4 is also convenient for cross-platform migration and deployment， which can expand application scope of CNN-based single image defogging model.

Key words: defogging, Video Direct Memory Access (VDMA), model quantization, model reconstruction, hardware IP core, High-Level Synthesis (HLS)

摘要：

针对基于卷积神经网络（CNN）的单图像去雾模型在移动/嵌入式端部署难，不易用做实时视频去雾的问题，提出一种基于Zynq片上系统（SoC）的去雾模型硬件重构加速方法。首先，提出量化?反量化算法，对两个代表去雾模型进行量化；其次，基于视频流存储器架构和软硬件协同、流水线等技术以及高级综合（HLS）工具，对量化后的去雾模型硬件重构并生成具有高性能扩展总线接口（AXI4）的硬件IP核。实验结果表明，在保证去雾效果的前提下，可以实现模型参数从float32到int5（5 bit）的量化，从而节省约84.4%的存储空间；所生成硬件IP核的最高像素时钟频率为182 Mpixel/s，能够实现1080P@60 frame/s的视频去雾；单帧640×480的雾图去雾仅需2.4 ms，而片上功耗仅为2.25 W。这种生成带有标准总线接口的硬件IP核也便于跨平台移植和部署，从而可以扩大这类去雾模型的应用范围。

关键词: 去雾, 视频直接存储器访问, 模型量化, 模型重构, 硬件IP核, 高级综合

CLC Number:

TP391.4

Guanjun WANG, Chunlian JIAN, Qiang XIANG. Hardware reconstruction acceleration method of convolutional neural network-based single image defogging model[J]. Journal of Computer Applications, 2022, 42(10): 3184-3190.

王官军, 简春莲, 向强. 基于卷积神经网络的单图像去雾模型硬件重构加速方法[J]. 《计算机应用》唯一官方网站, 2022, 42(10): 3184-3190.

Figures/Tables 12

References 24

1	PARIHAR A S， GUPTA Y K， SINGODIA Y， et al. A comparative study of image dehazing algorithms［C］// Proceedings of the 5th International Conference on Communication and Electronics Systems. Piscataway： IEEE， 2020： 766-771. 10.1109/icces48766.2020.9138037
2	HE K M， SUN J， TANG X O. Single image haze removal using dark channel prior［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2011， 33（12）： 2341-2353. 10.1109/tpami.2010.168
3	王道累，张天宇. 图像去雾算法的综述及分析［J］. 图学学报， 2020， 41（6）： 861-870. 10.11996/JG.j.2095-302X.2020060861
	WANG D L， ZHANG T Y. Review and analysis of image defogging algorithm［J］. Journal of Graphics， 2020， 41（6）： 861-870. 10.11996/JG.j.2095-302X.2020060861
4	LI B， ZHAO J J， FU H. DLT-Net： deep learning transmittance network for single image haze removal［J］. Signal， Image and Video Processing， 2020， 14（6）： 1245-1253. 10.1007/s11760-020-01665-9
5	SALAZAR-COLORES S， CRUZ-ACEVES I， RAMOS-ARREGUIN J M. Single image dehazing using a multilayer perceptron［J］. Journal of Electronic Imaging， 2018， 27（4）： No.043022. 10.1117/1.jei.27.4.043022
6	CAI B L， XU X M， JIA K， et al. DehazeNet： an end-to-end system for single image haze removal［J］. IEEE Transactions on Image Processing， 2016， 25（11）： 5187-5198. 10.1109/tip.2016.2598681
7	余春艳，林晖翔，徐小丹，等. 雾天退化模型参数估计与CUDA设计［J］. 计算机辅助设计与图形学学报， 2018， 30（2）： 327-335. 10.3724/sp.j.1089.2018.16288
	YU C Y， LIN H X， XU X D， et al. Parameter estimation of fog degradation model and CUDA design［J］. Journal of Computer-Aided Design and Computer Graphics. 2018， 30（2）： 327-335. 10.3724/sp.j.1089.2018.16288
8	LI B Y， PENG X L， WANG Z Y， et al. AOD-Net： all-in-one dehazing network ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 4780-4788. 10.1109/iccv.2017.511
9	QIAN W， ZHOU C， ZHANG D Y. FAOD-Net： a fast AOD-Net for dehazing single image［J］. Mathematical Problems in Engineering， 2020， 2020： No.4945214. 10.1155/2020/4945214
10	ZHANG J， TAO D C. FAMED-Net： a fast and accurate multi-scale end-to-end dehazing network［J］. IEEE Transactions on Image Processing， 2020， 29： 72-84. 10.1109/tip.2019.2922837
11	CHEN D D， HE M M， FAN Q N， et al. Gated context aggregation network for image dehazing and deraining［C］// Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2019： 1375-1383. 10.1109/wacv.2019.00151
12	张婷，赵杏，陈文欣. 基于条件生成对抗网络的图像去雾方法［J］. 计算机应用， 2021， 41（S2）：248-253.
	ZHANG T， ZHAO X， CHEN W X. Image dehazing method based on conditional generative adversarial network［J］. Journal of Computer Applications， 2021， 41（S2）：248-253.
13	张春雷，徐润，王郁杰，等. 基于FPGA的视频实时去雾算法及其硬件实现［J］. 半导体光电， 2021， 42（2）： 264-268， 274. 10.16818/j.issn1001-5868.2021.02.020
	ZHANG C L， XU R， WANG Y J， et al. FPGA-based video real-time dehazing algorithm and hardware implementation［J］. Semiconductor Optoelectronics， 2021， 42（2）： 264-268， 274. 10.16818/j.issn1001-5868.2021.02.020
14	VARALAKSHMI J， JOSE D， KUMAR P N. FPGA implementation of haze removal technique based on dark channel prior［C］// Proceedings of the 2019 International Conference on Computational Vision and Bio Inspired Computing， AISC 1108. Cham： Springer， 2020： 624-630.
15	LU J Z， DONG C. DSP-based image real-time dehazing optimization for improved dark-channel prior algorithm［J］. Journal of Real-Time Image Processing， 2020， 17（5）： 1675-1684. 10.1007/s11554-019-00933-3
16	陆斌，严利民，陈志恒. 硬件优化的高清视频实时去雾算法［J］. 计算机应用研究， 2020， 37（12）： 3807-3810.
	LU B， YAN L M， CHEN Z H. Hardware optimized real-time dehazing algorithm for high-definition video［J］. Application Research of Computers， 2020， 37（12）： 3807- 3810.
17	齐乐，张小刚，姚航. 基于HLS的实时图像去雾实现［J］. 计算机工程， 2016， 42（5）： 224-229. 10.3969/j.issn.1000-3428.2016.05.038
	QI L， ZHANG X G， YAO H. Real-time image dehazing realization based on HLS［J］. Computer Engineering， 2016， 42（5）： 224-229. 10.3969/j.issn.1000-3428.2016.05.038
18	JACOB B， KLIGYS S， CHEN B， et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 2704-2713. 10.1109/cvpr.2018.00286
19	KRISHNAMOORTHI R. Quantizing deep convolutional networks for efficient inference： a whitepaper［EB/OL］. （2018-06-21）［2020-01-29］..
20	XILINX. AXI Video Direct Memory Access v 6.3： LogiCORE IP product guide： Vivado design suite （PG020）［EB/OL］. （2022-06-08）［2022-08-02］..
21	XILINX. Vivado design suite tutorial： high-level synthesis （UG871 （v2020.1））［EB/OL］. （2020-08-07）［2021-03-23］.. 10.1109/access.2021.3067453
22	XILINX. Vivado design suite user guide： high-level synthesis （UG 902）［EB/OL］. （2021-05-04）［2021-06-01］.. 10.2172/1375449
23	NVDIA. 8 bit inference with TensorRT［EB/OL］. ［2021-05-23］.. 10.1109/iiswc53511.2021.00030
24	陈瑞. 基于卷积神经网络的图像去雾算法研究及FPGA实现［D］. 西安：西安理工大学， 2020：42-43. 10.5768/jao202041.0202001
	CHEN R. Research and FPGA implementation of image haze removal algorithm based on convolutional neural networks［D］. Xi’an： Xi’an University of Technology， 2020：42-43. 10.5768/jao202041.0202001

数据类型	均值		标准差		平均梯度
数据类型	DehazeNet	AOD-Net	DehazeNet	AOD-Net	DehazeNet	AOD-Net
float32	111.75	103.72	63.56	59.28	11.02	10.60
8 bit	0.55	-0.02	-0.01	0.00	0.05	0.00
7 bit	-1.80	-0.02	1.03	0.11	0.15	0.00
6 bit	2.25	-1.88	-0.17	0.58	-0.18	0.07
5 bit	-0.04	1.25	2.37	-0.76	-0.30	-0.02
4 bit	-9.41	8.85	8.08	-3.65	-0.43	-0.58
3 bit	9.11	-13.96	-11.06	3.22	0.05	0.56
2 bit	-27.57	-33.67	21.51	5.43	-5.88	-0.39
数据类型	MSE		PSNR/dB		SSIM
数据类型	DehazeNet	AOD-Net	DehazeNet	AOD-Net	DehazeNet	AOD-Net
float32	—	—	—	—	—	—
8 bit	0.96	0.32	24.71	27.81	0.99	0.99
7 bit	3.21	0.55	21.88	26.13	0.98	0.99
6 bit	5.02	2.46	21.23	22.36	0.98	0.99
5 bit	12.44	2.06	18.92	23.07	0.95	0.98
4 bit	127.68	35.86	13.98	16.53	0.80	0.94
3 bit	307.04	81.13	12.05	14.72	0.83	0.88
2 bit	1 454.90	483.89	8.78	11.00	0.42	0.56

数据类型	均值		标准差		平均梯度
数据类型	DehazeNet	AOD-Net	DehazeNet	AOD-Net	DehazeNet	AOD-Net
float32	111.75	103.72	63.56	59.28	11.02	10.60
8 bit	0.55	-0.02	-0.01	0.00	0.05	0.00
7 bit	-1.80	-0.02	1.03	0.11	0.15	0.00
6 bit	2.25	-1.88	-0.17	0.58	-0.18	0.07
5 bit	-0.04	1.25	2.37	-0.76	-0.30	-0.02
4 bit	-9.41	8.85	8.08	-3.65	-0.43	-0.58
3 bit	9.11	-13.96	-11.06	3.22	0.05	0.56
2 bit	-27.57	-33.67	21.51	5.43	-5.88	-0.39
数据类型	MSE		PSNR/dB		SSIM
数据类型	DehazeNet	AOD-Net	DehazeNet	AOD-Net	DehazeNet	AOD-Net
float32	—	—	—	—	—	—
8 bit	0.96	0.32	24.71	27.81	0.99	0.99
7 bit	3.21	0.55	21.88	26.13	0.98	0.99
6 bit	5.02	2.46	21.23	22.36	0.98	0.99
5 bit	12.44	2.06	18.92	23.07	0.95	0.98
4 bit	127.68	35.86	13.98	16.53	0.80	0.94
3 bit	307.04	81.13	12.05	14.72	0.83	0.88
2 bit	1 454.90	483.89	8.78	11.00	0.42	0.56

硬件资源名	实际消耗数	资源总数/10²	资源消耗率/%
BRAM_18K	114	10	11
DSP48E	540	9	60
FF	136 044	3 438	39
LUT	115 343	1 719	67

硬件资源名	实际消耗数	资源总数/10²	资源消耗率/%
BRAM_18K	114	10	11
DSP48E	540	9	60
FF	136 044	3 438	39
LUT	115 343	1 719	67

硬件资源名	实际消耗数	资源总数/10²	资源消耗率/%
BRAM_18K	114	10	11
DSP48E	551	9	61
FF	124 686	3 438	36
LUT	122 049	1 719	71

Hardware reconstruction acceleration method of convolutional neural network-based single image defogging model

基于卷积神经网络的单图像去雾模型硬件重构加速方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 24

Related Articles 11

Recommended Articles

Metrics

去雾模型	平均时间	去雾模型	平均时间
AOD-Net^［8］	0.650 0	GCA-Net^［11］	0.230 0
Dehaze-Net^［6］	5.090 0	文献［7］模型	0.048 9
FAOD-Net^［9］	0.340 0	硬件IP核的AOD-Net模型	0.002 4
FAMED-Net^［10］	0.890 0

[1]	ZHANG Jiangxin, ZHOU Jiabo, MENG Limin. Improved fast image defogging algorithm based on dark channel prior [J]. Journal of Computer Applications, 2017, 37(8): 2324-2328.
[2]	ZHANG Wang, JIA Jia, MENG Yuan, BAI Xu. Research and design of AES algorithm based on high-level synthesis [J]. Journal of Computer Applications, 2017, 37(5): 1341-1346.
[3]	LI Huihui, QIN Pinle, LIANG Jun. Defogging algorithm based on HSI luminance component and RGB space [J]. Journal of Computer Applications, 2016, 36(5): 1378-1382.
[4]	MA Qian, GE Baozhen, CHEN Lei. Correction technique for color difference of multi-sensor texture [J]. Journal of Computer Applications, 2016, 36(4): 1075-1079.
[5]	WANG Fan, YANG Yan, BAI Haiping. Contrast restoration algorithm for single image based on physicals model [J]. Journal of Computer Applications, 2015, 35(8): 2291-2294.
[6]	ZHANG Jingjing, CHEN Zihong, ZHANG Dexiang, YAN Qing, XUN Lina, ZHANG Weiguo. Polarized image dehazing algorithm based on dark channel prior [J]. Journal of Computer Applications, 2015, 35(12): 3576-3580.
[7]	WANG Jianxin ZHANG Youhui WANG Zhiwei ZHANG Jing LI Juan. Single image defogging algorithm based on HSI color space [J]. Journal of Computer Applications, 2014, 34(10): 2990-2995.
[8]	LIU Huan ZHU Ping XIAO Rong TANG Weidong. Belly shape modeling with new combined invariant moment based on stereo vision [J]. Journal of Computer Applications, 2013, 33(11): 3183-3186.
[9]	LIU Jin. Adaptive approach for point cloud based CAD model reconstruction [J]. Journal of Computer Applications, 2013, 33(09): 2617-2622.
[10]	. Low-cost 3D human reconstruction from multi-image [J]. Journal of Computer Applications, 2009, 29(07): 1910-1913.
[11]	. Interactive defogging method for image [J]. Journal of Computer Applications, 2006, 26(11): 2733-2735.