基于生成对抗网络的红外图像数据增强

doi:10.11772/j.issn.1001-9081.2019122253

计算机应用 ›› 2020, Vol. 40 ›› Issue (7): 2084-2088.DOI: 10.11772/j.issn.1001-9081.2019122253

• 虚拟现实与多媒体计算 • 上一篇下一篇

基于生成对抗网络的红外图像数据增强

陈佛计^1,2,3,4, 朱枫^1,2,3,4, 吴清潇^1,2,3,4, 郝颖明^1,2,3,4, 王恩德^1,2,3,4

1. 中国科学院沈阳自动化研究所, 沈阳 110016;
2. 中国科学院机器人与智能制造创新研究院, 沈阳 110016;
3. 中国科学院大学, 北京 100049;
4. 中国科学院光电信息处理重点实验室, 沈阳 110016

收稿日期:2020-01-09 修回日期:2020-03-01 出版日期:2020-07-10 发布日期:2020-03-29
通讯作者: 陈佛计
作者简介:陈佛计(1994-),男,山西忻州人,硕士,主要研究方向:图像生成、机器学习、机器人视觉;朱枫(1962-),男,辽宁沈阳人,研究员,博士生导师,博士,主要研究方向:机器人视觉;吴清潇(1978-),男,辽宁沈阳人,研究员,博士,主要研究方向:机器人视觉、机器视觉;郝颖明(1966-),女,辽宁沈阳人,研究员,博士,主要研究方向:视觉定位、图像优化、视觉检测、红外图像仿真、三维成像、数据处理;王恩德(1980-),男,辽宁沈阳人,研究员,博士,主要研究方向:图像目标识别、检测与跟踪、微弱信号检测。
基金资助:
国家自然科学基金资助项目（U1713216）。

Infrared image data augmentation based on generative adversarial network

CHEN Foji^1,2,3,4, ZHU Feng^1,2,3,4, WU Qingxiao^1,2,3,4, HAO Yingming^1,2,3,4, WANG Ende^1,2,3,4

1. Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang Liaoning 110016, China;
2. Institutes for Robotics and Intelligent Manufacturing Innovation, Chinese Academy of Sciences, Shenyang Liaoning 110016, China;
3. University of Chinese Academy of Sciences, Beijing 100049, China;
4. Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences, Shenyang Liaoning 110016, China

Received:2020-01-09 Revised:2020-03-01 Online:2020-07-10 Published:2020-03-29
Supported by:
This work is partially supported by the National Natural Science Foundation of China (U1713216).

摘要/Abstract

摘要： 深度学习在视觉任务中的良好表现很大程度上依赖于海量的数据和计算力的提升，但是在很多实际项目中通常难以提供足够的数据来完成任务。针对某些情况下红外图像少且难以获得的问题，提出一种基于彩色图像生成红外图像的方法来获取更多的红外图像数据。首先，用现有的彩色图像和红外图像数据构建成对的数据集；然后，基于卷积神经网络、转置卷积神经网络构建生成对抗网络（GAN）模型的生成器和鉴别器；接着，基于成对的数据集来训练GAN模型，直到生成器和鉴别器之间达到纳什平衡状态；最后，用训练好的生成器将彩色图像从彩色域变换到红外域。基于定量评估标准对实验结果进行了评估，结果表明，所提方法可以生成高质量的红外图像，并且相较于在损失函数中不加正则化项，在损失函数中加入L1和L2正则化约束后，该方法的FID分数值平均分别降低了23.95和20.89。作为一种无监督的数据增强方法，该方法也可以被应用于其他缺少数据的目标识别、目标检测、数据不平衡等视觉任务中。

关键词: 红外图像生成, 生成对抗网络, 图像转换, 数据增强, 生成图像质量评估

Abstract: The great performance of deep learning in many visual tasks largely depends on the big data volume and the improvement of computing power. But in many practical projects, it is usually difficult to provide enough data to complete the task. Concerning the problem that the number of infrared images is small and the infrared images are hard to collect, a method to generate infrared images based on color images was proposed to obtain more infrared image data. Firstly, the existing color image and infrared image data were employed to construct the paired datasets. Secondly, the generator and the discriminator of Generative Adversarial Network (GAN) model were formed based on the convolutional neural network and the transposed convolutional neural network. Thirdly, the GAN model was trained based on the paired datasets until the Nash equilibrium between the generator and the discriminator was reached. Finally, the trained generator was used to transform the color image from the color field to the infrared field. The experimental results were evaluated based on quantitative evaluation metrics. The evaluation results show that the proposed method can generate high-quality infrared images. In addition, after the L1 or L2 regularization constraint was added to the loss function, the FID (Fréchet Inception Distance) score was respectively reduced by 23.95, 20.89 on average compared to the FID score of loss function not adding the constraint. As an unsupervised data augmentation method, the method can also be applied to many other visual tasks that lack train data, such as target recognition, target detection and data imbalance.

Key words: infrared image generation, Generative Adversarial Network (GAN), image transformation, data augmentation, quality evaluation of generative image

中图分类号:

TP391.41

陈佛计, 朱枫, 吴清潇, 郝颖明, 王恩德. 基于生成对抗网络的红外图像数据增强[J]. 计算机应用, 2020, 40(7): 2084-2088.

CHEN Foji, ZHU Feng, WU Qingxiao, HAO Yingming, WANG Ende. Infrared image data augmentation based on generative adversarial network[J]. Journal of Computer Applications, 2020, 40(7): 2084-2088.

参考文献

[1] DAYAN P. Helmholtz machines and wake-sleep learning[M]//ARBIB M A. Handbook of Brain Theory and Neural Network. Cambridge:MIT Press,2000:522-525.
[2] HINTON G E. Deep belief networks[J]. Scholarpedia,2009,4(5):No. 5947.
[3] KINGMA D P,WELLING M. Auto-encoding variational Bayes[EB/OL].[2019-09-22]. https://arxiv.org/pdf/1312.6114.pdf.
[4] SALAKHUTDINOV R, MNIH A, HINTON G. Restricted Boltzmann machines for collaborative filtering[C]//Proceedings of the 24th International Conference on Machine Learning. New York:ACM,2007:791-798.
[5] SALAKHUTDINOV R,HINTON G. Deep Boltzmann machines[C]//Proceedings of the 2009 Artificial Intelligence and Statistics. Cambridge:JMLR.org,2009:448-455.
[6] VAN DEN OORD A,KALCHBRENNER N,KAVUKCUOGLU K. Pixel recurrent neural networks[EB/OL].[2019-09-22]. https://arxiv.org/pdf/1601.06759.pdf.
[7] GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2014:2672-2680.
[8] 林懿伦, 戴星原, 李力, 等. 人工智能研究的新前线:生成式对抗网络[J]. 自动化学报,2018,44(5):775-792.(LIN Y L,DAI X Y,LI L,et al. The new frontier of AI research:generative adversarial networks[J]. Acta Automatica Sinica,2018,44(5):775-792.)
[9] 曹仰杰, 贾丽丽, 陈永霞, 等. 生成式对抗网络及其计算机视觉应用研究综述[J]. 中国图象图形学报,2018,23(10):1433-1449.(CAO Y J,JIA L L,CHEN Y X,et al. Review of computer vision based on generative adversarial networks[J]. Journal of Image and Graphics,2018,23(10):1433-1449.)
[10] 陈文兵, 管正雄, 陈允杰. 基于条件生成式对抗网络的数据增强方法[J]. 计算机应用,2018,38(11):3305-3311.(CHEN W B,GUAN Z X,CHEN Y J. Data augmentation method based on conditional generative adversarial net model[J]. Journal of Computer Applications,2018,38(11):3305-3311.)
[11] CHEN X,DUAN Y,HOUTHOOFT R,et al. InfoGAN:interpretable representation learning by information maximizing generative adversarial nets[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. Red Hook,NY:Curran Associates Inc.,2016:2172-2180.
[12] ZHANG H,GOODFELLOW I,METAXAS D,et al. Self-attention generative adversarial networks[EB/OL].[2019-09-22]. https://arxiv.org/pdf/1805.08318.pdf.
[13] ISOLA P,ZHU J Y,ZHOU T,et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:5967-5976.
[14] ZHU J Y,PARK T,ISOLA P,et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:2242-2251.
[15] ODENA A,OLAH C,SHLENS J. Conditional image synthesis with auxiliary classifier GANs[C]//Proceedings of the 34th International Conference on Machine Learning. New York:JMLR.org, 2017:2642-2651.
[16] CHOI Y,CHOI M,KIM M,et al. StarGAN:unified generative adversarial networks for multi-domain image-to-image translation[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:8789-8797.
[17] 许洪, 王向军, 刘峰, 等. 基于可见光光谱图像的红外多光谱图像仿真生成[J]. 红外与激光工程,2009,38(2):200-204.(XU H,WANG X J,LIU F,et al. Infrared multispectral image simulation based on spectral images in visible bands[J]. Infrared and Laser Engineering,2009,38(2):200-204.)
[18] 陈珊, 孙继银. 基于可见光图像的红外场景仿真[J]. 红外与激光工程,2009,38(1):23-26,30.(CHEN S,SUN J Y. IR scene simulation based on visual image[J]. Infrared and Laser Engineering,2009,38(1):23-26,30.)
[19] ARJOVSKY M,CHINTALA S,BOTTOU L. Wasserstein GAN[EB/OL].[2019-09-22]. https://arxiv.org/pdf/1701.07875.pdf.
[20] BROCK A,DONAHUE J,SIMONYAN K. Large scale GAN training for high fidelity natural image synthesis[EB/OL].[2019-09-22]. https://arxiv.org/pdf/1809.11096.pdf.
[21] WANG T C,LIU M Y,ZHU J Y,et al. High-resolution image synthesis and semantic manipulation with conditional GANs[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:8798-8807.
[22] NG A. Sparse autoencoder[EB/OL].[2019-10-20]. https://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf.
[23] CHOE G,KIM S H,IM S,et al. RANUS:RGB and NIR urban scene dataset for deep scene parsing[J]. IEEE Robotics and Automation Letters,2018,3(3):1808-1815.
[24] SAKLA W,KONJEVOD G,MUNDHENK T N. Deep multi-modal vehicle detection in aerial ISR imagery[C]//Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision. Piscataway:IEEE,2017:916-923.
[25] HEUSEL M,RAMSAUER H,UNTERTHINER T,et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook,NY:Curran Associates Inc.,2017:6626-6637.
[26] SZEGEDY C,IOFFE S,VANHOUCKE V,et al. Inception-v4:inception-ResNet and the impact of residual connections on learning[C]//Proceedings of the 31st AAAI conference on Artificial Intelligence. Palo Alto,CA:AAAI Press,2017:4278-4284.

基于生成对抗网络的红外图像数据增强

Infrared image data augmentation based on generative adversarial network

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	管其杰, 张挺, 李德亚, 周绍景, 杜奕. 基于多分辨率生成对抗网络的空间数据不确定性重建方法[J]. 计算机应用, 2021, 41(8): 2306-2311.
[2]	孙潇, 徐金东. 基于级联生成对抗网络的遥感图像去雾方法[J]. 计算机应用, 2021, 41(8): 2440-2444.
[3]	汤桂花, 孙磊, 毛秀青, 戴乐育, 胡永进. 基于深度对齐网络的生成对抗网络伪造人脸检测[J]. 计算机应用, 2021, 41(7): 1922-1927.
[4]	王先武, 张挺, 吉欣, 杜奕. 基于带梯度惩罚深度卷积生成对抗网络的页岩三维数字岩心重构方法[J]. 计算机应用, 2021, 41(6): 1805-1811.
[5]	李衍志, 范勇, 高琳. 基于形态流的石油钻井水流异常检测[J]. 计算机应用, 2021, 41(6): 1842-1848.
[6]	井贝贝, 郭嘉, 王丽清, 陈静, 丁洪伟. 结合降噪卷积神经网络和条件生成对抗网络的图像双重盲降噪算法[J]. 计算机应用, 2021, 41(6): 1767-1774.
[7]	贾承勋, 赖华, 余正涛, 文永华, 于志强. 融合单语语言模型的汉越伪平行语料生成[J]. 计算机应用, 2021, 41(6): 1652-1658.
[8]	郭茂祖, 杨倩楠, 赵玲玲. 基于条件Wassertein生成对抗网络的图像生成[J]. 计算机应用, 2021, 41(5): 1432-1437.
[9]	孙鹤立, 孙玉柱, 张晓云. 基于生成对抗网络的事件描述生成[J]. 计算机应用, 2021, 41(5): 1256-1261.
[10]	甘岚, 沈鸿飞, 王瑶, 张跃进. 基于改进DCGAN的数据增强方法[J]. 计算机应用, 2021, 41(5): 1305-1313.
[11]	陆鑫伟, 余鹏飞, 李海燕, 李红松, 丁文谦. 基于注意力自身线性融合的弱监督细粒度图像分类算法[J]. 计算机应用, 2021, 41(5): 1319-1325.
[12]	段友祥, 张含笑, 孙歧峰, 孙友凯. 基于拉普拉斯金字塔生成对抗网络的图像超分辨率重建算法[J]. 计算机应用, 2021, 41(4): 1020-1026.
[13]	霍首君, 郝琰, 石慧宇, 董艳清, 曹锐. 基于深度卷积网络的运动想象脑电信号模式识别[J]. 计算机应用, 2021, 41(4): 1042-1048.
[14]	欧莉莉, 邵峰晶, 孙仁诚, 隋毅. 基于半监督方法的脑梗死图像识别[J]. 计算机应用, 2021, 41(4): 1221-1226.
[15]	李虹霞, 秦品乐, 闫寒梅, 曾建潮, 鲍骞月, 柴锐. 基于面部特征图对称的人脸正面化生成对抗网络算法[J]. 计算机应用, 2021, 41(3): 714-720.