Abstract: To handle the issues of lacking paired training samples and inconsistency between frames in translation from virtual to real driving scenes, a video translation model based on Generative Adversarial Networks was proposed in this paper. In order to solve the problem of lacking data samples, the model adopted a “dual networks” architecture, where the semantic segmentation scene was used as an intermediate transition to build front-part and back-part networks, respectively. In the front-part network, a convolution network and a deconvolution network were adopted, and the optical flow network was also used to extract the dynamic information between frames to implement continuous video translation from virtual to semantic segmentation scenes. In the back-part network, a conditional generative adversarial network was used in which a generator, an image discriminator and a video discriminator were designed and combined with the optical flow network to implement continuous video translation from semantic segmentation to real scenes. Data collected from an autonomous driving simulator and a public data set were used for training and testing. Virtual to real scene translation can be achieved in a variety of driving scenarios, and the translation effect is significantly better than the comparative algorithms. Experimental results show that the proposed model can handle the problems of the discontinuity between frames and the ambiguity for moving obstacles to obtain more continuous videos when applying in various driving scenarios.

Key words: virtual to real, video translation, Generative Adversarial Networks (GAN), optical flow network, driving scene

摘要： 针对虚拟到真实驾驶场景翻译中成对的数据样本缺乏以及前后帧不一致等问题，提出一种基于生成对抗网络的视频翻译模型。为解决数据样本缺乏问题，模型采取“双网络”架构，将语义分割场景作为中间过渡分别构建前、后端网络。在前端网络中，采用卷积和反卷积框架，并利用光流网络提取前后帧的动态信息，实现从虚拟场景到语义分割场景的连续的视频翻译；在后端网络中，采用条件生成对抗网络框架，设计生成器、图像判别器和视频判别器，并结合光流网络，实现从语义分割场景到真实场景的连续的视频翻译。实验利用从自动驾驶模拟器采集的数据与公开数据集进行训练和测试，在多种驾驶场景中能够实现虚拟到真实场景的翻译，翻译效果明显好于对比算法。结果表明，所提模型能够有效解决前后帧不连续和动态目标模糊的问题，使翻译的视频更为流畅，并且能适应多种复杂的驾驶场景。

关键词: 虚拟到真实, 视频翻译, 生成对抗网络, 光流网络, 驾驶场景