Fast video transcoding method based on Spark Streaming

doi:10.11772/j.issn.1001-9081.2018040942

Journal of Computer Applications ›› 2018, Vol. 38 ›› Issue (12): 3500-3508.DOI: 10.11772/j.issn.1001-9081.2018040942

Previous Articles Next Articles

Fast video transcoding method based on Spark Streaming

FU Mou¹, YANG Hekun¹, WU Tangmei¹, HE Run¹, FENG Chaosheng^1,2, KANG Sheng³

1. School of Computer Science, Sichuan Normal University, Chengdu Sichuan 610101, China;
2. Visual Computing & Virtual Reality Key Laboratory of Sichuan Province(Sichuan Normal University), Chengdu Sichuan 610101, China;
3. Sichuan Normal University Technology Park Development Company Limited, Chengdu Sichuan 610066, China

Received:2018-05-07 Revised:2018-07-04 Online:2018-12-10 Published:2018-12-15
Contact: 付眸
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61373163), the National Key Technology Support Program of China (2014BAH11F02, 2014BAH11F01), and the Key Project of Science & Technology Department of Sichuan Province (2015GZ079).

基于Spark Streaming的快速视频转码方法

付眸¹, 杨贺昆¹, 吴唐美¹, 何润¹, 冯朝胜^1,2, 康胜³

1. 四川师范大学计算机科学学院, 成都 610101;
2. 可视化计算与虚拟现实四川省重点实验室(四川师范大学), 成都 610101;
3. 四川师大科技园发展有限公司, 成都 610066

通讯作者: 付眸
作者简介:付眸(1991-),男,四川绵阳人,硕士研究生,主要研究方向:云计算、大数据分析、分布式流处理;杨贺昆(1993-),男,河南驻马店人,硕士研究生,主要研究方向:云计算、信息安全;吴唐美(1997-),女,四川成都人,主要研究方向:云计算、大数据分析;何润(1998-),男,四川宜宾人,主要研究方向:云计算、大数据分析;冯朝胜(1971-),男,四川广元人,教授,博士,主要研究方向:云计算、网络与数据安全;康胜(1973-),男,四川绵阳人,副教授,硕士,主要研究方向:信息管理。
基金资助:
国家自然科学基金资助项目（61373163）；国家科技支撑计划项目（2014BAH11F02，2014BAH11F01）；四川省科技支撑计划项目（2015GZ079）。

Abstract

Abstract: Aiming at the problems of slow transcoding speed of single-machine video transcoding method and limited efficiency improvement of parallel transcoding method for batch processing, a fast video transcoding method for stream processing based on Spark Streaming distributed stream processing framework was proposed. Firstly, an automated video slicing model was built by using the open source multimedia processing tool of FFmpeg and a programming algorithm was proposed. Then, in view of the characteristics of parallel video transcoding, the stream processing model of video transcoding was constructed by studying Resilient Distributed Datasets (RDD). Finally, the video merging scheme was designed to store the combined video files effectively. Based on the proposed fast video transcoding method, a fast video transcoding system based on Spark Streaming was designed and implemented. The experimental results show that, compared with the Hadoop video transcoding method for batch processing, the proposed method has improved the transcoding efficiency by 26.7%, and compared with the video parallel transcoding based on Hadoop platform, the proposed method has improved the transcoding efficiency by 20.1%.

Key words: video transcoding, Spark Streaming, distributed stream processing, FFmpeg, Resilient Distributed Datasets (RDD)

摘要： 针对单机视频转码方法转码速度较慢和面向批处理的并行转码方法效率提升有限的问题，基于Spark Streaming分布式流处理框架，提出了一种面向流处理的快速视频转码方法。首先，使用开源多媒体处理工具FFmpeg，构建了自动化的视频切片模型，提出编程算法；然后，针对并行视频转码的特点，对弹性分布式数据集（RDD）进行研究，构建了视频转码的流处理模型；最后，设计视频合并方案，将合并后的视频文件进行有效储存。根据所提出的快速视频转码方法设计与实现了基于Spark Streaming的快速视频转码系统。实验结果表明，与面向批处理Hadoop视频转码方法相比，所提方法转码效率提升了26.7%；与基于Hadoop平台的视频并行转码方法相比，该方法转码效率提升了20.1%。

关键词: 视频转码, Spark Streaming, 分布式流处理, FFmpeg, 弹性分布式数据集

CLC Number:

TP301.6

FU Mou, YANG Hekun, WU Tangmei, HE Run, FENG Chaosheng, KANG Sheng. Fast video transcoding method based on Spark Streaming[J]. Journal of Computer Applications, 2018, 38(12): 3500-3508.

付眸, 杨贺昆, 吴唐美, 何润, 冯朝胜, 康胜. 基于Spark Streaming的快速视频转码方法[J]. 计算机应用, 2018, 38(12): 3500-3508.

References

[1] 中国互联网络信息中心.第41次中国互联网络发展状况统计报告[R].北京:中国互联网络信息中心,2018.(China Internet Network Information Center. The 41st China Statistical Report on Internet Development[R]. Beijing:CNNIC, 2018.)
[2] 冯朝胜,秦志光,袁丁.云数据安全存储技术[J].计算机学报,2015,38(1):150-163.(FENG C S, QIN Z G, YUAN D. Techniques of secure storage for cloud data[J]. Chinese Journal of Computers, 2015, 38(1):150-163.)
[3] 冯朝胜,秦志光,袁丁,等.云计算环境下访问控制关键技术[J].电子学报,2015,43(2):312-319.(FENG C S, QIN Z G, YUAN D, et al. Key techniques of access control for cloud computing[J]. Acta Electronica Sinica, 2015, 43(2):312-319.)
[4] SULLIVAN G J, OHM J-R, HAN W-J, et al. Overview of the High Efficiency Video Coding (HEVC) standard[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012,22(12):1649-1668.
[6] O'DRISCOLL A, DAUGELAITE J, SLEATOR R D. ‘Big data’, Hadoop and cloud computing in genomics[J]. Journal of Biomedical Informatics, 2013, 46(5):774-781.
[7] 于金良,朱志祥,李聪颖.Hadoop MapReduce新旧架构的对比研究综述[J].计算机与数字工程,2017,45(1):83-87.(YU J L, ZHU Z X, LI C Y. Comparative study of old and new architecture of Hadoop MapReduce[J]. Computer & Digital Engineering, 2017, 45(1):83-87.)
[8] HASHEM I A T, ANUAR N B, GANI A, et al. MapReduce:review and open challenges[J]. Scientometrics, 2016, 109(1):389-422.
[9] CHANG Z H, JONG B F, WEI J W, et al. Distributed video transcoding on a heterogeneous computing platform[C]//Proceedings of the 2016 IEEE Asia Pacific Conference on Circuits and Systems. Piscataway, NJ:IEEE, 2016:444-447.
[10] ZENG H, ZHANG Z Y, SHI L L. Research and implementation of video codec based on FFmpeg[C]//Proceedings of the 2016 International Conference on Network and Information Systems for Computers. Piscataway, NJ:IEEE, 2016:184-188.
[11] 嵇志华,张瑶,孙庆丰.基于FFMPEG和MENCODER的视频转换PHP网页[J].林业勘查设计,2014(2):95-97.(JI Z H, ZHANG Y, SUN Q F. Based on FFMPEG and MENCODER video conversion Web page of PHP[J]. Forest Investigation Design, 2014(2):95-97.)
[12] KIM M, HAN S, CUI Y, et al. A Hadoop-based multimedia transcoding system for processing social media in the PaaS platform of SMCCSE[J]. KSⅡ Transactions on Internet & Information Systems, 2012, 6(11):2827-2848.
[13] SINGH A, SHARMA S, KUMAR S R, et al. Overview of PaaS and SaaS and its application in cloud computing[C]//Proceedings of the 2016 International Conference on Innovation and Challenges in Cyber Security. Piscataway, NJ:IEEE, 2016:172-176.
[14] KIM M, CUI Y, HAN S, et al. Performance optimization of a distributed transcoding system based on Hadoop for multimedia streaming services[J]. Advanced Science Letters, 2014, 20(10/11/12):1890-1894.
[15] SON S, KIM M. HVTS:Hadoop-based video transcoding system for media services[J]. IEICE Transactions on Fundamentals of Electronics Communications and Computer Sciences, 2017, E100-A(5):1248-1253.
[16] 庞一,张凤妍,孙立峰,等.面向多核处理器的视频编码并行加速算法综述[J].计算机科学与探索,2009,3(4):337-346.(PANG Y, ZHANG F Y, SUN L F, et al. Survey of parallel acceleration algorithms of video coding on multi-core processor[J]. Journal of Frontiers of Computer Science and Technology, 2009,3(4):337-346.)
[17] 巩子杰.分布式计算平台中任务调度算法的设计[D].天津:天津大学,2014:7-9.(GONG Z J. Design of task scheduling algorithm in the distributed computing platform[D]. Tianjin:Tianjin University, 2014:7-9.)
[18] SONG C W, SHEN W F, SUN L Q, et al. Distributed video transcoding based on MapReduce[C]//Proceedings of the 2014 IEEE/ACIS 13th International Conference on Computer and Information Science. Piscataway, NJ:IEEE, 2014:309-314.
[19] CHEN M G, CHEN W J, LIU Z Y, et al. Parallel video transcoding using Hadoop MapReduce[J]. Journal of Network Computing and Applications, 2016, 1(1):7-11.
[20] 谢山山.基于分布式的大规模视频转码系统的设计与实现[D].成都:电子科技大学,2015:23-48.(XIE S S. The design and implementation of large-scale video transcoding system based on distributed computing[D]. Chengdu:University of Electronic Science and Technology of China, 2015:23-48.)

[1]	ZHAO Wenfang, LIU Xulin. Real-time processing system for automatic weather station data on Spark Streaming architecture [J]. Journal of Computer Applications, 2018, 38(1): 38-43.
[2]	HAN Dezhi, CHEN Xuguang, LEI Yuxin, DAI Yongtao, ZHANG Xiao. Real-time data analysis system based on Spark Streaming and its application [J]. Journal of Computer Applications, 2017, 37(5): 1263-1269.
[3]	WU Zhixue. Real-time detailed classification energy consumption measurement system based on Spark Streaming [J]. Journal of Computer Applications, 2017, 37(4): 928-935.
[4]	SU Jianjun, MU Shiyou, YANG bo, SUN Xiaobin, ZHAO Haiwu, GU Xiao. Distortion analysis of digital video transcoding [J]. Journal of Computer Applications, 2017, 37(10): 2899-2902.
[5]	LU Zhuo-yi JIA Ke-bin XIAO Yun-zhi. Fast video transcoding algorithm based on hybrid characteristic of multi-scale videos [J]. Journal of Computer Applications, 2011, 31(11): 2997-3000.
[6]	Yong-sheng XIANG Ming FU. Fast mode decision of downscaling video transcoding for H.264 streams [J]. Journal of Computer Applications, 2008, 28(11): 2890-2892.

Fast video transcoding method based on Spark Streaming

基于Spark Streaming的快速视频转码方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 6

Recommended Articles

Metrics