Survey on tile-based viewport adaptive streaming scheme of panoramic video

doi:10.11772/j.issn.1001-9081.2023020209

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (2): 536-547.DOI: 10.11772/j.issn.1001-9081.2023020209

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Survey on tile-based viewport adaptive streaming scheme of panoramic video

Junjie LI, Yumei WANG(), Zhijun LI, Yu LIU

College of Artificial Intelligence，Beijing University of Posts and Telecommunications，Beijing 100876，China

Received:2023-03-02 Revised:2023-03-31 Accepted:2023-04-03 Online:2023-08-14 Published:2024-02-10
Contact: Yumei WANG
About author:LI Junjie， born in 2001， M. S. candidate. His research interests include panoramic video transmission.
LI Zhijun， born in 1997， M. S. candidate. His research interests include panoramic video transmission.
LIU Yu， born in 1978， Ph. D.， associate professor. Her research interests include network information theory， communication network theory， distributed information processing in wireless network.
Supported by:
National Key Research and Development Program of China(2022YFB2902705)

全景视频基于块的视口自适应传输方案综述

李俊杰, 望育梅(), 李志军, 刘雨

北京邮电大学人工智能学院，北京 100876

通讯作者: 望育梅
作者简介:李俊杰（2001—），男，江西上饶人，硕士研究生，主要研究方向：全景视频传输
李志军（1997—），男，广东佛山人，硕士研究生，主要研究方向：全景视频传输
刘雨（1978—），女，山东烟台人，副教授，博士生导师，博士，主要研究方向：网络信息论、通信网理论、无线网络中的分布式信息处理。
基金资助:
国家重点研发计划项目(2022YFB2902705)

Abstract

Abstract:

Panoramic videos have attracted wide attention due to their unique immersive and interactive experience. The high bandwidth and low delay required for wireless streaming of panoramic videos have brought challenges to existing network streaming systems. Tile-based viewport adaptive streaming can effectively alleviate the streaming pressure brought by panoramic video， and has become the current mainstream scheme and hot research topic. By analyzing the research status and development trend of tile-based viewport adaptive streaming， the two important modules of this streaming scheme， namely viewport prediction and bit rate allocation， were discussed， and the methods in relevant fields were summarized from different perspectives. Firstly， based on the panoramic video streaming framework， the relevant technologies were clarified. Secondly， the user experience quality indicators to evaluate the performance of the streaming system were introduced from the subjective and objective dimensions. Then， the classic research methods were summarized from the aspects of viewport prediction and bit rate allocation. Finally， the future development trend of panoramic video streaming was discussed based on the current research status.

Key words: panoramic video, viewport adaptive streaming, viewport prediction, bit rate allocation, Quality of Experience (QoE)

摘要：

全景视频由于独特的沉浸式、交互式体验受到广泛关注。全景视频传输所需的高带宽、低时延给现有网络传输系统带来了挑战。基于tile（块）的视口自适应传输可以有效缓解全景视频所带来的传输压力，成为当前的主流方案和研究热点。通过分析基于tile的视口自适应传输方案的研究现状和发展趋势，对该传输方案的两个重要模块，即视口预测与码率分配进行论述，从不同视角归纳总结相关领域的方法。首先，基于全景视频传输框架对相关技术进行阐明；其次，从主、客观两个维度分别介绍评估传输系统性能的用户体验质量的指标；再后，分别从视口预测、码率分配两方面进行归纳，系统梳理经典的研究方法；最后，基于当前研究现状讨论全景视频传输的未来发展趋势。

关键词: 全景视频, 视口自适应传输, 视口预测, 码率分配, 用户体验质量

CLC Number:

TP391

Junjie LI, Yumei WANG, Zhijun LI, Yu LIU. Survey on tile-based viewport adaptive streaming scheme of panoramic video[J]. Journal of Computer Applications, 2024, 44(2): 536-547.

李俊杰, 望育梅, 李志军, 刘雨. 全景视频基于块的视口自适应传输方案综述[J]. 《计算机应用》唯一官方网站, 2024, 44(2): 536-547.

Figures/Tables 7

References 113

1	金学骥.基于OMAF的全景视频传输机制分析［J］.电子技术与软件工程，2022（8）：184-188. 10.3724/sp.j.1089.2022.18895
	JIN X J. Analysis of panoramic video transmission mechanism based on OMAF［J］. Electronic Technology & Software Engineering， 2022（8）：184-188. 10.3724/sp.j.1089.2022.18895
2	Grand View Research. Virtual Reality （VR） market size， share & trends analysis report， 2022—2030［R/OL］.［2023-02-28］. .
3	Fortune Business Insights. Virtual reality global market analysis， insights and forecast， 2022—2029［R/OL］.［2023-02-28］..
4	缪辰启，罗铖.全景视频视口预测方法综述［J］.电视技术，2022， 46（2）： 10-13.
	MIAO C Q， LUO Y. A overview of panoramic video viewport prediction methods［J］. Video Engineering， 2022， 46（2）： 10-13.
5	OZCINAR C， CABRERA J， SMOLIC A. Visual attention-aware omnidirectional video streaming using optimal tiles for virtual reality［J］. IEEE Journal on Emerging and Selected Topics in Circuits and Systems， 2019， 9（1）： 217-230. 10.1109/jetcas.2019.2895096
6	WONG E S， WAHAB N H A， SAEED F， et al. 360-Degree video bandwidth reduction： technique and approaches comprehensive review［J］. Applied Sciences， 2022， 12（15）： 7581. 10.3390/app12157581
7	XIAO M， WANG S， ZHOU C， et al. MiniView layout for bandwidth-efficient 360-degree video［C］// Proceedings of the 26th ACM International Conference on Multimedia. New York： ACM， 2018： 914-922. 10.1145/3240508.3240705
8	CHIARIOTTI F. A survey on 360-degree video： coding， quality of experience and streaming［J］. Computer Communications， 2021， 177： 133-155. 10.1016/j.comcom.2021.06.029
9	BIATEK T， HAMIDOUCHE W， P-L CABARAT， et al. Scalable video coding for backward-compatible 360° video delivery over broadcast networks［J］. IEEE Transactions on Broadcasting， 2020， 66（2）： 322-332. 10.1109/tbc.2019.2941073
10	YAO R， HEATH T， DAVIES A， et al. Oculus VR best practices guide［J］. Oculus VR， 2014， 4： 27-35.
11	BAO Y， WU H， ZHANG T， et al. Shooting a moving target： motion-prediction-based transmission for 360-degree videos［C］// Proceedings of the 2016 IEEE International Conference on Big Data. Piscataway： IEEE， 2016： 1161-1170. 10.1109/bigdata.2016.7840720
12	PARK S， BHATTACHARYA A， YANG Z， et al. Mosaic： advancing user quality of experience in 360-degree video streaming with machine learning［J］. IEEE Transactions on Network and Service Management， 2021， 18（1）： 1000-1015. 10.1109/tnsm.2021.3053183
13	NGUYEN T C， YUN J-H. Predictive tile selection for 360-degree VR video streaming in bandwidth-limited networks［J］. IEEE Communications Letters， 2018， 22（9）： 1858-1861. 10.1109/lcomm.2018.2848915
14	HOSSEINI M， SWAMINATHAN V. Adaptive 360 VR video streaming： divide and conquer［C］// Proceedings of the 2016 IEEE International Symposium on Multimedia. Piscataway： IEEE， 2016： 107-110. 10.1109/ism.2016.0028
15	MANGIANTE S， KLAS G， NAVON A， et al. VR is on the edge： how to deliver 360 videos in mobile networks［C］// Proceedings of the 2017 Workshop on Virtual Reality and Augmented Reality Network. New York： ACM， 2017： 30-35. 10.1145/3097895.3097901
16	YADAV P K， OOI W T. Tile rate allocation for 360-degree tiled adaptive video streaming［C］// Proceedings of the 28th ACM International Conference on Multimedia. New York： ACM， 2020： 3724-3733. 10.1145/3394171.3413550
17	HANNUKSELA M M， WANG Y-K， HOURUNRANTA A. An overview of the OMAF standard for 360° video［C］// Proceedings of the 2019 Data Compression Conference. Piscataway： IEEE， 2019： 418-427. 10.1109/dcc.2019.00050
18	SALOMON D. Transformations and Projections in Computer Graphics［M］. London： Springer， 2006. 10.1007/978-1-84628-620-9
19	LI L， LI Z， BUDAGAVI M， et al. Projection based advanced motion model for cubic mapping for 360-degree video［C］// Proceedings of the 2017 IEEE International Conference on Image Processing. Piscataway： IEEE， 2017： 1427-1431. 10.1109/icip.2017.8296517
20	COBAN M， AUWERA G V， KARCZEWICZ M. VR/360 video truncated square pyramid geometry for OMAF： ISO/IEC ［S］. Geneva： ISO， 2016.
21	郭宗明，班怡璇，谢澜. 虚拟现实视频传输架构和关键技术［J］. 中兴通讯技术， 2017， 23（6）： 19-23. 10.3969/j.issn.1009-6868.2017.06.005
	GUO Z M， BAN Y X， XIE L.Architecture and key technologies of virtual reality video transmission and streaming［J］. ZTE Technology Journal， 2017， 23（6）： 19-23. 10.3969/j.issn.1009-6868.2017.06.005
22	GÓMEZ D， NÚÑEZ J A， FRAILE I， et al. TiCMP： a lightweight and efficient tiled cubemap projection strategy for immersive videos in Web-based players［C］// Proceedings of the 28th ACM SIGMM Workshop on Network and Operating Systems Support for Digital Audio and Video. New York： ACM， 2018： 1-6. 10.1145/3210445.3210456
23	POURAZAD M T， DOUTRE C， AZIMI M， et al. HEVC： the new gold standard for video compression： how does HEVC compare with H. 264/AVC？［J］. IEEE Consumer Electronics Magazine， 2012， 1（3）： 36-46. 10.1109/mce.2012.2192754
24	BAUERMANN I， MIELKE M， STEINBACH E. H.264 based coding of omnidirectional video［M］// Computer Vision and Graphics. Dordrecht： Springer， 2006： 209-215.
25	YE Y， BOYCE J M， HANHART P. Omnidirectional 360° video coding technology in responses to the joint call for proposals on video compression with capability beyond HEVC［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2020， 30（5）： 1241-1252. 10.1109/tcsvt.2019.2953827
26	SODAGAR I. The MPEG-DASH standard for multimedia streaming over the Internet［J］. IEEE Multimedia， 2011， 18（4）： 62-67. 10.1109/mmul.2011.71
27	叶成英，李建微，陈思喜. VR 全景视频传输研究进展［J］. 计算机应用研究， 2022， 39（6）： 1601-1607， 1621.
	YE C Y， LI J W， CHEN S X. Research progress of VR panoramic video transmission［J］. Application Research of Computers， 2022， 39（6）： 1601-1607， 1621.
28	ALMQUIST M， ALMQUIST V， KRISHNAMOORTHI V， et al. The prefetch aggressiveness tradeoff in 360 video streaming［C］// Proceedings of the 9th ACM Multimedia Systems Conference. New York： ACM， 2018： 258-269. 10.1145/3204949.3204970
29	MAVLANKAR A， GIROD B. Pre-fetching based on video analysis for interactive region-of-interest streaming of soccer sequences［C］// Proceedings of the 2009 16th IEEE International Conference on Image Processing. Piscataway： IEEE， 2009： 3061-3064. 10.1109/icip.2009.5414201
30	WANG Z， BOVIK A C， SHEIKH H R， et al. Image quality assessment： from error visibility to structural similarity［J］. IEEE Transactions on Image Processing， 2004， 13（4）： 600-612. 10.1109/tip.2003.819861
31	SKUPIN R， SANCHEZ Y， WANG Y-K， et al. Standardization status of 360 degree video coding and delivery［C］// Proceedings of the 2017 IEEE Visual Communications and Image Processing. Piscataway： IEEE， 2017： 1-4. 10.1109/vcip.2017.8305083
32	SUN Y， LU A， YU L. Weighted-to-spherically-uniform quality evaluation for omnidirectional video［J］. IEEE Signal Processing Letters， 2017， 24（9）： 1408-1412.
33	ZAKHARCHENKO V， CHOI K P， PARK J H. Quality metric for spherical panoramic video［C］// Proceedings of the SPIE 9970， Optics and Photonics for Information Processing X. ［S.l.］： SPIE， 2016： 57-65. 10.1117/12.2235885
34	TRAN H T T， NGOC N P， BUI C M， et al. An evaluation of quality metrics for 360 videos［C］// Proceedings of the 2017 9th International Conference on Ubiquitous and Future Networks. Piscataway： IEEE， 2017： 7-11. 10.1109/icufn.2017.7993736
35	ZHOU Y， YU M， MA H， et al. Weighted-to-spherically-uniform SSIM objective quality evaluation for panoramic video［C］// Proceedings of the 2018 14th IEEE International Conference on Signal Processing. Piscataway： IEEE， 2018： 54-57. 10.1109/icsp.2018.8652269
36	CHEN S， ZHANG Y， LI Y， et al. Spherical structural similarity index for objective omnidirectional video quality assessment［C］// Proceedings of the 2018 IEEE International Conference on Multimedia and Expo. Piscataway： IEEE， 2018： 1-6. 10.1109/icme.2018.8486584
37	杜丽娜，卓力，李嘉锋. 全景视频QoE评价研究进展［J］. 信号处理， 2022， 38（9）： 1831-1842. 10.16798/j.issn.1003-0530.2022.09.006
	DU L N， ZHUO L， LI J F. Survey on QoE evaluation of panoramic video［J］. Journal of Signal Processing， 2022， 38（9）： 1831-1842. 10.16798/j.issn.1003-0530.2022.09.006
38	YAQOOB A， G-M MUNTEAN. A combined field-of-view prediction-assisted viewport adaptive delivery scheme for 360° videos［J］. IEEE Transactions on Broadcasting， 2021， 67（3）： 746-760. 10.1109/tbc.2021.3105022
39	RONDÓN M F R， SASSATELLI L， APARICIO-PARDO R， et al. TRACK： A new method from a re-examination of deep architectures for head motion prediction in 360-degree videos［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 44（9）： 5681-5699.
40	CORBILLON X， SIMON G， DEVLIC A， et al. Viewport-adaptive navigable 360-degree video delivery［C］// Proceedings of the 2017 IEEE International Conference on Communications. Piscataway： IEEE， 2017： 1-7. 10.1109/icc.2017.7996611
41	DUANMU F， KURDOGLU E， HOSSEINI S A， et al. Prioritized buffer control in two-tier 360 video streaming［C］// Proceedings of the 2017 Workshop on Virtual Reality and Augmented Reality Network. New York： ACM， 2017： 13-18. 10.1145/3097895.3097898
42	QIAN F， JI L， HAN B， et al. Optimizing 360 video delivery over cellular networks［C］// Proceedings of the 5th Workshop on All Things Cellular： Operations， Applications and Challenges. New York： ACM， 2016： 1-6. 10.1145/2980055.2980056
43	QIAN F， HAN B， XIAO Q， et al. Flare： practical viewport-adaptive 360-degree video streaming for mobile devices［C］// Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. New York： ACM， 2018： 99-114. 10.1145/3241539.3241565
44	AZUMA R， BISHOP G. A frequency-domain analysis of head-motion prediction［C］// Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques. New York： ACM， 1995： 401-408. 10.1145/218380.218496
45	MAVLANKAR A， GIROD B. Video streaming with interactive pan/tilt/zoom［C］// High-Quality Visual Experience. Signals and Communication Technology. Berlin： Springer， 2010： 431-455. 10.1007/978-3-642-12802-8_19
46	DE LA FUENTE Y S， BHULLAR G S， SKUPIN R， et al. Delay impact on MPEG OMAF’s tile-based viewport-dependent 360° video streaming［J］. IEEE Journal on Emerging and Selected Topics in Circuits and Systems， 2019， 9（1）： 18-28. 10.1109/jetcas.2019.2899516
47	ZOU J， LI C， LIU C， et al. Probabilistic tile visibility-based server-side rate adaptation for adaptive 360-degree video streaming［J］. IEEE Journal of Selected Topics in Signal Processing， 2019， 14（1）： 161-176. 10.1109/jstsp.2019.2956716
48	FAN C-L， YEN S-C， HUANG C-Y， et al. Optimizing fixation prediction using recurrent neural networks for 360° video streaming in head-mounted virtual reality［J］. IEEE Transactions on Multimedia， 2019， 22（3）： 744-759. 10.1109/tmm.2019.2931807
49	BAO Y， WU H， ZHANG T， et al. Shooting a moving target： motion-prediction-based transmission for 360-degree videos［C］// Proceedings of the 2016 IEEE International Conference on Big Data . Piscataway： IEEE， 2016： 1161-1170. 10.1109/bigdata.2016.7840720
50	XU M， SONG Y， WANG J， et al. Predicting head movement in panoramic video： a deep reinforcement learning approach［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2019， 41（11）： 2693-2708. 10.1109/tpami.2018.2858783
51	LEE D， CHOI M， LEE J. Prediction of head movement in 360-degree videos using attention model［J］. Sensors， 2021， 21（11）： 3678. 10.3390/s21113678
52	XIE L， XU Z， BAN Y， et al. 360ProbDASH： improving QoE of 360 video streaming using tile-based HTTP adaptive streaming［C］// Proceedings of the 25th ACM International Conference on Multimedia. New York： ACM， 2017： 315-323. 10.1145/3123266.3123291
53	ZOU J， LI C， LIU C， et al. Probabilistic tile visibility-based server-side rate adaptation for adaptive 360-degree video streaming［J］. IEEE Journal of Selected Topics in Signal Processing， 2019， 14（1）： 161-176. 10.1109/jstsp.2019.2956716
54	JIANG Z， ZHANG X， XU Y， et al. Reinforcement learning based rate adaptation for 360-degree video streaming［J］. IEEE Transactions on Broadcasting， 2020， 67（2）： 409-423. 10.1109/tbc.2020.3028286
55	KIRULUTA A， EIZENMAN M， PASUPATHY S. Predictive head movement tracking using a Kalman filter［J］. IEEE Transactions on Systems， Man， and Cybernetics， Part B （Cybernetics）， 1997， 27（2）： 326-331. 10.1109/3477.558841
56	GÜL S， BOSSE S， PODBORSKI D， et al. Kalman filter-based head motion prediction for cloud-based mixed reality［C］// Proceedings of the 28th ACM International Conference on Multimedia. New York： ACM， 2020： 3632-3641. 10.1145/3394171.3413699
57	BAN Y， XIE L， XU Z， et al. CUB360： exploiting cross-users behaviors for viewport prediction in 360 video adaptive streaming［C］// Proceedings of the 2018 IEEE International Conference on Multimedia and Expo. Piscataway： IEEE， 2018： 1-6. 10.1109/icme.2018.8486606
58	CHAKARESKI J， AKSU R， CORBILLON X， et al. Viewport-driven rate-distortion optimized 360° video streaming［C］// Proceedings of the 2018 IEEE International Conference on Communications. Piscataway： IEEE， 2018： 1-7. 10.1109/icc.2018.8422859
59	XU Z， BAN Y， ZHANG K， et al. Tile-based QoE-driven HTTP/2 streaming system for 360 video［C］// Proceedings of the 2018 IEEE International Conference on Multimedia & Expo Workshops. Piscataway： IEEE， 2018： 1-4. 10.1109/icmew.2018.8551493
60	PETRANGELI S， SIMON G， SWAMINATHAN V. Trajectory-based viewport prediction for 360-degree virtual reality videos［C］// Proceedings of the 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality. Piscataway： IEEE， 2018： 157-160. 10.1109/aivr.2018.00033
61	XIE L， ZHANG X， GUO Z. CLS： a cross-user learning based system for improving qoe in 360-degree video adaptive streaming［C］// Proceedings of the 26th ACM International Conference on Multimedia. New York： ACM， 2018： 564-572. 10.1145/3240508.3240556
62	CORNIA M， BARALDI L， SERRA G， et al. A deep multi-level network for saliency prediction［C］// Proceedings of the 2016 23rd International Conference on Pattern Recognition. Piscataway： IEEE， 2016： 3488-3493. 10.1109/icpr.2016.7900174
63	WANG W， SHEN J， GUO F， et al. Revisiting video saliency： a large-scale benchmark and a new model［C］// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 4894-4903. 10.1109/cvpr.2018.00514
64	JIANG L， XU M， LIU T， et al. DeepVS： a deep learning based video saliency prediction approach［C］// Proceedings of the 2018 European Conference on Computer Vision. Cham： Springer， 2018： 625-642. 10.1007/978-3-030-01264-9_37
65	CHE Z， BORJI A， ZHAI G， et al. How is gaze influenced by image transformations？ dataset and model［J］. IEEE Transactions on Image Processing， 2019， 29： 2287-2300. 10.1109/tip.2019.2945857
66	DE ABREU A， OZCINAR C， SMOLIC A. Look around you： saliency maps for omnidirectional images in VR applications［C］// Proceedings of the 2017 Ninth International Conference on Quality of Multimedia Experience. Piscataway： IEEE， 2017： 1-6. 10.1109/qomex.2017.7965634
67	MONROY R， LUTZ S， CHALASANI T， et al. SalNet360： saliency maps for omni-directional images with CNN［J］. Signal Processing： Image Communication， 2018， 69： 26-34. 10.1016/j.image.2018.05.005
68	PAN J， SAYROL E， GIRO-I-NIETO X， et al. Shallow and deep convolutional networks for saliency prediction［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 598-606. 10.1109/cvpr.2016.71
69	PARK S， BHATTACHARYA A， YANG Z， et al. Advancing user quality of experience in 360-degree video streaming［C］// Proceedings of the 2019 IFIP Networking Conference. Piscataway： IEEE， 2019： 1-9. 10.23919/ifipnetworking46909.2019.8999460
70	CHAO F-Y， ZHANG L， HAMIDOUCHE W， et al. Salgan360： visual saliency prediction on 360 degree images with generative adversarial networks［C］// Proceedings of the 2018 IEEE International Conference on Multimedia & Expo Workshops. Piscataway： IEEE， 2018： 1-4. 10.1109/icmew.2018.8551543
71	EDER M， J-M FRAHM. Convolutions on spherical images［C］// Proceedings of the2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2019： 1-5. 10.1109/cvpr42600.2020.01244
72	ZHANG Y， DAI F， MA Y， et al. Saliency prediction network for 360° videos［J］. IEEE Journal of Selected Topics in Signal Processing， 2019， 14（1）： 27-37. 10.1109/jstsp.2019.2955824
73	SU Y-C， GRAUMAN K. Learning spherical convolution for fast features from 360 imagery［J］. Advances in Neural Information Processing Systems， 2017， 30： 529-539.
74	COORS B， CONDURACHE A P， GEIGER A. SphereNet： learning spherical representations for detection and classification in omnidirectional images［C］// Proceedings of the 2018 European Conference on Computer Vision. Cham： Springer， 2018： 525-541. 10.1007/978-3-030-01240-3_32
75	LI J， HAN L， ZHANG C， et al. Spherical convolution empowered viewport prediction in 360 video multicast with limited FoV feedback［J］. ACM Transactions on Multimedia Computing， Communications and Applications， 2023， 19（1）： No. 3. 10.1145/3511603
76	CHOPRA L， CHAKRABORTY S， MONDAL A， et al. PARIMA： Viewport adaptive 360-degree video streaming［C］// Proceedings of the 2021 Web Conference. New York： ACM， 2021： 2379-2391. 10.1145/3442381.3450070
77	TANG J， HUO Y， YANG S， et al. A viewport prediction framework for panoramic videos［C］// Proceedings of the 2020 International Joint Conference on Neural Networks. Piscataway： IEEE， 2020： 1-8. 10.1109/ijcnn48605.2020.9207562
78	FENG X， SWAMINATHAN V， WEI S. Viewport prediction for live 360-degree mobile video streaming using user-content hybrid motion tracking［J］. Proceedings of the ACM on Interactive， Mobile， Wearable and Ubiquitous Technologies， 2019， 3（2）： No. 43. 10.1145/3328914
79	FAN C-L， LEE J， W-C LO， et al. Fixation prediction for 360 video streaming in head-mounted virtual reality［C］// Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video. New York： ACM， 2017： 67-72. 10.1145/3083165.3083180
80	XU Y， DONG Y， WU J， et al. Gaze prediction in dynamic 360° immersive videos［C］// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 5333-5342. 10.1109/cvpr.2018.00559
81	NGUYEN A， YAN Z， NAHRSTEDT K. Your attention is unique： Detecting 360-degree video saliency in head-mounted display for head movement prediction［C］// Proceedings of the 26th ACM International Conference on Multimedia. New York： ACM， 2018： 1190-1198. 10.1145/3240508.3240669
82	LI Y， XU Y， XIE S， et al. Two-layer FoV prediction model for viewport dependent streaming of 360-degree videos［C］// Proceedings of the 2018 International Conference on Communications and Networking in China. Cham： Springer， 2019： 501-509. 10.1007/978-3-030-06161-6_49
83	VAN DER HOOFT J， VEGA M T， PETRANGELI S， et al. Tile-based adaptive streaming for virtual reality video［J］. ACM Transactions on Multimedia Computing， Communications， and Applications， 2019， 15（4）： No. 110. 10.1145/3362101
84	MAHZARI A， TAGHAVI NASRABADI A， SAMIEI A， et al. FoV-aware edge caching for adaptive 360° video streaming［C］// Proceedings of the 26th ACM International Conference on Multimedia. New York： ACM， 2018： 173-181. 10.1145/3240508.3240680
85	JIANG J， SEKAR V， ZHANG H. Improving fairness， efficiency， and stability in http-based adaptive video streaming with festive［J］.IEEE/ACM Transactions on Networking， 2014， 22（1）： 326-340. 10.1109/tnet.2013.2291681
86	YIN X， JINDAL A， SEKAR V， et al. A control-theoretic approach for dynamic adaptive video streaming over HTTP［C］// Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. New York： ACM， 2015： 325-338. 10.1145/2785956.2787486
87	HUANG T-Y， JOHARI R， McKEOWN N， et al. A buffer-based approach to rate adaptation： evidence from a large video streaming service［C］// Proceedings of the 2014 ACM Conference on SIGCOMM. New York： ACM， 2014： 187-198. 10.1145/2619239.2626296
88	DE CICCO L， CALDARALO V， PALMISANO V， et al. ELASTIC： a client-side controller for dynamic adaptive streaming over HTTP （DASH）［C］// Proceedings of the 2013 20th International Packet Video Workshop. Piscataway： IEEE， 2013： 1-8. 10.1109/pv.2013.6691442
89	SPITERI K， URGAONKAR R， SITARAMAN R K. BOLA： near-optimal bitrate adaptation for online videos［J］. IEEE/ACM Transactions on Networking， 2020， 28（4）： 1698-1711. 10.1109/tnet.2020.2996964
90	MAO H， NETRAVALI R， ALIZADEH M. Neural adaptive video streaming with pensieve［C］// Proceedings of the 2017 Conference of the ACM Special Interest Group on Data Communication. New York： ACM， 2017： 197-210. 10.1145/3098822.3098843
91	Encoder settings for Live 360 degree videos［EB/OL］. ［2023-01-13］..
92	NGUYEN D V， TRAN H T T， THANG T C. An evaluation of tile selection methods for viewport-adaptive streaming of 360-degree video［J］. ACM Transactions on Multimedia Computing， Communications， and Applications， 2020， 16（1）： No. 8. 10.1145/3373359
93	ZHOU C， XIAO M， LIU Y. ClusTile： toward minimizing bandwidth in 360-degree video streaming［C］// Proceedings of the IEEE INFOCOM 2018 — IEEE Conference on Computer Communications. Piscataway： IEEE， 2018： 962-970. 10.1109/infocom.2018.8486282
94	GUAN Y， ZHENG C， ZHANG X， et al. Pano： optimizing 360 video streaming with a better understanding of quality perception［C］// Proceedings of the 2019 ACM Special Interest Group on Data Communication. New York： ACM， 2019： 394-407. 10.1145/3341302.3342063
95	ZHANG Y， GUAN Y， BIAN K， et al. EPASS360： QoE-aware 360-degree video streaming over mobile devices［J］. IEEE Transactions on Mobile Computing， 2020， 20（7）： 2338-2353. 10.1109/tmc.2020.2978187
96	FAN C-L， YEN S-C， HUANG C-Y， et al. On the optimal encoding ladder of tiled 360° videos for head-mounted virtual reality［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2020， 31（4）： 1632-1647. 10.1109/tcsvt.2020.3007288
97	ZHANG H， BAN Y， GUO Z， et al. RAM360： robust adaptive multi-layer 360° video streaming with Lyapunov optimization［J］. IEEE Transactions on Multimedia， 2023， 25： 4225-4239. 10.1109/tmm.2022.3172550
98	SUN L， DUANMU F， LIU Y， et al. Multi-path multi-tier 360-degree video streaming in 5G networks［C］// Proceedings of the 9th ACM Multimedia Systems Conference. New York： ACM， 2018： 162-173. 10.1145/3204949.3204978
99	ZARE A， AMINLOU A， HANNUKSELA M M， et al. HEVC-compliant tile-based streaming of panoramic video for virtual reality applications［C］// Proceedings of the 24th ACM International Conference on Multimedia. New York： ACM， 2016： 601-605. 10.1145/2964284.2967292
100	YUAN H， ZHAO S， HOU J， et al. Spatial and temporal consistency-aware dynamic adaptive streaming for 360-degree videos［J］. IEEE Journal of Selected Topics in Signal Processing， 2019， 14（1）： 177-193. 10.1109/jstsp.2019.2957981
101	WEI X， ZHOU M， JIA W. Towards low-latency and high-quality adaptive 360-degree streaming［J］. IEEE Transactions on Industrial Informatics， 2023， 19（5）： 6326-6336. 10.1109/tii.2022.3192398
102	CHEN X， WU D， AHMAD I. Optimized viewport‐adaptive 360‐degree video streaming［J］. CAAI Transactions on Intelligence Technology， 2021， 6（3）： 347-359. 10.1049/cit2.12011
103	ZHANG L， SUO Y， WU X， et al. TBRA： tiling and bitrate adaptation for mobile 360-degree video streaming［C］// Proceedings of the 29th ACM International Conference on Multimedia. New York： ACM， 2021： 4007-4015. 10.1145/3474085.3475590
104	CHEN X， TAN T， CAO G. Popularity-aware 360-degree video streaming［C］// Proceedings of the IEEE INFOCOM 2021 — IEEE Conference on Computer Communications. Piscataway： IEEE， 2021： 1-10. 10.1109/infocom42981.2021.9488856
105	TU Z， ZONG T， XI X， et al. Content adaptive tiling method based on user access preference for streaming panoramic video［C］// Proceedings of the 2018 IEEE International Conference on Consumer Electronics. Piscataway： IEEE， 2018： 1-4. 10.1109/icce.2018.8326152
106	WANG S， YANG S， LI H， et al. SalientVR： Saliency-driven mobile 360-degree video streaming with gaze information［C］// Proceedings of the 28th Annual International Conference on Mobile Computing and Networking. New York： ACM， 2022： 542-555. 10.1145/3495243.3517018
107	WANG S， YANG S， SU H， et al. Robust saliency-driven quality adaptation for mobile 360-degree video streaming［J］. IEEE Transactions on Mobile Computing， 2024， 23（2）： 1312-1329.
108	LILLICRAP T P， HUNT J J， PRITZEL A， et al. Continuous control with deep reinforcement learning［EB/OL］. ［2023-02-01］. .
109	GADALETA M， CHIARIOTTI F， ROSSI M， et al. D-DASH： a deep Q-learning framework for DASH video streaming［J］. IEEE Transactions on Cognitive Communications and Networking， 2017， 3（4）： 703-718. 10.1109/tccn.2017.2755007
110	ZHANG Y， ZHAO P， BIAN K， et al. DRL360： 360-degree video streaming with deep reinforcement learning［C］// Proceedings of the IEEE INFOCOM 2019 — IEEE Conference on Computer Communications. Piscataway： IEEE， 2019： 1252-1260. 10.1109/infocom.2019.8737361
111	FU J， CHEN X， ZHANG Z， et al. 360SRL： a sequential reinforcement learning approach for ABR tile-based 360 video streaming［C］// Proceedings of the 2019 IEEE International Conference on Multimedia and Expo. Piscataway： IEEE， 2019： 290-295. 10.1109/icme.2019.00058
112	KAN N， ZOU J， LI C， et al. RAPT360： reinforcement learning-based rate adaptation for 360-degree video streaming with adaptive prediction and tiling［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2021， 32（3）： 1607-1623. 10.1109/tcsvt.2021.3076585
113	WU C， WANG Z， SUN L. PAAS： a preference-aware deep reinforcement learning approach for 360° video streaming［C］// Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video. New York： ACM， 2021： 34-41. 10.1145/3458306.3460995

算法名称	算法分类	算法概述
Festival	Throughput-based	监测历史网络带宽，基于20个样本数据使用调和平均算法，预测未来的可用带宽，决策对应的码率
MPC	Hybrid	基于缓存器容量、调和平均算法预测得到的网络带宽选择视频码率，从而最大化用户QoE指标。算法的核心是使用MPC的预测和控制机制改善自适应码率分配的算法性能
BBA0	Buffer-based	使用当前缓存区容量选择要下载的下一个片段的码率。使用了缓存区-码率映射函数表示缓存区大小与视频码率之间的离散双射关系
Elastic	Hybrid	基于反馈控制理论，对可用带宽更精确地预测，进而生成更好的码率决策方案
BOLA	Buffer-based	基于当前缓存区的容量进行码率决策，更新上一个chunk的码率，将自适应码率分配建模为效用函数最大化求解问题，使用Lyapunov优化方法进行求解，可以适应不同的观看设备、视频种类与网络条件等
Pensieve	Learning-based	首次提出基于Actor-Critic模型进行码率决策，接收播放器状态信息，学习码率分配方案

算法名称	算法分类	算法概述
Festival	Throughput-based	监测历史网络带宽，基于20个样本数据使用调和平均算法，预测未来的可用带宽，决策对应的码率
MPC	Hybrid	基于缓存器容量、调和平均算法预测得到的网络带宽选择视频码率，从而最大化用户QoE指标。算法的核心是使用MPC的预测和控制机制改善自适应码率分配的算法性能
BBA0	Buffer-based	使用当前缓存区容量选择要下载的下一个片段的码率。使用了缓存区-码率映射函数表示缓存区大小与视频码率之间的离散双射关系
Elastic	Hybrid	基于反馈控制理论，对可用带宽更精确地预测，进而生成更好的码率决策方案
BOLA	Buffer-based	基于当前缓存区的容量进行码率决策，更新上一个chunk的码率，将自适应码率分配建模为效用函数最大化求解问题，使用Lyapunov优化方法进行求解，可以适应不同的观看设备、视频种类与网络条件等
Pensieve	Learning-based	首次提出基于Actor-Critic模型进行码率决策，接收播放器状态信息，学习码率分配方案

[1]	Aiguo JI, Yunzhe LUAN. Buffer compensation based video bitrate adaptation algorithm [J]. Journal of Computer Applications, 2022, 42(9): 2816-2822.
[2]	HUANG Sheng, FU Yuanpeng, ZHANG Qianyun. Rate smooth switching algorithm based on DASH standard [J]. Journal of Computer Applications, 2019, 39(4): 1122-1126.
[3]	HUANG Sheng, HU Lingwei, FU Yuanpeng. State machine based video rate adaptation algorithm [J]. Journal of Computer Applications, 2018, 38(7): 2001-2004.
[4]	LIN Chang, LI Guoping, ZHAO Haiwu, WANG Guozhong, GU Xiao. Double ring mapping projection for panoramic video [J]. Journal of Computer Applications, 2017, 37(9): 2631-2635.
[5]	YUAN Bin, LI Wenwei. Design and implementation of QoE measuring tool for personal cloud storage services [J]. Journal of Computer Applications, 2016, 36(9): 2409-2415.
[6]	Jing-hua LIN Wei-min LEI Ling-nan LI Song BAI. Optimizing trick-play responsiveness in IPTV application [J]. Journal of Computer Applications, 2009, 29(11): 2897-2900.

Survey on tile-based viewport adaptive streaming scheme of panoramic video

全景视频基于块的视口自适应传输方案综述

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 113

Related Articles 6

Recommended Articles

Metrics