With increasing advancements in technologies for capturing 360° videos, advances in streaming such videos have become a popular research topic. However, streaming 360° videos require high bandwidth, thus escalating the need for developing optimized streaming algorithms. Researchers have proposed various methods to tackle the problem, considering the network bandwidth or attempt to predict future viewports in advance. However, most of the existing works either (1) do not consider video contents to predict user viewport, or (2) do not adapt to user preferences dynamically, or (3) require a lot of training data for new videos, thus making them potentially unfit for video streaming purposes. We develop PARIMA, a fast and efficient online viewport prediction model that uses past viewports of users along with the trajectories of prime objects as a representative of video content to predict future viewports. We claim that the head movement of a user majorly depends upon the trajectories of the prime objects in the video. We employ a pyramid-based bitrate allocation scheme and perform a comprehensive evaluation of the performance of PARIMA. In our evaluation, we show that PARIMA outperforms state-of-the-art approaches, improving the Quality of Experience by over 30% while maintaining a short response time.

PARIMA: Viewport Adaptive 360-Degree Video Streaming

Virtual reality: a survey of enabling technologies and its applications in IoT

With increasing advancements in technologies for capturing 360° videos, advances in streaming such videos have become a popular research topic. However, streaming 360° videos require high bandwidth, thus escalating the need for developing optimized streaming algorithms. Researchers have proposed various methods to tackle the problem, considering the network bandwidth or attempt to predict future viewports in advance. However, most of the existing works either (1) do not consider video contents to predict user viewport, or (2) do not adapt to user preferences dynamically, or (3) require a lot of training data for new videos, thus making them potentially unfit for video streaming purposes. We develop PARIMA, a fast and efficient online viewport prediction model that uses past viewports of users along with the trajectories of prime objects as a representative of video content to predict future viewports. We claim that the head movement of a user majorly depends upon the trajectories of the prime objects in the video. We employ a pyramid-based bitrate allocation scheme and perform a comprehensive evaluation of the performance of PARIMA. In our evaluation, we show that PARIMA outperforms state-of-the-art approaches, improving the Quality of Experience by over 30\% while maintaining a short response time.

360-degree video streaming shows great potential to revolutionize the streaming market, by providing much better immersive experience than standard video streams. However, its wide adoption is hindered by the surging demand of network bandwidth due to multi-screen video transmission. To reduce the bandwidth cost, one promising approach is to predict a user’s field of view (FoV), and then prefetch video tiles that a user will view a few seconds ahead. The challenge lies in that user behaviors cannot be properly captured with very limited information, especially the viewing time spent on each tile and the FoV switching behavior are hard to predict. In this paper, we propose a novel 360-degree video streaming algorithm called TVG-Streaming to optimize user experiences by learning user view behaviors. Different from previous approaches, our idea is to exploit tile-view graphs (TVGs) generated by real user behaviors and accurately estimate the probability that each tile falls in the FoV. With the tile view probability, we can determine the bitrate of each tile for delivery and buffering with limited bandwidth budget so as to maximize users’ quality of experience (QoE). For evaluation, we conduct extensive experiments using real traces and the results show that our proposed TVG-Streaming algorithm significantly outperforms other algorithms by at least 20% improvement in terms of users’ QoE.

TVG-Streaming : Learning User Behaviors for QoE-Optimized 360-Degree Video Streaming

Tile-based streaming techniques have been widely used to save bandwidth in 360° video streaming. However, it is a challenge to determine the right tile size which directly affects the bandwidth usage. To address this problem, we propose to encode the video by considering the viewing popularity, where the popularly viewed areas are encoded as macrotiles to save bandwidth. We propose techniques to identify and build macrotiles, and adjust their sizes considering practical issues such as head movement randomness. In some cases, a user’s viewing area may not be covered by the constructed macrotiles, and then the conventional tiling scheme is used. To support popularity-aware 360° video streaming, the client selects the right tiles (a macrotile or a set of conventional tiles) with the right quality level to maximize the QoE under bandwidth constraint. We formulate this problem as an optimization problem which is NP-hard, and then propose a heuristic algorithm to solve it. Through extensive evaluations based on real traces, we demonstrate that the proposed algorithm can significantly improve the QoE and save the bandwidth usage.

Popularity-Aware 360-Degree Video Streaming

360-degree videos have gained increasing popularity due to its capability to provide users with immersive viewing experience. Given the limited network bandwidth, it is a common approach to only stream video tiles in the user's Field-of-View (FoV) with high quality. However, it is difficult to perform accurate FoV prediction due to diverse user behaviors and time-varying network conditions. In this paper, we re-design the 360-degree video streaming systems by leveraging the technique of super-resolution (SR). The basic idea of our proposed SR360 framework is to utilize abundant computation resources on the user devices to trade off a reduction of network bandwidth. In the SR360 framework, a video tile with low resolution can be boosted to a video tile with high resolution using SR techniques at the client side. We adopt the theory of deep reinforcement learning (DRL) to make a set of decisions jointly, including user FoV prediction, bitrate allocation and SR enhancement. By conducting extensive trace-driven evaluations, we compare the performance of our proposed SR360 with other state-of-the-art methods and the results show that SR360 significantly outperforms other methods by at least 30% on average under different QoE metrics.

SR360: boosting 360-degree video streaming with super-resolution

Recent advancements of neural-enhanced techniques, especially super-resolution (SR), show great potential in revolutionizing the landscape of Internet video delivery. However, there are still quite a few key questions (e.g., how to choose a proper resolution configuration for training samples, how to set the training patch size, how to perform the best patch selection, how to set the update frequency of SR model) that have not been well investigated and understood. In this paper, we perform a dedicated measurement study to revisit super-resolution techniques for Internet video streaming. Our measurements are based on real-world video datasets, and the results provide a number of important insights: (1) It is possible that the SR model trained with low-resolution patches (e.g., (540p, 1080p) pairs) can achieve almost the same performance as that trained with high-resolution patches (e.g., (1080p, 2160p) pairs); (2) Compared to the saliency of training patches, the size of training patches has little impact on the performance of trained SR model; (3) The improvement of video quality brought by more frequent SR model update is not very significant. We also discuss the implications of our findings for system design, and we believe that our work is essential for paving the way for the success of future neural-enhanced video streaming systems.

Revisiting super-resolution for internet video streaming

The prevalence of personal devices motivates the rapid development of crowdsourced livecast in recent years. However, there exists huge diversity of upstream bandwidth among amateur broadcasters. Moreover, the highest video quality that can be streamed is limited by the hardware configuration of broadcaster devices (e.g., 540p for low-end mobile devices). The above factors pose significant challenges to the ingestion of high-resolution live video streams, and result in poor quality-of-experience (QoE) for viewers. In this paper, we propose a novel live video ingest approach called CrowdSR for crowdsourced livecast. CrowdSR can transform a low-resolution video stream uploaded by weak devices into a high-resolution video stream via super-resolution, and then deliver the stream to viewers. CrowdSR can exploit crowdsourced high-resolution video patches from similar broadcasters to speedup model training. Different from previous work, our approach does not require any modification at the client side, and thus is more practical and easy to implement. Finally, we implement and evaluate CrowdSR by conducting a series of real-world experiments. The results show that CrowdSR significantly outperforms the baseline approaches by 0.42-1.09 dB in terms of PSNR and 0.006-0.014 in terms of SSIM.

CrowdSR: enabling high-quality video ingest in crowdsourced livecast via super-resolution

The course of parallel programming is becoming more and more important for the education of students majoring in computer science. However, it is not easy to learn parallel programming well due to its high theory and practice requirements. In this paper, we design and implement an automatic assignment generation and assessment system to help students learn parallel programming. The assignments can be generated according to user behaviors and thus able to guide students to learn parallel programming step by step. Besides, it can automatically generate an overall assessment of student assignments by using fuzzy string matching, which provides an approximate reference score of objective questions. Subjective questions can be assessed directly by comparing the answer to the reference answer. This system also provides a friendly user interface for students to complete online assignments and let teachers manage their question database. In our teaching practice, students can learn parallel programming more effectively with the help of such an assignment generation and assessment system.

Automatic Generation and Assessment of Student Assignments for Parallel Programming Learning

HPC (High Performance Computing) is of great significance due to its excellent performance in computing acceleration. However, unlike other techniques in computer science, learning HPC requires advanced computing resources such as large-scale clusters which directly increase the cost of study for students. To help students to learn HPC programming easily, we design and develop a lightweight container-based experimental platform to provide students with easily accessible and customizable HPC practice environments. In our platform, we integrate multiple practical functional modules for students, teachers, and administrators respectively. It is convenient for a user to access high-performance computing resources via a web portal, use highly customizable basic environments and have nice graphical hands-on interactive HPC learning experiences from our platform.

Zelong Wang

Papers

SR360: boosting 360-degree video streaming with super-resolution

Revisiting super-resolution for internet video streaming

CrowdSR: enabling high-quality video ingest in crowdsourced livecast via super-resolution

Automatic Generation and Assessment of Student Assignments for Parallel Programming Learning

Building a Lightweight Container-Based Experimental Platform for HPC Education.