Nowadays, 360° video/image has been increasingly popular and drawn great attention. The spherical viewing range of 360° video/image accounts for huge data, which pose the challenges to 360° video/image processing in solving the bottleneck of storage, transmission, etc. Accordingly, the recent years have witnessed the explosive emergence of works on 360° video/image processing. In this article, we review the state-of-the-art works on 360° video/image processing from the aspects of perception, assessment and compression. First, this article reviews both datasets and visual attention modelling approaches for 360° video/image. Second, we survey the related works on both subjective and objective visual quality assessment (VQA) of 360° video/image. Third, we overview the compression approaches for 360° video/image, which either utilize the spherical characteristics or visual attention models. Finally, we summarize this overview article and outlook the future research trends on 360° video/image processing.

State-of-the-Art in 360° Video/Image Processing: Perception, Assessment and Compression

Service-aware 6G: An intelligent and open network based on the convergence of communication, computing and caching

In this chapter we are going to explore games that all have certain similarities:

a)

They are contests between two players.




b)

They terminate after a finite number of moves.




c)

There are no elements of chance involved (no dice are used or cards shuffled) . The result is that the game is determined solely by the players’ decisions.




d)

Both players can freely observe each other’s moves and evaluate the changes that these make in the game.

Games of Strategy

In this paper, we propose a novel deep learning-based virtual reality image quality assessment method that automatically predicts the visual quality of an omnidirectional image. In order to assess the visual quality in viewing the omnidirectional image, we propose deep networks consisting of virtual reality (VR) quality score predictor and human perception guider. The proposed VR quality score predictor learns the positional and visual characteristics of the omnidirectional image by encoding the positional feature and visual feature of a patch on the omnidirectional image. With the encoded positional feature and visual feature, patch weight and patch quality score are estimated. Then, by aggregating all weights and scores of the patches, the image quality score is predicted. The proposed human perception guider evaluates the predicted quality score by referring to the human subjective score (i.e., ground-truth obtained by subjects) using an adversarial learning. With adversarial learning, the VR quality score predictor is trained to accurately predict the quality score in order to deceive the guider, while the proposed human perception guider is trained to precisely distinguish between the predictor score and the ground-truth subjective score. To verify the performance of the proposed method, we conducted comprehensive subjective experiments and evaluated the performance of the proposed method. The experimental results show that the proposed method outperforms the existing two-dimentional image quality models and the state-of-the-art image quality models for omnidirectional images.

Deep Virtual Reality Image Quality Assessment With Human Perception Guider for Omnidirectional Image

Objective quality assessment plays a crucial role in the evaluation and optimization processes of Virtual Reality (VR) technologies, for which state-of-the-art objective quality evaluation metrics for omnidirectional video, i.e., 360 degree video, are typically derived from traditional MSE (or PSNR). Here we propose an objective omnidirectional video quality assessment method based on structural similarity (SSIM) in the spherical domain. Adopting the relationship of the structural similarity between the 2-D plane and sphere, the interference brought by the projection between the two domains can be well handled in the assessment process. The performance of the proposed spherical structural similarity (S-SSIM) index is evaluated with a subjective omnidirectional video quality assessment database. As demonstrated in the experimental results, the proposed S-SSIM outperforms state-of-the-art objective quality assessment metrics in omnidirectional video quality assessment.

Spherical Structural Similarity Index for Objective Omnidirectional Video Quality Assessment

With the development of virtual reality, higher quality panoramic videos are in great demand to guarantee the immersive viewing experience. Therefore, quality assessment attaches much importance to correlated technologies. Considering the geometric transformation in projection and the limited resolution of head-mounted device (HMD), a modified display protocol of the high resolution sequences for the subjective rating test is proposed, in which an optimal display resolution is determined based on the geometry constraints between screen and human eyes. By sampling the videos to the optimal resolution before coding, the proposed method significantly alleviates the interference of HMD sampling while displaying, thus ensuring the reliability of subjective quality opinion in terms of video coding. Using the proposed display protocol, a subjective quality database for panoramic videos is established for video coding applications. The proposed database contains 50 distorted sequences obtained from ten raw panoramic video sequences. Distortions are introduced with the High Efficiency Video Coding compression. Each sequence is evaluated by 30 subjects on video quality, following the absolute category rating with hidden reference method. The rating scores and differential mean opinion scores (DMOSs) are recorded and included in the database. With the proposed database, several state-of-the-art objective quality assessment methods are further evaluated with correlation analysis. The database, including the video sequences, subjective rating scores and DMOS, can be used to facilitate future researches on coding applications.

Subjective Panoramic Video Quality Assessment Database for Coding Applications

Rate control plays an important role in practical video coding. In this paper, we present a rate control method for Versatile Video Coding (VVC). We conduct some statistical analysis to investigate the influence of the skip block upon the rate-distortion parameter estimation and quality dependency among frames. Based on the analysis, we propose the rate-distortion parameter updating strategy and establish a quality dependency factor based frame-level bit allocation scheme. Experimental results show that the proposed method can achieve up to 7.31%/5.44% BD-rate improvements in Low-Delay/Random-Access configuration upon previous rate control method in VVC respectively. The presented rate control algorithm has already been adopted into VVC VTM platform by Joint Video Experts Team (JVET).

Rate Control For Versatile Video Coding

In this paper, we provide the description of our approach designed for participating the CVPR 2018 Challenge on Learned Image Compression (CLIC). Our approach is a hybrid image coder based on CNN-optimized in-loop filter and mode coding, with uncertainty based resource allocation for compressing the task images. Two solutions were submitted, i.e., “iipTiramisu” and its speedup version “iipTiramisuS”, resulting in 32.14 dB and 32.06 dB in PSNR, respectively. These two results have been ranked No. 1 and 2 on the leaderboard.

/pdf/cnn-optimized-image-compression-with-uncertainty-based-45obdikv2t.pdf

CNN-Optimized Image Compression with Uncertainty based Resource Allocation

One of the key challenges in no-reference video quality assessment (NR-VQA) is the absence of the reference video to measure the similarity or difference between the distorted video and the original one. In this paper, an encoder-decoder model is proposed to predict pixel-by-pixel similarity maps from the distorted video. The model takes multiple frames as input since correlated pixels of adjacent frames can be exploited to recover the similarity map of the middle frame of the distorted video clip. In addition, to further exploit the temporal perception mechanism of the human visual system (HVS), which is relevant to the perceptual video distortion measurement, visual persistence and temporal memory effects are considered in the spatio-temporal pooling network design. Experimental results demonstrate that our proposed method outperforms state-of-the-art NR-VQA metrics.

No-Reference Video Quality Assessment Based On Similarity Map Estimation

Region of Interests (ROI) coding has shown advantages in subjectively optimized video coding. In this paper, we propose a new CTU-Ievel rate allocation scheme based on the Stackelberg Game model to enhance the visual quality of ROI which formulates the rate allocation process as a noncooperative Stackelberg Game between the ROI and the non-ROI. In this game, ROI is the leader who takes the priority. Based on the formulated game, the rate allocation problem can be expressed as a utility optimization problem. By solving the corresponding utility optimization problem, a novel CTU-level rate allocation strategy is established, in which a trade-off between the ROI's quality and the overall quality can be achieved. The experimental results show that our proposed scheme can improve the quality of ROI significantly with a negligible overall quality degradation.

Zizheng Liu

Papers

Subjective Panoramic Video Quality Assessment Database for Coding Applications

Rate Control For Versatile Video Coding

CNN-Optimized Image Compression with Uncertainty based Resource Allocation

No-Reference Video Quality Assessment Based On Similarity Map Estimation

Stackelberg Game Based Rate Allocation for HEVC Region of Interest Coding