Nowadays, 360° video/image has been increasingly popular and drawn great attention. The spherical viewing range of 360° video/image accounts for huge data, which pose the challenges to 360° video/image processing in solving the bottleneck of storage, transmission, etc. Accordingly, the recent years have witnessed the explosive emergence of works on 360° video/image processing. In this article, we review the state-of-the-art works on 360° video/image processing from the aspects of perception, assessment and compression. First, this article reviews both datasets and visual attention modelling approaches for 360° video/image. Second, we survey the related works on both subjective and objective visual quality assessment (VQA) of 360° video/image. Third, we overview the compression approaches for 360° video/image, which either utilize the spherical characteristics or visual attention models. Finally, we summarize this overview article and outlook the future research trends on 360° video/image processing.

State-of-the-Art in 360° Video/Image Processing: Perception, Assessment and Compression

In this paper, we propose a novel deep learning-based virtual reality image quality assessment method that automatically predicts the visual quality of an omnidirectional image. In order to assess the visual quality in viewing the omnidirectional image, we propose deep networks consisting of virtual reality (VR) quality score predictor and human perception guider. The proposed VR quality score predictor learns the positional and visual characteristics of the omnidirectional image by encoding the positional feature and visual feature of a patch on the omnidirectional image. With the encoded positional feature and visual feature, patch weight and patch quality score are estimated. Then, by aggregating all weights and scores of the patches, the image quality score is predicted. The proposed human perception guider evaluates the predicted quality score by referring to the human subjective score (i.e., ground-truth obtained by subjects) using an adversarial learning. With adversarial learning, the VR quality score predictor is trained to accurately predict the quality score in order to deceive the guider, while the proposed human perception guider is trained to precisely distinguish between the predictor score and the ground-truth subjective score. To verify the performance of the proposed method, we conducted comprehensive subjective experiments and evaluated the performance of the proposed method. The experimental results show that the proposed method outperforms the existing two-dimentional image quality models and the state-of-the-art image quality models for omnidirectional images.

Deep Virtual Reality Image Quality Assessment With Human Perception Guider for Omnidirectional Image

Objective quality assessment plays a crucial role in the evaluation and optimization processes of Virtual Reality (VR) technologies, for which state-of-the-art objective quality evaluation metrics for omnidirectional video, i.e., 360 degree video, are typically derived from traditional MSE (or PSNR). Here we propose an objective omnidirectional video quality assessment method based on structural similarity (SSIM) in the spherical domain. Adopting the relationship of the structural similarity between the 2-D plane and sphere, the interference brought by the projection between the two domains can be well handled in the assessment process. The performance of the proposed spherical structural similarity (S-SSIM) index is evaluated with a subjective omnidirectional video quality assessment database. As demonstrated in the experimental results, the proposed S-SSIM outperforms state-of-the-art objective quality assessment metrics in omnidirectional video quality assessment.

Spherical Structural Similarity Index for Objective Omnidirectional Video Quality Assessment

Compression has been an important research topic for many decades, to produce a significant impact on data transmission and storage. Recent advances have shown a great potential of learning based image and video compression. Inspired from related works, in this paper, we present an image compression architecture using a convolutional autoencoder, and then generalize image compression to video compression, by adding an interpolation loop into both encoder and decoder sides. Our basic idea is to realize spatial-temporal energy compaction in learning image and video compression. Thereby, we propose to add a spatial energy compaction-based penalty into loss function, to achieve higher image compression performance. Furthermore, based on temporal energy distribution, we propose to select the number of frames in one interpolation loop, adapting to the motion characteristics of video contents. Experimental results demonstrate that our proposed image compression outperforms the latest image compression standard with MS-SSIM quality metric, and provides higher performance compared with state-of-the-art learning compression methods at high bit rates, which benefits from our spatial energy compaction approach. Meanwhile, our proposed video compression approach with temporal energy compaction can significantly outperform MPEG-4, and is competitive with commonly used H.264. Both our image and video compression can produce more visually pleasant results than traditional standards.

/pdf/learning-image-and-video-compression-through-spatial-4ms45qvvvp.pdf

Learning Image and Video Compression Through Spatial-Temporal Energy Compaction

Nowadays, 360° video/image has been increasingly popular and drawn great attention. The spherical viewing range of 360° video/image accounts for huge data, which pose the challenges to 360° video/image processing in solving the bottleneck of storage, transmission, etc. Accordingly, the recent years have witnessed the explosive emergence of works on 360° video/image processing. In this paper, we review the state-of-the-art works on 360° video/image processing from the aspects of perception, assessment and compression. First, this paper reviews both datasets and visual attention modelling approaches for 360° video/image. Second, we survey the related works on both subjective and objective visual quality assessment (VQA) of 360° video/image. Third, we overview the compression approaches for 360° video/image, which either utilize the spherical characteristics or visual attention models. Finally, we summarize this overview paper and outlook the future research trends on 360° video/image processing.

State-of-the-art in 360{\deg} Video/Image Processing: Perception, Assessment and Compression

With the development of virtual reality, higher quality panoramic videos are in great demand to guarantee the immersive viewing experience. Therefore, quality assessment attaches much importance to correlated technologies. Considering the geometric transformation in projection and the limited resolution of head-mounted device (HMD), a modified display protocol of the high resolution sequences for the subjective rating test is proposed, in which an optimal display resolution is determined based on the geometry constraints between screen and human eyes. By sampling the videos to the optimal resolution before coding, the proposed method significantly alleviates the interference of HMD sampling while displaying, thus ensuring the reliability of subjective quality opinion in terms of video coding. Using the proposed display protocol, a subjective quality database for panoramic videos is established for video coding applications. The proposed database contains 50 distorted sequences obtained from ten raw panoramic video sequences. Distortions are introduced with the High Efficiency Video Coding compression. Each sequence is evaluated by 30 subjects on video quality, following the absolute category rating with hidden reference method. The rating scores and differential mean opinion scores (DMOSs) are recorded and included in the database. With the proposed database, several state-of-the-art objective quality assessment methods are further evaluated with correlation analysis. The database, including the video sequences, subjective rating scores and DMOS, can be used to facilitate future researches on coding applications.

Subjective Panoramic Video Quality Assessment Database for Coding Applications

In this paper, a multi-objective optimization of quality based coding tree unit (CTU) level rate control method, named MORC, for low-delay video coding of Versatile Video Coding (VVC) is proposed. Unlike the traditional rate control algorithms, objectives of minimizing average distortion and minimizing quality fluctuation are introduced and considered simultaneously to obtain the optimal solution. Weighting method is then used to convert the multi-objective optimization problem into a simplistic single-objective problem. Accordingly, given the target rate constraint, rate allocation is formulated as a convex optimization problem to achieve better trade-off among three multiple objectives: minimizing average video distortion, meeting rate constraint and minimizing video quality fluctuation. Based on the D- $\lambda $ model, a two-stage method is further proposed to solve the optimization problem. Experiments have been conducted on the test model of VVC, which is VTM 3.0. Compared with the state-of-the-art rate control algorithms, the proposed method can obtain the Pareto optimal solution, which dominates the solution of other methods and is able to achieve better trade-off among multiple objectives.

Multi-Objective Optimization of Quality in VVC Rate Control for Low-Delay Video Coding

In this paper, we provide the description of our approach designed for participating the CVPR 2018 Challenge on Learned Image Compression (CLIC). Our approach is a hybrid image coder based on CNN-optimized in-loop filter and mode coding, with uncertainty based resource allocation for compressing the task images. Two solutions were submitted, i.e., “iipTiramisu” and its speedup version “iipTiramisuS”, resulting in 32.14 dB and 32.06 dB in PSNR, respectively. These two results have been ranked No. 1 and 2 on the leaderboard.

/pdf/cnn-optimized-image-compression-with-uncertainty-based-45obdikv2t.pdf

CNN-Optimized Image Compression with Uncertainty based Resource Allocation

Spectral decorrelation has been considered an important approach in multispectral image compression to remove the redundancy among bands. In this paper, we propose a novel adaptive spectral decorrelation method based on clustering analysis for Moderate Resolution Imaging Spectroradiometer (MODIS) image compression. The remote sensing image bands are divided into different clusters by the method based on density peak clustering. Then, reversible Karhunen–Loeve transform and polynomial least square estimation are employed to reduce the band redundancy, achieving an effective spectral decorrelation. As shown by the experimental results, our method achieves remarkable bit-saving when compared to the state-of-the-art algorithms on MODIS image data set.

An Adaptive Spectral Decorrelation Method for Lossless MODIS Image Compression

In this paper, an LSTM based rate-distortion (R-D) prediction method for low-delay video coding has been proposed. Unlike the traditional rate control algorithms, LSTM is introduced to learn the latent pattern of the R-D relationship in the progress of video coding. Temporal information, hierarchical coding structure information and the content of the frame which is to be encoded have been used to achieve more accurate prediction. Based on the proposed network, a new R-D model parameters prediction method is proposed and tested on test model of Versatile Video Coding (VVC). According to the experimental results, compared with the state-of-the-art method used in VVC, the proposed method can achieve better performance.

Feiyang Liu

Papers

Subjective Panoramic Video Quality Assessment Database for Coding Applications

Multi-Objective Optimization of Quality in VVC Rate Control for Low-Delay Video Coding

CNN-Optimized Image Compression with Uncertainty based Resource Allocation

An Adaptive Spectral Decorrelation Method for Lossless MODIS Image Compression

An LSTM based Rate and Distortion Prediction Method for Low-delay Video Coding