In region-of-interest (ROI)-based video coding, ROI parts of the frame are encoded with higher quality than non-ROI parts. At low bit rates, such encoding may produce attention-grabbing coding artifacts, which may draw viewer's attention away from ROI, thereby degrading visual quality. In this paper, we present a saliency-aware video compression method for ROI-based video coding. The proposed method aims at reducing salient coding artifacts in non-ROI parts of the frame in order to keep user's attention on ROI. Further, the method allows saliency to increase in high quality parts of the frame, and allows saliency to reduce in non-ROI parts. Experimental results indicate that the proposed method is able to improve visual quality of encoded video relative to conventional rate distortion optimized video coding, as well as two state-of-the art perceptual video coding methods.

Saliency-Aware Video Compression

We propose a rate-distortion optimization (RDO) scheme based on the structural similarity (SSIM) index, which was found to be a better indicator of perceived image quality than mean-squared error, but has not been fully exploited in the context of image and video coding. At the frame level, an adaptive Lagrange multiplier selection method is proposed based on a novel reduced-reference statistical SSIM estimation algorithm and a rate model that combines the side information with the entropy of the transformed residuals. At the macroblock level, the Lagrange multiplier is further adjusted based on an information theoretical approach that takes into account both the motion information content and perceptual uncertainty of visual speed perception. Finally, the mode for H.264/AVC coding is selected by the SSIM index and the adjusted Lagrange multiplier. Extensive experiments show that the proposed scheme can achieve significantly better rate-SSIM performance and provide better visual quality than conventional RDO coding schemes.

/pdf/ssim-motivated-rate-distortion-optimization-for-video-coding-1dhl6c6hnj.pdf

SSIM-Motivated Rate-Distortion Optimization for Video Coding

In today's hybrid video coding, Rate-Distortion Optimization (RDO) plays a critical role. It aims at minimizing the distortion under a constraint on the rate. Currently, the most popular RDO algorithm for one-pass coding is the one recommended in the H.264/AVC reference software. It, or HR-lambda for convenience, is actually a kind of universal method which performs the optimization only according to the quantization process while ignoring the properties of input sequences. Intuitively, it is not efficient all the time and an adaptive scheme should be better. Therefore, a new algorithm Lap- lambda is presented in this paper. Based on the Laplace distribution of transformed residuals, the proposed Lap-lambda is able to adaptively optimize the input sequences so that the overall coding efficiency is improved. Cases which cannot be well captured by the proposed models are considered via escape methods. Comprehensive simulations verify that compared with HR-lambda , Lap-lambda shows a much better or similar performance in all scenarios. Particularly, significant gains of 1.79 dB and 1.60 dB in PSNR are obtained for slow sequences and B-frames, respectively.

Laplace Distribution Based Lagrangian Rate Distortion Optimization for Hybrid Video Coding

H.264/AVC encoder employs a complex mode-decision technique based on rate-distortion optimization. It calculates rate-distortion cost (RDcost) for all possible modes to choose the best one having the minimum RDcost. This paper presents a frame-layer rate control for H.264/AVC that computes the Lagrange multiplier (/spl lambda//sub MODE/) for mode decision by using a quantization parameter (QP) which may be different from that used for encoding. At the same time, we also compare actual bits produced by previous macroblocks (MBs) with the total bits allocated to these MBs to further modify /spl lambda//sub MODE/. The objective of these measures aims to produce bits as close to the frame target bits for rate control as possible. This is very important in the case of low-bit-rate tight buffer applications. In order to obtain an accurate QP for a frame, we employ a complexity-based bit-allocation scheme and a QP adjustment method. Simulation results comparing with the H.264 Joint Video Team (JVT) rate control method show that the H.264 encoder, using the proposed algorithm, achieves a visual quality improvement of about 0.56 dB, performs better for buffer overflow and underflow, and achieves a smaller PSNR deviation.

On Lagrange multiplier and quantizer adjustment for H.264 frame-layer video rate control

It is generally recognized that severe video distortions that are transient in space and/or time have a large effect on overall perceived video quality. In order to understand this phenomena, we study the distribution of spatio-temporally local quality scores obtained from several video quality assessment (VQA) algorithms on videos suffering from compression and lossy transmission over communication channels. We propose a content adaptive spatial and temporal pooling strategy based on the observed distribution. Our method adaptively emphasizes “worst” scores along both the spatial and temporal dimensions of a video sequence and also considers the perceptual effect of large-area cohesive motion flow such as egomotion. We demonstrate the efficacy of the method by testing it using three different VQA algorithms on the LIVE Video Quality database and the EPFL-PoliMI video quality database.

/pdf/video-quality-pooling-adaptive-to-perceptual-distortion-16msuc8bxe.pdf

Video Quality Pooling Adaptive to Perceptual Distortion Severity

In recent years, rate control plays an increasing important role in real-time video communication applications using MPEG-4 AVC/H.264. An important step in many existing rate control algorithms, which employs the quadratic rate-distortion (R-D) model, is to determine the target bits for each P frame. This paper aims in improving video distortion, due to high motions or scene changes, by more accurately predicting frame complexity using the statistics of previously encoded frames. We use mean absolute difference (MAD) ratio as a measure for global frame encoding complexity. Bit budget is allocated to frames according to their MAD ratio, combined with the bits computed based on their buffer status. Simulation results show that the H.264 coder, using our proposed algorithm with virtually little computational complexity added, effectively alleviates PSNR surges and sharp drops for frames caused by high motions or scene changes.

Improved frame-layer rate control for H.264 using MAD ratio

This paper proposes a new simple and efficient method to detect abrupt scene change based on only pixel values. Conventional pixel-based techniques can produce a significant number of false detections and missed detections when high motion and brightness variations are present in the video. To increase scene change detection accuracy yet maintain a low computational complexity, a two-phase strategy is developed. Frames are first tested against the mean absolute frame differences (MAFD) via a relaxed threshold, which rejects about 90% of the non-scene change frames. The rest 10% of the frames are then normalized via a histogram equalization process. A top-down approach is applied to refine the decision via four features: MAFD and three other features based on normalized pixel values - signed difference of mean absolute frame difference (SDMAFD*), absolute difference of frame variance (ADFV*), and mean absolute frame differences (MAFD*). Experimental results show that our method contributes to higher detection rate and lower missed detection rate while maintaining a low computational complexity, which is attractive for real-time video applications.

Fast pixel-based video scene change detection

Motion estimation is a critical yet computationally intensive task for video encoding. In this paper, we present an enhancement over a normalized partial distortion search (NPDS) algorithm to further reduce block matching motion estimation complexity while retaining video fidelity. The novelty of our algorithm is that, in addition to the halfway-stop technique in NPDS, a dual-halfway-stop (DHS) method, which is based on a dynamic threshold, is proposed, so that block matching is not performed against all matching candidates. An adaptive search range (ASR) mechanism based on inter block distortion further constrains the searching process. Simulation results show that the proposed algorithm has a remarkable computational speedup when compared to that of full search and NPDS algorithms. Particularly, it requires less computation by 92-99% and encounters an average of only 0.08 dB PSNR video degradation when compared to that of full search. The speedup is also very significant when compared to that of fast motion estimation algorithms. This paper describes our work that led to our joint video team (JVT) adopted contribution (included in software JM 10.1 onwards) as well as later enhancements, collectively known as simplified and unified multi-hexagon search (SUMH), a simplified fast motion estimation.

Improved Normalized Partial Distortion Search With Dual-Halfway-Stop for Rapid Block Motion Estimation

In this paper, we propose an efficient and practical algorithm to dynamically adapt the Lagrange multipliers for each macroblock based on the context of the neighboring or upper layer blocks to improve rate-distortion performance. Our method improves the accuracy for the detection of true motion vectors as well as the most efficient encoding modes for luma, which are used for deriving the motion vectors, and modes for chroma. Simulation results for H.264/advanced video coding video demonstrate that our method reduces bit rate significantly and achieves peak signal-to-noise ratio gain over those of the joint model (JM) software for all sequences tested, with negligible extra computational cost. The improvement is particularly significant for high motion high-resolution videos. This paper describes our work that led to our Joint Video Team adopted contribution (included in software JM 12.0 onward), collectively known as context adaptive Lagrange multiplier (CALM).

Xiaoquan Yi

Papers

Improved frame-layer rate control for H.264 using MAD ratio

Fast pixel-based video scene change detection

Improved Normalized Partial Distortion Search With Dual-Halfway-Stop for Rapid Block Motion Estimation

Context Adaptive Lagrange Multiplier (CALM) for Rate-Distortion Optimal Motion Estimation in Video Coding

Improved H.264 rate control by enhanced MAD-based frame complexity prediction