There is disclosed a method, an apparatus, a server, a client and a non-transitory computer readable medium comprising a computer program stored therein for video coding and decoding. Depth pictures from a plurality of viewing angles are projected into a single viewing angle, making it possible to have pixel-wise joint filtering to be applied to all projected depth values. This approach enables to suppress the noise in the depth map data and provides improved performance for a view synthesis.

Apparatus, a method and a computer program for video coding and decoding

Kalman filter is a minimum-variance estimation for dynamic systems and has attracted much attention with the increasing demands of target tracking. Various algorithms of Kalman filter was proposed for deriving optimal state estimation in the last thirty years. This paper briefly surveys the recent developments about Kalman filter (KF), Extended Kalman filter (EKF) and Unscented Kalman filter (UKF). The basic theories of Kalman filter are introduced, and the merits and demerits of them are analyzed and compared. Finally relevant conclusions and development trends are given.

Kalman Filter and Its Application

Numerous efforts have been made to design various low-level saliency cues for RGBD saliency detection, such as color and depth contrast features as well as background and color compactness priors. However, how these low-level saliency cues interact with each other and how they can be effectively incorporated to generate a master saliency map remain challenging problems. In this paper, we design a new convolutional neural network (CNN) to automatically learn the interaction mechanism for RGBD salient object detection. In contrast to existing works, in which raw image pixels are fed directly to the CNN, the proposed method takes advantage of the knowledge obtained in traditional saliency detection by adopting various flexible and interpretable saliency feature vectors as inputs. This guides the CNN to learn a combination of existing features to predict saliency more effectively, which presents a less complex problem than operating on the pixels directly. We then integrate a superpixel-based Laplacian propagation framework with the trained CNN to extract a spatially consistent saliency map by exploiting the intrinsic structure of the input image. Extensive quantitative and qualitative experimental evaluations on three data sets demonstrate that the proposed method consistently outperforms the state-of-the-art methods.

RGBD Salient Object Detection via Deep Fusion

Recent work in salient object detection has considered the incorporation of depth cues from RGB-D images. In most cases, depth contrast is used as the main feature. However, areas of high contrast in background regions cause false positives for such methods, as the background frequently contains regions that are highly variable in depth. Here, we propose a novel RGB-D saliency feature. Local Background Enclosure (LBE) captures the spread of angular directions which are background with respect to the candidate region and the object that it is part of. We show that our feature improves over state-of-the-art RGB-D saliency approaches as well as RGB methods on the RGBD1000 and NJUDS2000 datasets.

/pdf/local-background-enclosure-for-rgb-d-salient-object-2aos10w4y3.pdf

Local Background Enclosure for RGB-D Salient Object Detection

Perceptual quality assessment is a challenging issue in 3D signal processing research. It is important to study 3D signal directly instead of studying simple extension of the 2D metrics directly to the 3D case as in some previous studies. In this paper, we propose a new perceptual full-reference quality assessment metric of stereoscopic images by considering the binocular visual characteristics. The major technical contribution of this paper is that the binocular perception and combination properties are considered in quality assessment. To be more specific, we first perform left-right consistency checks and compare matching error between the corresponding pixels in binocular disparity calculation, and classify the stereoscopic images into non-corresponding, binocular fusion, and binocular suppression regions. Also, local phase and local amplitude maps are extracted from the original and distorted stereoscopic images as features in quality assessment. Then, each region is evaluated independently by considering its binocular perception property, and all evaluation results are integrated into an overall score. Besides, a binocular just noticeable difference model is used to reflect the visual sensitivity for the binocular fusion and suppression regions. Experimental results show that compared with the relevant existing metrics, the proposed metric can achieve higher consistency with subjective assessment of stereoscopic images.

Perceptual Full-Reference Quality Assessment of Stereoscopic Images by Considering Binocular Visual Characteristics

Compared with traditional mono-view video, three-dimensional video (3DV) provides user interactive functionalities and stereoscopic perception, which makes people more interested in pop-out regions or the regions with small depth value. Thus, traditional visual attention model for mono-view video can hardly be directly applied to stereoscopic visual attention (SVA) analysis for 3DV. In this paper, we propose a bottom-up SVA model to simulate human visual system with stereoscopic vision more accurately. The proposed model is based on multiple perceptual stimuli including depth information, luminance, color, orientation and motion contrast. Then, a depth based dynamic fusion is proposed to integrate these features. The experimental results on multi-view video test sequences show that the proposed model maintains high robustness and is able to efficiently simulate SVA of human eyes.

Stereoscopic visual attention model for 3d video

The recent years have witnessed three-dimensional (3-D) video technology to become increasingly popular, as it can provide high-quality and immersive experience to end users, where view rendering with depth-image-based rendering (DIBR) technique is employed to generate the virtual views. Distortions in depth map may induce geometry changes in the virtual views, and distortions in texture video may be propagated to the virtual views. Thus, effective compression of both texture videos and depth maps is important for 3-D video system. From the perspective of bit allocation, asymmetric coding of the texture videos and depth maps is an effective way to get the optimal solution of 3-D video compression and view rendering problems. In this paper, a novel asymmetric coding method of multi-view video plus depth (MVD) based 3-D video is proposed on purpose of providing high-quality view rendering. In the proposed method, two models are proposed to characterize view rendering distortion and binocular suppression in 3-D video. Then, an asymmetric coding method of MVD-based 3-D video is proposed by combining two models in encoding framework. Finally, a chrominance reconstruction algorithm is presented to achieve accurate reconstruction. Experimental results show that compared with other methods, the proposed method can obtain higher performance of view rendering under the total bitrate constraint. Moreover, the perceptual visual quality of 3-D video is almost unaffected with the proposed method.

Asymmetric Coding of Multi-View Video Plus Depth Based 3-D Video for View Rendering

Three-dimensional (3D) video technology is becoming increasingly popular, as it can provide stereoscopic perception and immersive experience to end users. Most three-dimensional video compression schemes had been developed for stereoscopic video using traditional two-dimensional video compression techniques. However, stereoscopic perception had not been considered in these methods to further improve the coding efficiency. In this paper, a novel stereoscopic video coding method is proposed with asymmetric luminance and chrominance qualities based on the suppression theory of binocular vision. We base our method on two hypotheses: a quantitative just noticeable distortion (JND) threshold exists between left and right views, and chrominance degradation in right view will not affect the stereoscopic effect. Then, two subjective perception experiments are conducted to verify the above hypotheses. Finally, an asymmetric stereoscopic video coding method is engineered deriving from the hypotheses. Experimental results show that the proposed method saves bitrate significantly by from 15% to 35%, and can achieve superior reconstruction quality at trivial degradation cost.

Stereoscopic video coding with asymmetric luminance and chrominance qualities

We propose a Stereoscopic Visual Attention- (SVA-) based regional bit allocation optimization for Multiview Video Coding (MVC) by the exploiting visual redundancies from human perceptions. We propose a novel SVA model, where multiple perceptual stimuli including depth, motion, intensity, color, and orientation contrast are utilized, to simulate the visual attention mechanisms of human visual system with stereoscopic perception. Then, a semantic region-of-interest (ROI) is extracted based on the saliency maps of SVA. Both objective and subjective evaluations of extracted ROIs indicated that the proposed SVA model based on ROI extraction scheme outperforms the schemes only using spatial or/and temporal visual attention clues. Finally, by using the extracted SVA-based ROIs, a regional bit allocation optimization scheme is presented to allocate more bits on SVA-based ROIs for high image quality and fewer bits on background regions for efficient compression purpose. Experimental results on MVC show that the proposed regional bit allocation algorithm can achieve over 20 ∼ 30% bit-rate saving while maintaining the subjective image quality. Meanwhile, the image quality of ROIs is improved by 0.46 ∼ 0.61 dB at the cost of insensitive image quality degradation of the background image.

/pdf/stereoscopic-visual-attention-based-regional-bit-allocation-2flsp5pvsb.pdf

Stereoscopic visual attention-based regional bit allocation optimization for multiview video coding

A content-adaptive color correction algorithm for multi-view video is proposed due to variation in lighting or camera parameters. We first establish color correction property between the target image and source image. Then color correction matrix can be obtained by global correction or preferred region matching correction. Finally, video tracking technique is used to correct multi-view video sequences. Experimental results show the proposed algorithm has better correction effect for different multi-view video sequences.

Ken Chen

Papers

Stereoscopic visual attention model for 3d video

Asymmetric Coding of Multi-View Video Plus Depth Based 3-D Video for View Rendering

Stereoscopic video coding with asymmetric luminance and chrominance qualities

Stereoscopic visual attention-based regional bit allocation optimization for multiview video coding

A Content-Adaptive Multi-View Video Color Correction Algorithm