With the fast development of visual noise-shaping related applications (visual compression, error resilience, watermarking, encryption, and display), there is an increasingly significant demand on incorporating perceptual characteristics into these applications for improved performance. In this paper, a very important mechanism of the human brain, visual attention, is introduced for visual sensitivity and visual quality evaluation. Based upon the analysis, a new numerical measure for visual attention's modulatory aftereffects, perceptual quality significance map (PQSM), is proposed. To a certain extent, the PQSM reflects the processing ability of the human brain on local visual contents statistically. The PQSM is generated with the integration of local perceptual stimuli from color contrast, texture contrast, motion, as well as cognitive features (skin color and face in this study). Experimental results with subjective viewing demonstrate the performance improvement on two PQSM-modulated visual sensitivity models and two PQSM-based visual quality metrics.

Modeling visual attention's modulatory aftereffects on visual sensitivity and quality evaluation

We propose a novel algorithm to detect visual saliency from video signals by combining both spatial and temporal information and statistical uncertainty measures. The main novelty of the proposed method is twofold. First, separate spatial and temporal saliency maps are generated, where the computation of temporal saliency incorporates a recent psychological study of human visual speed perception. Second, the spatial and temporal saliency maps are merged into one using a spatiotemporally adaptive entropy-based uncertainty weighting approach. The spatial uncertainty weighing incorporates the characteristics of proximity and continuity of spatial saliency, while the temporal uncertainty weighting takes into account the variations of background motion and local contrast. Experimental results show that the proposed spatiotemporal uncertainty weighting algorithm significantly outperforms state-of-the-art video saliency detection models.

/pdf/video-saliency-incorporating-spatiotemporal-cues-and-3coi9s0tfy.pdf

Video Saliency Incorporating Spatiotemporal Cues and Uncertainty Weighting

Three-dimensional (2D+T) wavelet coding of video using SPIHT has been shown to outperform standard predictive video coders on complex high-motion sequences, and is competitive with standard predictive video coders on simple low-motion sequences. However, on a number of typical moderate-motion sequences characterized by largely rigid motions, 3D SPIHT performs several dB worse than motion-compensated predictive coders, because it is does not take advantage of the real physical motion underlying the scene. We introduce global motion compensation for 3D subband video coders, and find 0.5 to 2 dB gain on sequences with dominant background motion. Our approach is a hybrid of video coding based on sprites, or mosaics, and subband coding.

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/WangXCM99.pdf

Three-dimensional wavelet coding of video with global motion compensation

We describe a novel approach for image sequence segmentation. It contains three parts: global motion compensation, robust frame differencing, and curve evolution. It is computationally efficient, does not require dense-field motion estimation, and is insensitive to noise and global/background motion. It works for black-and-white and color image sequences. The efficacy of this approach is demonstrated on both TV and surveillance image sequences.

Image sequence segmentation using 3-D structure tensor and curve evolution

A full multimedia production such as a seminar, conference, lecture, etc. can be captured in real time using multiple cameras (11, 13). A live movie of a speaker together with the speaker's flipping still images or slide show can be viewed interactively within the same video display screen (31-46). The complete production can be stored on a hard drive (17) for retrieval on demand, or sent live to a host server for live distribution throughout a data network. It is also possible to store the complete presentation on portable storage media, and/or to send it as an e-mail.

Automated conversion of a visual presentation into digital data format

In the H.263 Version 2 (H.263+) coding standard, the global motion compensation can be introduced by using Reference Picture Resampling (Annex. P) syntax. Such an application requires that the global motion parameters be estimated automatically. We propose a global motion estimation algorithm based on the Taylor expansion equation and robust regression technique using probabilistic thresholding. The experimental results confirm that the proposed algorithm can improve both the coding efficiency and the quality of motion compensation on sequences involving camera movement.

J. Kittler

Papers

Global motion estimation and robust regression for video coding