Finding ways to monitor and control the perceptual quality of digital visual media has become a pressing concern as the volume being transported and viewed continues to increase exponentially. This paper discusses the principles and methods of modern algorithms for automatically predicting the quality of visual signals. By casting the problem as analogous to assessing the efficacy of a visual communication system, it is possible to divide the quality assessment problem into understandable modeling subproblems. Along the way, we will visit models of natural images and videos, of visual perception, and a broad spectrum of applications.

/pdf/automatic-prediction-of-perceptual-image-and-video-quality-1276xl9y2c.pdf

Automatic Prediction of Perceptual Image and Video Quality

With the fast development of visual noise-shaping related applications (visual compression, error resilience, watermarking, encryption, and display), there is an increasingly significant demand on incorporating perceptual characteristics into these applications for improved performance. In this paper, a very important mechanism of the human brain, visual attention, is introduced for visual sensitivity and visual quality evaluation. Based upon the analysis, a new numerical measure for visual attention's modulatory aftereffects, perceptual quality significance map (PQSM), is proposed. To a certain extent, the PQSM reflects the processing ability of the human brain on local visual contents statistically. The PQSM is generated with the integration of local perceptual stimuli from color contrast, texture contrast, motion, as well as cognitive features (skin color and face in this study). Experimental results with subjective viewing demonstrate the performance improvement on two PQSM-modulated visual sensitivity models and two PQSM-based visual quality metrics.

Modeling visual attention's modulatory aftereffects on visual sensitivity and quality evaluation

In this paper, a novel approach to the impulsive noise removal in color images is presented. The proposed technique employs the switching scheme based on the impulse detection mechanism using the so-called peer group concept. Compared to the vector median filter and other commonly used multichannel filters, the proposed technique consistently yields very good results in suppressing both the random and fixed-valued impulsive noise. The main advantage of the proposed noise detection framework is its enormous computational speed, which enables efficient filtering of color images in real-time applications.

Fast detection and impulsive noise removal in color images

We consider the problem of predicting packet loss visibility in MPEG-2 video. We use two modeling approaches: CART and GLM. The former classifies each packet loss as visible or not; the latter predicts the probability that a packet loss is visible. For each modeling approach, we develop three methods, which differ in the amount of information available to them. A reduced reference method has access to limited information based on the video at the encoder's side and has access to the video at the decoder's side. A no-reference pixel-based method has access to the video at the decoder's side but lacks access to information at the encoder's side. A no-reference bitstream-based method does not have access to the decoded video either; it has access only to the compressed video bitstream, potentially affected by packet losses. We design our models using the results of a subjective test based on 1080 packet losses in 72 minutes of video.

/pdf/modeling-packet-loss-visibility-in-mpeg-2-video-41xh4feqq7.pdf

Modeling packet-loss visibility in MPEG-2 video

This paper reviews the basic background knowledge necessary to design effective no-reference (NR) quality estimators (QEs) for images and video. We describe a three-stage framework for NR QE that encompasses the range of potential use scenarios for the NR QE and allows knowledge of the human visual system to be incorporated throughout. We survey the measurement stage of the framework, considering methods that rely on bitstream, pixels, or both. By exploring both the accuracy requirements of potential uses as well as evaluation criteria to stress-test a QE, we set the stage for our community to make substantial future improvements to the challenging problem of NR quality estimation.

No-reference image and video quality estimation: Applications and human-motivated design

A computationally efficient algorithm is proposed to remove noise impulses from speech and audio signals while retaining its features and tonal quality. The proposed method is based on the SD-ROM (Signal Dependent Rank Order Mean) algorithm. This technique has successfully been used to remove impulse noise from images. It has the advantage of being relatively fast, simple and robust. The algorithm estimates the likelihood the sample under inspection is corrupt relative to the neighboring samples and replaces a sample detected as corrupted by a value based on the neighboring samples. This algorithm also has the advantage of being 'configurable' to the type of noise impulses in the sample, as the thresholds used to detect noise impulses can be varied to suit the signal.

/pdf/an-efficient-method-for-the-removal-of-impulse-noise-from-wottysh8hp.pdf

An efficient method for the removal of impulse noise from speech and audio signals

Vector SD-ROM Filter for Removal of Impulse Noise from Colour Images

Many approaches have been proposed recently for estimating the perceived fidelity of digitally compressed and reconstructed video clips. Several metrics rely on the accurate estimation of detection thresholds, basically assuming that perceived quality depends significantly on the error threshold. To test that assumption, we designed an experiment to measure the actual detection threshold of some MPEG-2 artifacts in typical video sequences. In each of the test clips, we briefly replaced a region with a test stimulus created using an MPEG-2 encoder. The location, time, and strength of the artifact were varied between test videos. At the end of each clip, each subject provided a detection response and, for the detected artifacts, location and annoyance information. From the data, we determined the detection threshold for each artifact. Using the thresholds, we computed a perceptually weighted error metric. In this paper, we describe the experiment in more detail, summarize the experimental results including the threshold calculations, and compare the weighted error measure to the output of a commercial fidelity metric. Finally, we discuss our conclusions on the validity of predicting quality from threshold measurements.

/pdf/detectability-and-annoyance-value-of-mpeg-2-artifacts-3e7hzobkx6.pdf

Detectability and annoyance value of MPEG-2 artifacts inserted in uncompressed video sequences

In this work, we used two types of impairments in a psychophysical experiment to measure the overall annoyance and individual strength of three impairment features (fuzzy, blocky, and blurry). The impairments were generated by compressing the original videos with MPEG-2 at two different bitrates: 1.0 and 7.5 Mbps. The heavily compressed videos presented blurry and blocky impairments, while the lightly compressed videos presented 'fuzzy' impairments, using a word provided by our test subjects. These impairments were then linearly combined in different proportions and strengths, generating videos in which all three impairment features are present. Our goal was to determine how these impairment features combine to produce the overall annoyance. A modified Minkowski metric was used to describe the 'combination rule' which relates the strengths of the impairment features to the overall annoyance. For the data set containing all test sequences, the optimal value found for the Minkowski parameter p was 1.55. From the data obtained, we also estimated the psychometric and annoyance functions. We found that for blocky-blurry and fuzzy artifacts there is no consistent difference between either the thresholds or mid-annoyance strengths.

Perceptual contributions of blocky, blurry, and fuzzy impairments to overall annoyance

In this paper, we describe the results of a series of psychophysical experiments that investigated the relationships among defect visibility, content importance, and perceived impairment in digital video. Various types of controlled defects were inserted into normal video and shown to our test subjects. The defects varied in their strength, location, appearance, size, and duration. We measured three specific subjective quantities: the defect detection probabilities, the perceived impairment, and the content importance. Impairment was found to be tightly related to defect visibility but only weakly related to content importance.

M.S. Moore

Papers

An efficient method for the removal of impulse noise from speech and audio signals

Vector SD-ROM Filter for Removal of Impulse Noise from Colour Images

Detectability and annoyance value of MPEG-2 artifacts inserted in uncompressed video sequences

Perceptual contributions of blocky, blurry, and fuzzy impairments to overall annoyance

Defect visibility and content importance: effects on perceived impairment