Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.

/pdf/image-quality-assessment-from-error-visibility-to-structural-1rlwcqe34t.pdf

Image quality assessment: from error visibility to structural similarity

Introduction to a Transient World. Fourier Kingdom. Discrete Revolution. Time Meets Frequency. Frames. Wavelet Zoom. Wavelet Bases. Wavelet Packet and Local Cosine Bases. An Approximation Tour. Estimations are Approximations. Transform Coding. Appendix A: Mathematical Complements. Appendix B: Software Toolboxes.

http://ahvazdownload.persiangig.com/document/article/39.pdf

A wavelet tour of signal processing

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Moreover, since the release of the pix2pix software associated with this paper, hundreds of twitter users have posted their own artistic experiments using our system. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without handengineering our loss functions either.

/pdf/image-to-image-translation-with-conditional-adversarial-5058t24et2.pdf

Image-to-Image Translation with Conditional Adversarial Networks

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.

Deposits of clastic carbonate-dominated (calciclastic) sedimentary slope systems in the rock record have been identified mostly as linearly-consistent carbonate apron deposits, even though most ancient clastic carbonate slope deposits fit the submarine fan systems better. Calciclastic submarine fans are consequently rarely described and are poorly understood. Subsequently, very little is known especially in mud-dominated calciclastic submarine fan systems. Presented in this study are a sedimentological core and petrographic characterisation of samples from eleven boreholes from the Lower Carboniferous of Bowland Basin (Northwest England) that reveals a >250 m thick calciturbidite complex deposited in a calciclastic submarine fan setting. Seven facies are recognised from core and thin section characterisation and are grouped into three carbonate turbidite sequences. They include: 1) Calciturbidites, comprising mostly of highto low-density, wavy-laminated bioclast-rich facies; 2) low-density densite mudstones which are characterised by planar laminated and unlaminated muddominated facies; and 3) Calcidebrites which are muddy or hyper-concentrated debrisflow deposits occurring as poorly-sorted, chaotic, mud-supported floatstones. These

Phd by thesis

Three-dimensional microscopy

We propose a new prototype model for no-reference video quality assessment (VQA) based on the natural statistics of space-time chips of videos. Space-time chips (ST-chips) are a new, quality-aware feature space which we define as space-time localized cuts of video data in directions that are determined by the local motion flow. We use parametrized distribution fits to the bandpass histograms of space-time chips to characterize quality, and show that the parameters from these models are affected by distortion and can hence be used to objectively predict the quality of videos. Our prototype method, which we call ChipQA-0, is agnostic to the types of distortion affecting the video, and is based on identifying and quantifying deviations from the expected statistics of natural, undistorted ST-chips in order to predict video quality. We train and test our resulting model on several large VQA databases and show that our model achieves high correlation against human judgments of video quality and is competitive with state-of-the-art models.

/pdf/no-reference-video-quality-assessment-using-space-time-chips-1hujfqn4tv.pdf

No-Reference Video Quality Assessment Using Space-Time Chips

The mere presence of spatiotemporal distortions in digital videos does not have to imply quality degradation since distortion visibility can be strongly reduced by the perceptual phenomenon of visual masking. Flicker is a particularly annoying occurrence, which can arise from a variety of distortion processes. Yet flicker can also be suppressed by masking. We propose a perceptual flicker visibility prediction model which is based on a recently discovered visual change silencing phenomenon. The proposed model predicts flicker visibility on both static and moving regions without any need for content-dependent thresholds. Using a simple model of cortical responses to video flicker, an energy model of motion perception, and a divisive normalization stage, the system captures the local spectral signatures of flicker distortions and predicts perceptual flicker visibility. The model not only predicts silenced flicker distortions in the presence of motion, but also provides a pixel-wise flicker visibility index. Results show that the predicted flicker visibility model correlates well with human percepts of flicker distortions tested on the LIVE Flicker Video Database and is highly competitive with current flicker visibility prediction methods. Introduction Digital videos are increasingly pervasive due to the rapid proliferation of video streaming services, video sharing in social networks, and the global increase of mobile video traffic [1], [2]. The dramatic growth of digital videos and user demand for highquality video have necessitated the development of precise automatic perceptual video quality assessment (VQA) tools to help provide satisfactory levels of Quality of Experience (QoE) to the end user [3]. To achieve optimal video quality under limited bandwidth and power consumption, video coding technologies commonly employ lossy coding schemes, which cause compression artifacts that can lead to degradation of perceptual video quality [4]. In addition, compressed videos can suffer from transmission distortions, including packet losses and playback interruptions triggered by channel throughput fluctuations. Since humans are generally the ultimate arbiter of the received videos, predicting and reducing perceptual visual distortions of compressed digital videos is of great interest [5]. Researchers have performed a large number of subjective studies to understand essential factors that influence video quality by analyzing compression artifacts or transmission distortions of the compressed videos [6], by investigating dynamic time varying distortions [7], and by probing the time varying subjective quality of rate adaptive videos [8]. Substantial progress has also been made toward understanding and modeling low-level visual processes in the vision system extending from the retina to primary visual cortex and extra-striate cortex [9]. As a result, perceptual models of disruptions to natural scene statistics [10] and of visual masking [11] have been widely applied to predict perceptual visual quality. Spatial distortions are effectively predicted by VQA algorithms such as SSIM [12], VQM [13], MOVIE [14], STRRED [15], and Video-BLIINDS [16]. Spatial masking is well-modeled in modern perceptual image and video quality assessment tools, video compression, and watermarking. However, temporal visual masking is not well-modeled although one type of it has been observed to occur near scene changes [17], and been used in the context of early video compression methods [18-20]. Among temporal distortions, flicker distortion is particularly challenging to predict and often occurs on low bit-rate compressed videos. Flicker distortion is (spatially local or global) temporal fluctuation of luminance or chrominance in videos. Local flicker occurs mainly due to coarse quantization, varying prediction modes, mismatching of inter-frame blocks, improper deinterlacing, and dynamic rate changes caused by adaptive rate control methods [21-25]. Mosquito noise and stationary area fluctuations are also often categorized under local flicker. Mosquito noise is a joint effect of object motion and time-varying spatial artifacts such as ringing and motion prediction errors near high-contrast sharp edges or moving objects, while stationary area fluctuations result from different types of prediction, quantization levels, or a combination of these factors on static regions [4], [21]. Current flicker visibility prediction methods that operate on a compressed video measure the Sum of Squared Differences (SSD) between the block difference of an original video and the block difference of a compressed video. The block difference is obtained between successive frames on macroblocks. When the sum of squared block differences on an original video falls below a threshold, a static region is indicated [22]. The ratio between luminance level fluctuation in the compressed video and that in the original video has also been used [23]. To improve the prediction of flicker-prone blocks, a normalized fraction model was proposed [24], where the difference of SSDs between the original and compressed block differences is divided by the sum of the SSDs. These methods have the virtue of simplicity, but the resulting flicker prediction performance is limited and content-dependent. Another method included the influence of motion on flicker prediction, where motion compensation was applied prior to SSD calculation [25]. The mean absolute discrete temporal derivatives of the average DC coefficient of DCT blocks was used to measure sudden local changes (flicker) in a VQA model [16]. Current flicker prediction methods are limited to block-wise accuracy. Further, human visual system (HVS)-based perceptual flicker visibility e.g., considering temporal visual masking, has not yet been extensively studied. Recently, Suchow and Alvarez [26] demonstrated a striking “motion silencing” illusion, in the form of a powerful temporal visual masking phenomenon called change silencing, where the salient temporal changes of objects in luminance, color, size, and shape appear to cease in the presence of large object motions. This motion-induced failure to detect change not only suggests a tight coupling between motion and object appearance, but also reveals that commonly occurring temporal distortions such as flicker may be dramatically suppressed by the presence of motion. To understand the mechanism of motion silencing, physiologically plausible explanations have been proposed [26-29]. However, since the effect has only been studied on highly synthetic stimuli such as moving dots, we performed a series of human subjective studies on naturalistic videos, where flicker visibility is observed to be strongly reduced by large coherent object motions [30-33]. A consistent physiological and computational model that detects motion silencing might be useful to probe perceptual flicker visibility on compressed videos. In this paper, we propose a new perceptual flicker visibility prediction model based on motion silencing. The new perceptual flicker visibility prediction model is a significant step towards improving the performance of VQA models by making possible a model of temporal masking of temporal distortions. The new model measures the bandpass filter responses to a reference video and a corresponding flicker video using a localized multiscale 3D space time Gabor filter bank [34], [35], a spatiotemporal energy model of motion perception [36], and a divisive normalization model of nonlinear gain control in primary visual cortex [37]. We observed that flicker produces locally separated spectral signatures that almost lie along the same orientation as the motion tuned plane of the reference video but at a distance. The captured V1 responses for the flicker induced spectral signatures generally decreased when object speeds increase. Next, we measured the local difference of bandpass responses at each space-time frequency orientation and defined the sum of the magnitude responses as a perceptual flicker visibility index. The proposed model predicts temporal masking effects on flicker distortions and thereby shows highly competitive performance against previous flicker visibility prediction methods. Background: Motion Perception Motion perception is the process of inferring the speed and direction of moving objects. Since motion perception is important for understanding flicker distortions in videos, we model motion perception in the frequency domain. Watson and Ahumada [38] proposed a model of how humans sense the velocity of moving images, where the motion-sensing elements appear locally tuned to specific spatiotemporal frequencies. Assuming that complex motions of video without any scene changes can be constructed by piecing together spatiotemporally localized image patches undergoing translation, we can model the local spectral signatures of videos when an image patch moves [38]. An arbitrary space-time image patch can be represented by a function a(x, y, t) at each point x, y, and time t, and its Fourier transform by A(u, v, w) where u, v, and w are spatial and temporal frequency variables corresponding to x, y and t, respectively. Let λ and φ denote the image patch horizontal and vertical velocity components. When an image patch translates at constant velocity [λ, φ], the moving video sequence becomes b(x, y, t) = a(x – λt, y – φt, t). The spectrum of a stationary image patch lies on the u, v plane, while the Fourier transform shears into an oblique plane through the origin when the image patch moves. The orientation of this plane indicates the speed and direction of motion. Prediction of Perceptual Flicker Visibility Linear Decomposition Natural environments are inherently multi-scale and multiorientation, and objects move multi-directionally at diverse speeds. To efficiently encode visual signals, the vision system decomposes (a) (b) (c) Figure 1. Gabor filter bank in the frequency domain. (a) G

/pdf/perceptual-flicker-visibility-prediction-model-j4b5p4r9tz.pdf

Perceptual Flicker Visibility Prediction Model.

We consider the problem of estimating a dense depth map from a single monocular image. Inspired by psychophysical evidence of visual processing in human vision systems (HVS) and natural scene statistics (NSS) models of image and range, we propose a Bayesian framework to recover detailed 3D scene structure by exploiting the statistical relationships between local image features and depth variations inherent in natural images. By observing that similar depth structures may exist in different types of luminance/chrominance textured regions in natural scenes, we build a dictionary of canonical range patterns as the prior, and fit a multivariate Gaussian mixture (MGM) model to associate local image features to different range patterns as the likelihood. Compared with the state-of-the-art depth estimation method, we achieve similar performance in terms of pixel-wise estimated range error, but superior capability of recovering relative distant relationships between different parts of the image.

Depth estimation from monocular color images using natural scene statistics models

C.G. Boncelet's algorithm (SIAM J. Sci. Stat. Comput., vol.8, p.868-76, Sept. 1987) is used to explore the OS filter design/analysis problem. In particular, the optimal filter for restoring nonrandom signals immersed in Markov noise, using the mean square error as an optimality criterion, is studied. The noise processes are modeled either as causal first-order autoregressive Gaussian or as first-order moving-average Gaussian. Various structural signal constraints are improved on the solution by stating them as local unbiasedness constraints. >

Alan C. Bovik

Papers

Three-dimensional microscopy

No-Reference Video Quality Assessment Using Space-Time Chips

Perceptual Flicker Visibility Prediction Model.

Depth estimation from monocular color images using natural scene statistics models

Least-squares order statistic filters for signal restoration in dependent noise