Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.

/pdf/image-quality-assessment-from-error-visibility-to-structural-1rlwcqe34t.pdf

Image quality assessment: from error visibility to structural similarity

Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.

/pdf/batch-normalization-accelerating-deep-network-training-by-3dkqye6twt.pdf

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Multiresolution representations are effective for analyzing the information content of images. The properties of the operator which approximates a signal at a given resolution were studied. It is shown that the difference of information between the approximation of a signal at the resolutions 2/sup j+1/ and 2/sup j/ (where j is an integer) can be extracted by decomposing this signal on a wavelet orthonormal basis of L/sup 2/(R/sup n/), the vector space of measurable, square-integrable n-dimensional functions. In L/sup 2/(R), a wavelet orthonormal basis is a family of functions which is built by dilating and translating a unique function psi (x). This decomposition defines an orthogonal multiresolution representation called a wavelet representation. It is computed with a pyramidal algorithm based on convolutions with quadrature mirror filters. Wavelet representation lies between the spatial and Fourier domains. For images, the wavelet representation differentiates several spatial orientations. The application of this representation to data compression in image coding, texture discrimination and fractal analysis is discussed. >

/pdf/a-theory-for-multiresolution-signal-decomposition-the-530tw5pzq7.pdf

A theory for multiresolution signal decomposition: the wavelet representation

Suppose x is an unknown vector in Ropfm (a digital image or signal); we plan to measure n general linear functionals of x and then reconstruct. If x is known to be compressible by transform coding with a known transform, and we reconstruct via the nonlinear procedure defined here, the number of measurements n can be dramatically smaller than the size m. Thus, certain natural classes of images with m pixels need only n=O(m1/4log5/2(m)) nonadaptive nonpixel samples for faithful recovery, as opposed to the usual m pixel samples. More specifically, suppose x has a sparse representation in some orthonormal basis (e.g., wavelet, Fourier) or tight frame (e.g., curvelet, Gabor)-so the coefficients belong to an lscrp ball for 0<ples1. The N most important coefficients in that expansion allow reconstruction with lscr2 error O(N1/2-1p/). It is possible to design n=O(Nlog(m)) nonadaptive measurements allowing reconstruction with accuracy comparable to that attainable with direct knowledge of the N most important coefficients. Moreover, a good approximation to those N important coefficients is extracted from the n measurements by solving a linear program-Basis Pursuit in signal processing. The nonadaptive measurements have the character of "random" linear combinations of basis/frame elements. Our results use the notions of optimal recovery, of n-widths, and information-based complexity. We estimate the Gel'fand n-widths of lscrp balls in high-dimensional Euclidean space in the case 0<ples1, and give a criterion identifying near- optimal subspaces for Gel'fand n-widths. We show that "most" subspaces are near-optimal, and show that convex optimization (Basis Pursuit) is a near-optimal way to extract information derived from these near-optimal subspaces

Compressed sensing

Many multi-dimensional signal processing problems require the computation of signal gradients or directional derivatives. Traditional derivative estimates based on adjacent or central differences are often inappropriate for multi-dimensional problems. As replacements for these traditional operators, the author designs a set of matched pairs of derivative filters and lowpass prefilters. The author demonstrates the superiority of these filters over simple difference operators. >

Design of multi-dimensional derivative filters

The central theme of the thesis is that the failure of image motion algorithms is due primarily to the use of vector fields as a representation for visual motion We argue that the translational vector field representation is inherently impoverished and error-prone Furthermore, there is evidence that a direct optical flow representation scheme is not used by biological systems for motion analysis Instead, we advocate distributed representations of motion, in which the encoding of image plane velocity is implicit
As a simple example of this idea, and in consideration of the errors in the flow vectors, we re-cast the traditional optical flow problem as a probabilistic one, modeling the measurement and constraint errors as random variables The resulting framework produces probability distributions of optical flow, allowing proper handling of the uncertainties inherent in the optical flow computation, and facilitating the combination with information from other sources We demonstrate the advantages of this probabilistic approach on a set of examples In order to overcome the temporal aliasing commonly found in time-sampled imagery (eg, video), we develop a probabilistic "coarse-to-fine" algorithm that functions much like a Kalman filter over scale We implement an efficient version of this algorithm and show its success in computing Gaussian distributions of optical flow of both synthetic and real image sequences
We then extend the notion of distributed representation to a generalized framework that is capable of representing multiple motions at a point We develop an example representation through a series of modifications of the differential approach to optical flow estimation We show that this example is capable of representing multiple motions at a single image location and we demonstrate its use near occlusion boundaries and on simple synthetic examples containing transparent objects
Finally, we show that these distributed representation are effective as models for biological motion representation We show qualitative comparisons of stages of the algorithm with neurons found in mammalian visual systems, suggesting experiments to test the validity of the model We demonstrate that such a model can account quantitatively for a set of psychophysical data on the perception of moving sinusoidal plaid patterns (Copies available exclusively from MIT Libraries, Rm 14-0551, Cambridge, MA 02139-4307 Ph 617-253-5668; Fax 617-253-1690) (Abstract shortened by UMI)

Distributed representation and analysis of visual motion

We present a method for the efficient re-rendering of a scene under a directional illuminant at an arbitrary orientation. We take advantage of the linearity of the rendering operator with respect to illumination for a fixed scene and camera geometry. Re-rendering is accomplished via linear combination of a set of pre-rendered “basis” images. The theory of steerable functions provides the machinery to derive an appropriate set of basis images. We demonstrate the technique on both simple and complex scenes illuminated by an approximation to natural skylight. We show re-rendering simulations under conditions of varying sun position and cloudiness.

/pdf/efficient-re-rendering-of-naturally-illuminated-environments-5f3tt1if3t.pdf

Efficient Re-rendering of Naturally Illuminated Environments

We describe an information-theoretic framework for fitting neural spike responses with a Linear-Nonlinear-Poisson cascade model. This framework unifies the spike-triggered average and spike-triggered covariance approaches to neural characterization, and recovers a set of linear filters that maximize mean and variance-dependent information between stimuli and spike responses. The resulting approach has several useful properties: (1) it recovers a set of linear filters sorted according to their informativeness about the neural response; (2) it is both computationally efficient and robust, allowing recovery of multiple linear filters from a data set of relatively modest size; (3) it provides an explicit “default” model of the nonlinear stage mapping the filter responses to spike rate, in the form of a ratio of Gaussians. (4) it is equivalent to maximum likelihood estimation of this default model, but also converges to the correct filter estimates whenever the conditions for the consistency of spike-triggered average or covariance analysis are met; (5) it can be augmented with additional constraints, such as space-time separability, on the filters. We demonstrate the effectiveness of the method by applying it to simulated responses of a Hodgkin-Huxley neuron, and the recorded extracellular responses of macaque retinal ganglion cells and V1 cells.

/pdf/dimensionality-reduction-in-neural-models-an-information-3zyhstcrnd.pdf

Dimensionality reduction in neural models: an information-theoretic generalization of spike-triggered average and covariance analysis.

Generalized non-separable extensions of quadrature mirror filter (QMF) banks to two and three dimensions, in which the orientation specificity of the high-pass filters is greatly improved, are described. In particular, extensions to two dimensions with hexagonal symmetry, and 3-D spatiotemporal extensions with rhombic-dodecahedral symmetry, are discussed. Although these filters are conceived and designed on nonstandard sampling lattices, they can be applied to rectangularly sampled images. As in one dimension, these transformations can be hierarchically cascaded to form a multiscale pyramid representation. A set of example filters is designed and applied to the problems of image compression, progressive transmission, orientation analysis, and motion analysis. >

/pdf/non-separable-extensions-of-quadrature-mirror-filters-to-2ysd1emf1g.pdf

Eero P. Simoncelli

Papers

Design of multi-dimensional derivative filters

Distributed representation and analysis of visual motion

Efficient Re-rendering of Naturally Illuminated Environments

Dimensionality reduction in neural models: an information-theoretic generalization of spike-triggered average and covariance analysis.

Non-separable extensions of quadrature mirror filters to multiple dimensions