Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

Suppose x is an unknown vector in Ropfm (a digital image or signal); we plan to measure n general linear functionals of x and then reconstruct. If x is known to be compressible by transform coding with a known transform, and we reconstruct via the nonlinear procedure defined here, the number of measurements n can be dramatically smaller than the size m. Thus, certain natural classes of images with m pixels need only n=O(m1/4log5/2(m)) nonadaptive nonpixel samples for faithful recovery, as opposed to the usual m pixel samples. More specifically, suppose x has a sparse representation in some orthonormal basis (e.g., wavelet, Fourier) or tight frame (e.g., curvelet, Gabor)-so the coefficients belong to an lscrp ball for 0<ples1. The N most important coefficients in that expansion allow reconstruction with lscr2 error O(N1/2-1p/). It is possible to design n=O(Nlog(m)) nonadaptive measurements allowing reconstruction with accuracy comparable to that attainable with direct knowledge of the N most important coefficients. Moreover, a good approximation to those N important coefficients is extracted from the n measurements by solving a linear program-Basis Pursuit in signal processing. The nonadaptive measurements have the character of "random" linear combinations of basis/frame elements. Our results use the notions of optimal recovery, of n-widths, and information-based complexity. We estimate the Gel'fand n-widths of lscrp balls in high-dimensional Euclidean space in the case 0<ples1, and give a criterion identifying near- optimal subspaces for Gel'fand n-widths. We show that "most" subspaces are near-optimal, and show that convex optimization (Basis Pursuit) is a near-optimal way to extract information derived from these near-optimal subspaces

Compressed sensing

Introduction to a Transient World. Fourier Kingdom. Discrete Revolution. Time Meets Frequency. Frames. Wavelet Zoom. Wavelet Bases. Wavelet Packet and Local Cosine Bases. An Approximation Tour. Estimations are Approximations. Transform Coding. Appendix A: Mathematical Complements. Appendix B: Software Toolboxes.

http://ahvazdownload.persiangig.com/document/article/39.pdf

A wavelet tour of signal processing

The time-frequency and time-scale communities have recently developed a large number of overcomplete waveform dictionaries --- stationary wavelets, wavelet packets, cosine packets, chirplets, and warplets, to name a few. Decomposition into overcomplete systems is not unique, and several methods for decomposition have been proposed, including the method of frames (MOF), Matching pursuit (MP), and, for special dictionaries, the best orthogonal basis (BOB).
Basis Pursuit (BP) is a principle for decomposing a signal into an "optimal" superposition of dictionary elements, where optimal means having the smallest l1 norm of coefficients among all such decompositions. We give examples exhibiting several advantages over MOF, MP, and BOB, including better sparsity and superresolution. BP has interesting relations to ideas in areas as diverse as ill-posed problems, in abstract harmonic analysis, total variation denoising, and multiscale edge denoising.
BP in highly overcomplete dictionaries leads to large-scale optimization problems. With signals of length 8192 and a wavelet packet dictionary, one gets an equivalent linear program of size 8192 by 212,992. Such problems can be attacked successfully only because of recent advances in linear programming by interior-point methods. We obtain reasonable success with a primal-dual logarithmic barrier method and conjugate-gradient solver.

Atomic Decomposition by Basis Pursuit

In recent years there has been a growing interest in the study of sparse representation of signals. Using an overcomplete dictionary that contains prototype signal-atoms, signals are described by sparse linear combinations of these atoms. Applications that use sparse representation are many and include compression, regularization in inverse problems, feature extraction, and more. Recent activity in this field has concentrated mainly on the study of pursuit algorithms that decompose signals with respect to a given dictionary. Designing dictionaries to better fit the above model can be done by either selecting one from a prespecified set of linear transforms or adapting the dictionary to a set of training signals. Both of these techniques have been considered, but this topic is largely still open. In this paper we propose a novel algorithm for adapting dictionaries in order to achieve sparse signal representations. Given a set of training signals, we seek the dictionary that leads to the best representation for each member in this set, under strict sparsity constraints. We present a new method-the K-SVD algorithm-generalizing the K-means clustering process. K-SVD is an iterative method that alternates between sparse coding of the examples based on the current dictionary and a process of updating the dictionary atoms to better fit the data. The update of the dictionary columns is combined with an update of the sparse representations, thereby accelerating convergence. The K-SVD algorithm is flexible and can work with any pursuit method (e.g., basis pursuit, FOCUSS, or matching pursuit). We analyze this algorithm and demonstrate its results both on synthetic tests and in applications on real image data

https://users.ece.utexas.edu/~sanghavi/courses/papers/KSVD.pdf

$rm K$ -SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation

We describe a recursive algorithm to compute representations of functions with respect to nonorthogonal and possibly overcomplete dictionaries of elementary building blocks e.g. affine (wavelet) frames. We propose a modification to the matching pursuit algorithm of Mallat and Zhang (1992) that maintains full backward orthogonality of the residual (error) at every step and thereby leads to improved convergence. We refer to this modified algorithm as orthogonal matching pursuit (OMP). It is shown that all additional computation required for the OMP algorithm may be performed recursively. >

/pdf/orthogonal-matching-pursuit-recursive-function-approximation-5f0w6oel2e.pdf

Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition

The problem of automated design of phase-shifting masks for enhanced-resolution optical lithography is examined. We propose a computationally viable algorithm for the rapid design of phase-shifting masks for arbitrary two-dimensional patterns. Our approach is based on the use of a class of optimal coherent approximations to partially coherent imaging systems described by the Hopkins model. These approximations lead to substantial computational and analytical benefits, and, in addition, the resultant approximation error can be quite small for imaging systems with coherence factor σ ≤ 0.5. These approximate models allow us to reduce the mask-design problem to the classical phase-retrieval problem in optics. A fast iterative algorithm, closely related to the Gerchberg–Saxton algorithm, is then applied to generate (suboptimal) phase-shifting masks. Analytical results related to practical requirements for phase-shifting masks are also presented. These results address questions related to the number of discrete phase levels required for arbitrary patterns and provide some insight into alternative strategies for the use of phase-shifting masks. A number of simulated phase-shifting mask-design examples are provided to illustrate the methods and ideas presented.

Phase-shifting masks for microlithography: automated design and mask requirements

In earlier work by Pati and Krishnaprasad (1992) it was shown that rational wavelet frame decompositions of the Hardy space H/sup 2/(II/sup +/) may be used to efficiently capture time-frequency localized behavior of stable linear systems, for purposes of system identification and model-reduction. In this paper we examine the problem of efficient computation of low-order rational wavelet approximations of stable linear systems. We describe a variant of the matching pursuit algorithm of Mallat and Zhang (1992) that utilizes successive projections onto two-dimensional subspaces to construct rational wavelet approximants. The methods described here are illustrated by means of both simulations and experimental results. >

/pdf/a-fast-recursive-algorithm-for-system-identification-and-45sir0x7ed.pdf

A fast recursive algorithm for system identification and model reduction using rational wavelets

In this paper we describe the application of rational wavelet decompositions of the Hardy space H/sup 2/(II/sup +/) to rational approximation of transfer functions of stable, linear, time-invariant systems. The intent of the current paper is to provide an account of the authors' earlier work (1992) in this area and to describe some developments in efficient computational methods for rational wavelet approximation. It is shown by means of several examples that rational wavelets can provide 'good' approximations to some important classes of linear systems. >

Rational wavelets in model reduction and system identification

We introduce the so-called moment expansion of a defocused image as a tool for analyzing and improving the depth of focus in optical imaging. It is shown that a number of previously noted defocus phenomena can be readily derived or explained in terms of moment expansion. Some potential applications of the moment expansion to phase-shifting mask and pupil filter design for optical lithography are also briefly noted.

Y.C. Pati

Papers

Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition

Phase-shifting masks for microlithography: automated design and mask requirements

A fast recursive algorithm for system identification and model reduction using rational wavelets

Rational wavelets in model reduction and system identification

Depth of focus and the moment expansion.