Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. This motivates longer term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation, and manifold learning.

/pdf/representation-learning-a-review-and-new-perspectives-4sidphwbhg.pdf

Representation Learning: A Review and New Perspectives

Can machine learning deliver AI? Theoretical results, inspiration from the brain and cognition, as well as machine learning experiments suggest that in order to learn the kind of complicated functions that can represent high-level abstractions (e.g. in vision, language, and other AI-level tasks), one would need deep architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers, graphical models with many levels of latent variables, or in complicated propositional formulae re-using many sub-formulae. Each level of the architecture represents features at a different level of abstraction, defined as a composition of lower-level features. Searching the parameter space of deep architectures is a difficult task, but new algorithms have been discovered and a new sub-area has emerged in the machine learning community since 2006, following these discoveries. Learning algorithms such as those for Deep Belief Networks and other related unsupervised learning algorithms have recently been proposed to train deep architectures, yielding exciting results and beating the state-of-the-art in certain areas. Learning Deep Architectures for AI discusses the motivations for and principles of learning algorithms for deep architectures. By analyzing and comparing recent results with different learning algorithms for deep architectures, explanations for their success are proposed and discussed, highlighting challenges and suggesting avenues for future explorations in this area.

/pdf/learning-deep-architectures-for-ai-2358znl654.pdf

Learning Deep Architectures for AI

We explore an original strategy for building deep networks, based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs. The resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders. It is however shown on a benchmark of classification problems to yield significantly lower classification error, thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from natural image patches and larger stroke detectors from digit images. This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations.

Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising 1 criterion

/pdf/stacked-denoising-autoencoders-learning-useful-2ag7q9txaq.pdf

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion

We investigate the power of threshold circuits of small depth. In particular, we give functions that require exponential size unweighted threshold circuits of depth 3 when we restrict the bottom fanin. We also prove that there are monotone functionsf

 k
 that can be computed in depthk and linear size ⋎, ⋏-circuits but require exponential size to compute by a depthk−1 monotone weighted threshold circuit.

On the power of small-depth threshold circuits

Small-depth circuits that contain threshold gates (with or without weights) and parity gates are studied. All circuits considered are of polynomial size. Several results that complete the work of characterizing possible inclusions between many classes defined by small-depth circuits are proved. >

Majority gates vs. general weighted threshold gates

The power of threshold circuits of small depth is investigated. In particular, functions that require exponential-size unweighted threshold circuits of depth 3 when the bottom fan-in is restricted are given. It is proved that there are monotone functions f/sub k/ that can be computed on depth k and linear size AND, OR circuits but require exponential-size to be computed by a depth-(k-1) monotone weighted threshold circuit. >

We study the computational complexity of solving systems of equations over a finite group. An equation over a group G is an expression of the form w1.w2....wk = 1G, where each wi is either a variable, an inverted variable, or a group constant and 1G is the identity element of G . A solution to such an equation is an assignment of the variables (to values in G) which realizes the equality. A system of equations is a collection of such equations; a solution is then an assigmnent which simultaneously realizes each equation. We show that the problem of determining if a (single) equation has a solution is NP-complete for all nonsolvable groups G. For nilpotent groups, this same problem is shown to be in P. The analogous problem for systems of such equations is shown to be NP-complete if G is non-Abelian, and in P otherwise. Finally, we observe some connections between these problems and the theory of nonuniform automata.

The complexity of solving equations over finite groups

We prove that a single threshold gate with arbitrary weights can be simulated by an explicit polynomial-size, depth-2 majority circuit. In general we show that a polynomial-size, depth-d threshold circuit can be simulated uniformly by a polynomial-size majority circuit of depth d + 1. Goldmann, Hastad, and Razborov showed in [Comput. Complexity, 2 (1992), pp. 277--300] that a nonuniform simulation exists. Our construction answers two open questions posed by them: we give an explicit construction, whereas they use a randomized existence argument, and we show that such a simulation is possible even if the depth d grows with the number of variables n (their simulation gives polynomial-size circuits only when d is constant).

Mikael Goldmann

Papers

On the power of small-depth threshold circuits

Majority gates vs. general weighted threshold gates

On the power of small-depth threshold circuits

The complexity of solving equations over finite groups

Simulating Threshold Circuits by Majority Circuits