Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

From the Publisher:
A valuable reference for the novice as well as for the expert who needs a wider scope of coverage within the area of cryptography, this book provides easy and rapid access of information and includes more than 200 algorithms and protocols; more than 200 tables and figures; more than 1,000 numbered definitions, facts, examples, notes, and remarks; and over 1,250 significant references, including brief comments on each paper.

Handbook of Applied Cryptography

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. This motivates longer term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation, and manifold learning.

/pdf/representation-learning-a-review-and-new-perspectives-4sidphwbhg.pdf

Representation Learning: A Review and New Perspectives

This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors.

/pdf/tensor-decompositions-and-applications-46vl2ih5gn.pdf

Tensor Decompositions and Applications

Can machine learning deliver AI? Theoretical results, inspiration from the brain and cognition, as well as machine learning experiments suggest that in order to learn the kind of complicated functions that can represent high-level abstractions (e.g. in vision, language, and other AI-level tasks), one would need deep architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers, graphical models with many levels of latent variables, or in complicated propositional formulae re-using many sub-formulae. Each level of the architecture represents features at a different level of abstraction, defined as a composition of lower-level features. Searching the parameter space of deep architectures is a difficult task, but new algorithms have been discovered and a new sub-area has emerged in the machine learning community since 2006, following these discoveries. Learning algorithms such as those for Deep Belief Networks and other related unsupervised learning algorithms have recently been proposed to train deep architectures, yielding exciting results and beating the state-of-the-art in certain areas. Learning Deep Architectures for AI discusses the motivations for and principles of learning algorithms for deep architectures. By analyzing and comparing recent results with different learning algorithms for deep architectures, explanations for their success are proposed and discussed, highlighting challenges and suggesting avenues for future explorations in this area.

/pdf/learning-deep-architectures-for-ai-2358znl654.pdf

Learning Deep Architectures for AI

We investigate the relationship between circuit bottom fan-in and circuit size when circuit depth is fixed. We show that in order to compute certain functions, a moderate reduction in circuit bottom fan-in will cause significant increase in circuit size. In particular, we prove that there are functions that are computable by circuits of linear size and depth k with bottom fan-in 2 but require exponential size for circuits of depth k with bottom fan-in 1. A general scheme is established to study the trade-off between circuit bottom fan-in and circuit size. Based on this scheme, we are able to prove, for example, that for any integer c, there are functions that are computable by circuits of linear size and depth k with bottom fan-in $O(\log n)$ but that require exponential size for circuits of depth k with bottom fan-in c, and that for any constant $\epsilon > 0$, there are functions that are computable by circuits of linear size and depth k with bottom fan-in $\log n$ but that require superpolynomial size for circuits of depth k with bottom fan-in $O(\log^{1-\epsilon} n)$. A consequence of these results is that the three input read-modes of alternating Turing machines proposed in the literature are all distinct.

/pdf/circuit-bottom-fan-in-and-computational-power-3jct83x4j2.pdf

Circuit Bottom Fan-in and Computational Power

( ;iven n strings arranged in alphabetical order, how many characters must we probe to determine whether a k-character query string is present? If k is a constant, we can solve i IIe problem with @(log n) probes by means of binary search, .Ind this is optimal, but what happens for larger values of k? The question 1s a fundamental one; we are simply askiiig for the complexity of searching a dictionary for a string, ~vhere the common assumption that entire strings can be compared in constant time is replaced by the more conservative assumption that only single characters can be compmed in constant time. For s&iciently long strings, the latter assumption seems more realistic. At first glance the problem may appear easy — some liind of generalized binary search should do the trick. However, closer acquaintance with the problem reveals a surprising, intricacy. 13eing slightly more precise, we consider finite strings of characters drawn from an arbitrary, ordered alphabet.

The complexity of searching a sorted array of strings

It is well-known that every lattice L has a basis consisting of relatively short vectors. To state a quantitative version of this fact, let 21(L) denote the i-th successive minimum of L, which is the smallest value such that there are i linearly independent vectors in L in the ball {x: [I x II _-< ~.i(L)}. In particular, let A, be the minimal constant such that every lattice of rank n possesses a basis which satisfies

Simultaneously good bases of a lattice and its reciprocal lattice

We prove improved inapproximability results for hypergraph coloring using the low-degree polynomial code (aka the “short code” of Barak et al. [SIAM J. Comput., 44 (2015), pp. 1287--1324]) and the techniques proposed by Dinur and Guruswami [Israel J. Math., 209 (2015), pp. 611--649] to incorporate this code for inapproximability results. In particular, we prove quasi NP-hardness of the following problems on $n$-vertex hypergraphs: coloring a 2-colorable 8-uniform hypergraph with $2^{2^{\Omega(\sqrt{\log \log n})}}$ colors; coloring a 4-colorable 4-uniform hypergraph with $2^{2^{\Omega(\sqrt{\log \log n})}}$ colors; and coloring a 3-colorable 3-uniform hypergraph with $(\log n)^{\Omega(1/\log\log\log n)}$ colors. For the first two cases, the hardness results obtained are superpolynomial in what was previously known, and in the last case it is an exponential improvement. In fact, prior to this result, $(\log n)^{O(1)}$ colors was the strongest quantitative bound on the number of colors ruled out by inapprox...

Super-polylogarithmic hypergraph coloring hardness via low-degree long codes.

In this paper we further study the complexity of zero-knowledge interactive proofs. We prove that there is an oracle A such that there is a language L which is recognizable by a two round, perfect zero-knowledge interactive proof relative to A, but such that L ∋ BPPA. This gives interesting implications for what can be demonstrated about zero-knowledge interactive proofs using standard methods.

Johan Håstad

Papers

Circuit Bottom Fan-in and Computational Power

The complexity of searching a sorted array of strings

Simultaneously good bases of a lattice and its reciprocal lattice

Super-polylogarithmic hypergraph coloring hardness via low-degree long codes.

Relativized perfect zero knowledge is not BPP