Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

From the Publisher:
A valuable reference for the novice as well as for the expert who needs a wider scope of coverage within the area of cryptography, this book provides easy and rapid access of information and includes more than 200 algorithms and protocols; more than 200 tables and figures; more than 1,000 numbered definitions, facts, examples, notes, and remarks; and over 1,250 significant references, including brief comments on each paper.

Handbook of Applied Cryptography

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. This motivates longer term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation, and manifold learning.

/pdf/representation-learning-a-review-and-new-perspectives-4sidphwbhg.pdf

Representation Learning: A Review and New Perspectives

This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors.

/pdf/tensor-decompositions-and-applications-46vl2ih5gn.pdf

Tensor Decompositions and Applications

Can machine learning deliver AI? Theoretical results, inspiration from the brain and cognition, as well as machine learning experiments suggest that in order to learn the kind of complicated functions that can represent high-level abstractions (e.g. in vision, language, and other AI-level tasks), one would need deep architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers, graphical models with many levels of latent variables, or in complicated propositional formulae re-using many sub-formulae. Each level of the architecture represents features at a different level of abstraction, defined as a composition of lower-level features. Searching the parameter space of deep architectures is a difficult task, but new algorithms have been discovered and a new sub-area has emerged in the machine learning community since 2006, following these discoveries. Learning algorithms such as those for Deep Belief Networks and other related unsupervised learning algorithms have recently been proposed to train deep architectures, yielding exciting results and beating the state-of-the-art in certain areas. Learning Deep Architectures for AI discusses the motivations for and principles of learning algorithms for deep architectures. By analyzing and comparing recent results with different learning algorithms for deep architectures, explanations for their success are proposed and discussed, highlighting challenges and suggesting avenues for future explorations in this area.

/pdf/learning-deep-architectures-for-ai-2358znl654.pdf

Learning Deep Architectures for AI

It is proved that if $n$ is a power of 2, then there is a threshold function on $n$ inputs that requires weights of size around $2^{(n\log n)/2-n}$. This almost matches the known upper bounds.

/pdf/on-the-size-of-weights-for-threshold-gates-d84uhu17ke.pdf

On the Size of Weights for Threshold Gates

We prove optimal O(log n/log log n) lower bounds on the time for CRCW PRAM's with polynomially bounded numbers of processors or memory cells to compute parity and a number of related problems. We also exhibit a strict time hierarchy of explicit Boolean functions of n bits on such machines which holds up to O(log n/log log n) time. Furthermore, we show that almost all Boolean functions of n bits require log n - log log n + O(1) time when the number of processors is at most polynomial in n. Our bounds do not place restrictions on the uniformity of the algorithms nor on the instruction sets of the machines.

/pdf/optimal-bounds-for-decision-problems-on-the-crcw-pram-4mfv643g6m.pdf

Optimal bounds for decision problems on the CRCW PRAM

1. I n t r o d u c t i o n Randomness have been playing an ever increasing role in computer science. The demand for cheap random bits and repeatabil i ty of random experiments have resulted in the need for pseudorandom generators. Clearly one has to be careful in the design of a pseudorandom generator to make sure that the output behaves "randomly". Classically this meant that the sequence produced by the generator passed certain statistical tests (see Knuth [K]). This point of view was dramatically changed in 1982 by the papers by Blum and Micali IBM] and Yao [Y]. One important change was a new definition of what should be required of a pseudorandom generator, namely its output should be indistinguishable from truly random bits by all probabilistic polynomial t ime algorithms. The second important contribution was the construction of pseudorandom generators satisfying this new stringent definition (under some specific assumption). Blum and Micali's generator IBM] was proved secure under the assumption that the discrete logar i thm problem was hard and Yao generalized this to show how to construct a secure generator from any oneway permutat ion. Of course it would be very nice to prove that a given generator is secure without any assumptions, Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. but to unconditionally prove that a generator is secure is at least as difficult as proving that N P ~ P and thus this seems too much to ask for. The next best thing is to investigate exactly what assumptions are required. In 1985, Levin [L] proved that necessary and sufficient conditions for the existence of a secure generator is the existence of a function which is oneway on its iterates. This condition is not, however, completely natural and thus the search for more natural conditions continued. Goldreich, Krawczyk and Luby [GKL] gave a construction for a secure generator given the existence of a regular oneway function. Then recently, Impagliazzo, Levin and Luby JILL] proved that the existence of a oneway function in the nonuniform model (i.e. cannot be inverted by polynomial size circuits) implies the existence of nonuniformly secure generators (i.e. cannot be distinguished from random bits by polynomial size circuits). Their teclmique did not extend to show that the existence of a oneway function in the uniform model guarantees the existence of generators which are uniformly secure, and the main open question in the area was to prove that this is the case. The main result of this paper is to indeed prove that oneway functions in the uniform model imply uniformly secure generators. Since it is easy to see that secure generators imply oneway functions we have hence obtained natural necessary and sufficient conditions for the existence of secure generators. Our construction uses mostly the same ideas as [ILL]. A key concept of JILL] is that of a false entropy generator (FEG). A function g is a FEG if the distribution G generated by picking a random x and computing g(x) is indistinguishable from a distribution D, where D has higher entropy than G. In the uniform case we also want D to be polynomial time samplable i.e. that there is a probabilistic polynomial t ime algorithm that outputs a random element from D: JILL] proved that the existence of a FEG implies the existence of a secure pseudorandom generator (this has been strengthened by Oded Go, tire© 1990 ACM 089791-361-2/90/0005/0395 $1.50 395 ich [G] to show that if there are two distributions that computationally indistinguishable but statistically different then there exists secure generators). Let f be a oneway permutat ion and b a predicate that is hard to approximate given the information f (x) . Consider g(x) = (f(~:), b(x)) and let G be the distribution generated by g on a random input. Since b is hard to approximate, G is indistinguishable from D = (f(x), r) where r is a random bit and D has higher entropy than G since if f is a permutation, then f (x) determines z and hence b(z) uniquely. Thus g is a FEG. It seems like this construction on top of a oneway function f needs a predicate b which is hard given f . However by the elegant result by Goldreich and Levin [GL] we know that we can always take b to be a random xor. Let us now turn to the case where f is not a permutation. In this case ( f (x ) ,b (z ) ) is still indistinguishable from (f(x), r) but g might still not be a FEG. The reason is that f ( x ) does not any more determine b(x) and hence it is not clear that changing b(x) to a random bit increases the entropy, fILL] solved this problem by output t ing also a hashfunction h(x) in order to determine x uniquely, and letting g'(x) = (f(z) , h(z), b(x)). The corresponding distribution G ~ now has lower entropy then the distribution given by ( f (x) , h(z), r) where r is a random bit. However, releasing extra information about x might help an algorithm that tries to invert f and if h(x) outputs too many bits, the mapping x (f(z), h(x)) might not be oneway and thus the output of g' could be distinguishable from (f(x), h(z), r). It turns out that it is both profitable and safe to change b(x) to a random bit when h outputs approximately log i f -1 ( f(z)) I bits. However this creates a nonuniform competing distribution (since cannot compute this number) and this is the source of the nonuniformity of fILL]. We get around this problem by generating many xi and computing f(xi), b(xi) and h(xi) for hash functions of random sizes. We then output all f(xi), h(xi), 11~[1~ and M where M is random matrix and B is the vector of the b(xi) values. If the size of M is chosen correctly this is indistinguishable from the distribution where everything is identical except that B is chosen randomly and this latter distribution has a higher entropy. The main difficulty in the proof is to prove that MB is indistinguishable from M Y where Y is picked randomly. The proof of this is similar to the proof of Levin [L] of the sharp version of Yao's XOR-lemma. The present lemma proves more than is needed for the present application. Namely, we prove that if B consists of bits that each can be guessed with probability at most (1 + p)/2 (over the input and the coinflips of the algorithm) then the conclusion follows (for suitable choice for the size of M). However, in our application, with probability 1 p (over the input), no probabilistic polynomial time algorithm can gain any significant advantage over guessing, and with probability p (over the input) the bit might be totally predictable. Using this stronger property it is possible to give a simpler proof. This simpler proof will appear in a forthcoming journal paper, which will be a combination of the present paper and fILL]. In this paper we prove the stronger lemma by the more complicated proof since we believe that the stronger version of the lemma is interesting on its

Pseudo-random generators under uniform assumptions

Optimal O(log n/log log n) lower bounds on the time for CRCW PRAMS with polynomially bounded numbers of processors or memory cells to compute parity and a number of related problems are proven. A strict time hierarchy of explicit Boolean functions of n bits on such machines that holds up to O(log n/log log n) time is also exhibited. That is, for every time bound T within this range a function is exhibited that can be easily computed using polynomial resources in time T but requires more than polynomial resources to be computed in time T - 1. Finally, it is shown that almost all Boolean functions of n bits require log n - log log n + O(1) time when the number of processors is at most polynomial in n. The bounds do not place restrictions on the uniformity of the algorithms nor on the instruction sets of the machines.

/pdf/optimal-bounds-for-decision-problems-on-the-crcw-pram-1zs0nt2juf.pdf

Informally, an error-correcting code has "nice" list-decodability properties if every Hamming ball of "large" radius has a "small" number of codewords in it. We report linear codes with nontrivial list-decodability: i.e., codes of large rate that are nicely list-decodable, and codes of large distance that are not nicely list-decodable. Specifically, on the positive side, we show that there exist codes of rate R and block length n that have at most c codewords in every Hamming ball of radius H/sup -1/(1-R-1/c)/spl middot/n. This answers the main open question from the work of Elias (1957). This result also has consequences for the construction of concatenated codes of good rate that are list decodable from a large fraction of errors, improving previous results of Guruswami and Sudan (see IEEE Trans. Inform. Theory, vol.45, p.1757-67, Sept. 1999, and Proc. 32nd ACM Symp. Theory of Computing (STOC), Portland, OR, p. 181-190, May 2000) in this vein. Specifically, for every /spl epsi/ > 0, we present a polynomial time constructible asymptotically good family of binary codes of rate /spl Omega/(/spl epsi//sup 4/) that can be list-decoded in polynomial time from up to a fraction (1/2-/spl epsi/) of errors, using lists of size O(/spl epsi//sup -2/). On the negative side, we show that for every /spl delta/ and c, there exists /spl tau/ 0, and an infinite family of linear codes {C/sub i/}/sub i/ such that if n/sub i/ denotes the block length of C/sub i/, then C/sub i/ has minimum distance at least /spl delta/ /spl middot/ n/sub i/ and contains more than c/sub 1/ /spl middot/ n/sub i//sup c/ codewords in some Hamming ball of radius /spl tau/ /spl middot/ n/sub i/. While this result is still far from known bounds on the list-decodability of linear codes, it is the first to bound the "radius for list-decodability by a polynomial-sized list" away from the minimum distance of the code.

/pdf/combinatorial-bounds-for-list-decoding-6dwjxua0ep.pdf

Johan Håstad

Papers

On the Size of Weights for Threshold Gates

Optimal bounds for decision problems on the CRCW PRAM

Pseudo-random generators under uniform assumptions

Optimal bounds for decision problems on the CRCW PRAM

Combinatorial bounds for list decoding