Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

From the Publisher:
A valuable reference for the novice as well as for the expert who needs a wider scope of coverage within the area of cryptography, this book provides easy and rapid access of information and includes more than 200 algorithms and protocols; more than 200 tables and figures; more than 1,000 numbered definitions, facts, examples, notes, and remarks; and over 1,250 significant references, including brief comments on each paper.

Handbook of Applied Cryptography

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. This motivates longer term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation, and manifold learning.

/pdf/representation-learning-a-review-and-new-perspectives-4sidphwbhg.pdf

Representation Learning: A Review and New Perspectives

This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order tensors (i.e., $N$-way arrays with $N \geq 3$) have applications in psycho-metrics, chemometrics, signal processing, numerical linear algebra, computer vision, numerical analysis, data mining, neuroscience, graph analysis, and elsewhere. Two particular tensor decompositions can be considered to be higher-order extensions of the matrix singular value decomposition: CANDECOMP/PARAFAC (CP) decomposes a tensor as a sum of rank-one tensors, and the Tucker decomposition is a higher-order form of principal component analysis. There are many other tensor decompositions, including INDSCAL, PARAFAC2, CANDELINC, DEDICOM, and PARATUCK2 as well as nonnegative variants of all of the above. The N-way Toolbox, Tensor Toolbox, and Multilinear Engine are examples of software packages for working with tensors.

/pdf/tensor-decompositions-and-applications-46vl2ih5gn.pdf

Tensor Decompositions and Applications

Can machine learning deliver AI? Theoretical results, inspiration from the brain and cognition, as well as machine learning experiments suggest that in order to learn the kind of complicated functions that can represent high-level abstractions (e.g. in vision, language, and other AI-level tasks), one would need deep architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers, graphical models with many levels of latent variables, or in complicated propositional formulae re-using many sub-formulae. Each level of the architecture represents features at a different level of abstraction, defined as a composition of lower-level features. Searching the parameter space of deep architectures is a difficult task, but new algorithms have been discovered and a new sub-area has emerged in the machine learning community since 2006, following these discoveries. Learning algorithms such as those for Deep Belief Networks and other related unsupervised learning algorithms have recently been proposed to train deep architectures, yielding exciting results and beating the state-of-the-art in certain areas. Learning Deep Architectures for AI discusses the motivations for and principles of learning algorithms for deep architectures. By analyzing and comparing recent results with different learning algorithms for deep architectures, explanations for their success are proposed and discussed, highlighting challenges and suggesting avenues for future explorations in this area.

/pdf/learning-deep-architectures-for-ai-2358znl654.pdf

Learning Deep Architectures for AI

We study the security of individual bits in an RSA encrypted message EN(x). We show that given EN(x), predicting any single bit in x with only a nonnegligible advantage over the trivial guessing strategy, is (through a polynomial-time reduction) as hard as breaking RSA. Moreover, we prove that blocks of O(log log N) bits of x are computationally indistinguishable from random bits. The results carry over to the Rabin encryption scheme.Considering the discrete exponentiation function gx modulo p, with probability 1 − o(1) over random choices of the prime p, the analog results are demonstrated. The results do not rely on group representation, and therefore applies to general cyclic groups as well. Finally, we prove that the bits of ax + b modulo p give hard core predicates for any one-way function f.All our results follow from a general result on the chosen multiplier hidden number problem: given an integer N, and access to an algorithm Px that on input a random a i ZN, returns a guess of the ith bit of ax mod N, recover x. We show that for any i, if Px has at least a nonnegligible advantage in predicting the ith bit, we either recover x, or, obtain a nontrivial factor of N in polynomial time. The result also extends to prove the results about simultaneous security of blocks of O(log log N) bits.

/pdf/the-security-of-all-rsa-and-discrete-log-bits-1ril1gdaf2.pdf

The security of all RSA and discrete log bits

For every fixed finite field $\F_q$, $p \in (0,1-1/q)$ and $\epsilon > 0$, we prove that with high probability a random subspace $C$ of $\F_q^n$ of dimension $(1-H_q(p)-\epsilon)n$ has the property that every Hamming ball of radius $pn$ has at most $O(1/\epsilon)$ codewords. 
This answers a basic open question concerning the list-decodability of linear codes, showing that a list size of $O(1/\epsilon)$ suffices to have rate within $\epsilon$ of the "capacity" $1-H_q(p)$. Our result matches up to constant factors the list-size achieved by general random codes, and gives an exponential improvement over the best previously known list-size bound of $q^{O(1/\epsilon)}$. 
The main technical ingredient in our proof is a strong upper bound on the probability that $\ell$ random vectors chosen from a Hamming ball centered at the origin have too many (more than $\Theta(\ell)$) vectors from their linear span also belong to the ball.

On the List-Decodability of Random Linear Codes

We initiate the study of a new measure of approximation. This measure compares the performance of an approximation algorithm to the random assignment algorithm. Since the random assignment algorithm is known to give essentially the best possible polynomial time approximation algorithm for many optimization problems, this is a useful measure.In this paper, we focus on this measure for the optimization problems, Max-Lin-2, in which we need to maximize the number of satisfied linear equations in a system of linear equations modulo 2, and Max-$k$-Lin-2, a special case of the above problem in which each equation has at most $k$ variables. The main techniques we use, in our approximation algorithms and inapproximability results for this measure, are from Fourier analysis and derandomization.

/pdf/on-the-advantage-over-a-random-assignment-4k723e4wt8.pdf

On the advantage over a random assignment

We investigate a simple method of fraud management for secure devices that may serve as an alternative or complement to conventional hardware-based tamper resistance. Under normal operating conditions in our scheme, a secure device includes an authentication code in its communications, e.g., in the digital signatures it issues. This code may be verified by a fraud management center under a pre-determined key σ. When the device detects an attempted break-in, it modifies σ. This results in a change to the authentication codes issued by the device such that the fraud management center can detect the apparent break-in. Hence, in contrast to the case with typical tamper-resistance schemes, the deployer of our proposed scheme seeks to trace break-ins, rather than prevent them. In reference to the wartime practice of physically capturing and subverting underground radio transmitters – a practice analogous to the capture and use of secret information on secure devices – we denote this idea by the German term funkspiel, meaning “radio game.” One challenge in constructing a funkspiel scheme is to ensure that an attacker privy to the authentication codes of the secure device both before and after the break-in, as well as the secrets of the device following the break-in, cannot detect the alteration to σ. Additional challenges ∗Some of this work was done while visiting RSA Laboratories.

/pdf/funkspiel-schemes-an-alternative-to-conventional-tamper-1211xpmldu.pdf

Funkspiel schemes: an alternative to conventional tamper resistance

We prove that for any positive integer k, there is a constant ck such that a randomly selected set of ck nk log n Boolean vectors with high probability supports a balanced k-wise independent distribution. In the case of k ≤ 2 a more elaborate argument gives the stronger bound ck nk. Using a recent result by Austrin and Mossel this shows that a predicate on t bits, chosen at random among predicates accepting c2 t2 input vectors, is, assuming the Unique Games Conjecture, likely to be approximation resistant. These results are close to tight: we show that there are other constants, ck', such that a randomly selected set of cardinality ck' nk points is unlikely to support a balanced k-wise independent distribution and, for some c>0, a random predicate accepting ct2/log t input vectors is non-trivially approximable with high probability. In a different application of the result of Austrin and Mossel we prove that, again assuming the Unique Games Conjecture, any predicate on t bits accepting at least (32/33) • 2t inputs is approximation resistant. The results extend from the Boolean domain to larger finite domains.

/pdf/randomly-supported-independence-and-resistance-1shtm5fatw.pdf

Johan Håstad

Papers

The security of all RSA and discrete log bits

On the List-Decodability of Random Linear Codes

On the advantage over a random assignment

Funkspiel schemes: an alternative to conventional tamper resistance

Randomly supported independence and resistance