Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. This motivates longer term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation, and manifold learning.

/pdf/representation-learning-a-review-and-new-perspectives-4sidphwbhg.pdf

Representation Learning: A Review and New Perspectives

/pdf/semi-supervised-learning-literature-survey-4ccxy3rmmj.pdf

Semi-Supervised Learning Literature Survey

Convex bodies: the brunn–minkowski theory

The goal of this paper is not to introduce a single algorithm or method, but to make theoretical steps towards fully understanding the training dynamics of generative adversarial networks. In order to substantiate our theoretical analysis, we perform targeted experiments to verify our assumptions, illustrate our claims, and quantify the phenomena. This paper is divided into three sections. The first section introduces the problem at hand. The second section is dedicated to studying and proving rigorously the problems including instability and saturation that arize when training generative adversarial networks. The third section examines a practical and theoretically grounded direction towards solving these problems, while introducing new tools to study them.

Towards Principled Methods for Training Generative Adversarial Networks

The hypothesis that high dimensional data tend to lie in the vicinity of a low dimensional manifold is the basis of manifold learning. The goal of this paper is to develop an algorithm (with accompanying complexity guarantees) for fitting a manifold to an unknown probability distribution supported in a separable Hilbert space, only using i.i.d samples from that distribution. More precisely, our setting is the following. Suppose that data are drawn independently at random from a probability distribution $P$ supported on the unit ball of a separable Hilbert space $H$. Let $G(d, V, \tau)$ be the set of submanifolds of the unit ball of $H$ whose volume is at most $V$ and reach (which is the supremum of all $r$ such that any point at a distance less than $r$ has a unique nearest point on the manifold) is at least $\tau$. Let $L(M, P)$ denote mean-squared distance of a random point from the probability distribution $P$ to $M$. 
We obtain an algorithm that tests the manifold hypothesis in the following sense. 
The algorithm takes i.i.d random samples from $P$ as input, and determines which of the following two is true (at least one must be): 
(a) There exists $M \in G(d, CV, \frac{\tau}{C})$ such that $L(M, P) \leq C \epsilon.$ 
(b) There exists no $M \in G(d, V/C, C\tau)$ such that $L(M, P) \leq \frac{\epsilon}{C}.$ 
The answer is correct with probability at least $1-\delta$.

Testing the manifold hypothesis

The hypothesis that high dimensional data tends to lie in the vicinity of a low dimensional manifold is the basis of a collection of methodologies termed Manifold Learning. In this paper, we study statistical aspects of the question of fitting a manifold with a nearly optimal least squared error. Given upper bounds on the dimension, volume, and curvature, we show that Empirical Risk Minimization can produce a nearly optimal manifold using a number of random samples that is independent of the ambient dimension of the space in which data lie. We obtain an upper bound on the required number of samples that depends polynomially on the curvature, exponentially on the intrinsic dimension, and linearly on the intrinsic volume. For constant error, we prove a matching minimax lower bound on the sample complexity that shows that this dependence on intrinsic dimension, volume and curvature is unavoidable. Whether the known lower bound of O(k/e2 + log 1/δ/e2) for the sample complexity of Empirical Risk minimization on k-means applied to data in a unit ball of arbitrary dimension is tight, has been an open question since 1997 [3]. Here e is the desired bound on the error and δ is a bound on the probability of failure. We improve the best currently known upper bound [14] of O(k2/e2 + log 1/δ/e2) to O(k/e2 (min (k, + log4 k/e/e2)) + log 1/δ/e2). Based on these results, we devise a simple algorithm for k-means and another that uses a family of convex programs to fit a piecewise linear curve of a specified length to high dimensional data, where the sample complexity is independent of the ambient dimension.

/pdf/sample-complexity-of-testing-the-manifold-hypothesis-1eymyymuey.pdf

Sample Complexity of Testing the Manifold Hypothesis

Mathematical Preliminaries Sets Vectors and matrices Linear inequality systems Graphs Graphs: basic notions Graphs and vector spaces Basic operations on graphs and vector spaces Problems Graph algorithms Duality Notes Matroids Axiom systems for matroids Dual of a matroid Minors of matroids Connectedness in matroids Matroids and the greedy algorithm Notes Electrical Networks In terms of multiterminal devices In terms of 2-terminal devices Standard devices Common methods of analysis Procedures used in circuit simulators State equations for dynamic networks Multiports in electrical networks Some elementary results of network theory Notes Topological Hybrid Analysis Electrical network: a formal description Some basic topological results A theorem on topological hybrid analysis Structure of constraints and optimization Notes The Implicit Duality Theorem and its Applications The vector space version *Quasi orthogonality Applications of the implicit duality theorem *Linear inequality systems *ast Integrality systems Problems Notes Multiport Decomposition Multiport decomposition of vector spaces Analysis through multiport decomposition Port minimization * Multiport decomposition for network reduction Problems Submodular Functions Submodularity Basic operations on semimodular functions *Other operations on semimodular functions Polymatroid and matroid rank functions Connectedness for semimodular functions *Semimodular polyhedra Symmetric submodular functions Problems Notes Convolution of Submodular Functions Convolution Matroids, polymatroids and convolution The principal partition *The refined partial order of the principal partition Algorithms for PP *Aligned polymatroid rank functions Notes Matroid Union Submodular functions induced through a bipartite graph Matroid union: algorithm and structure PP of the rank function of a matroid Notes Dilworth Truncation of Submodular Functions Dilworth truncation The principal lattice of partitions *Approximation algorithms through PLP for the min cost partition problem The PLP of duals and truncations *The principal lattice of partitions associated with special fusions Building submodular functions with desired PLP Notes Algorithms for the PLP of a Submodular Function Minimizing the partition associate of a submodular function Construction of the P-sequence of partitions Construction of the DTL Complexity of construction of the PLP Construction of the PLP of the dual PLP algorithms for (omega R Gamma) and -(omega R Epsilon L ) Structural changes in minimizing partitions Relation between PP and PLP Fast algorithms for principal partition of the rank function of a graph The Hybrid Rank Problem The hybrid rank problem - first formulation The hybrid rank problem - second formulation The hybrid rank problem - third formulation The hybrid rank problem - fourth formulation

Submodular functions and electrical networks

Kostka numbers and Littlewood-Richardson coefficients appear in combinatorics and representation theory. Interest in their computation stems from the fact that they are present in quantum mechanical computations since Wigner [15]. In recent times, there have been a number of algorithms proposed to perform this task [1---3, 11, 12]. The issue of their computational complexity has received at-tention in the past, and was raised recently by E. Rassart in [11]. We prove that the problem of computing either quantity is #P-complete. Thus, unless P = NP, which is widely disbelieved, there do not exist efficient algorithms that compute these numbers.

/pdf/on-the-complexity-of-computing-kostka-numbers-and-littlewood-3sf2nyhkas.pdf

On the complexity of computing Kostka numbers and Littlewood-Richardson coefficients

Let K be a polytope in Rn defined by m linear inequalities. We give a new Markov chain algorithm to draw a nearly uniform sample from K. The underlying Markov chain is the first to have a mixing time that is strongly polynomial when started from a “central” point. We use this result to design an affine interior point algorithm that does a single random walk to solve linear programs approximately.

/pdf/random-walks-on-polytopes-and-an-affine-interior-point-3e1b6vyizj.pdf

Hariharan Narayanan

Papers

Testing the manifold hypothesis

Sample Complexity of Testing the Manifold Hypothesis

Submodular functions and electrical networks

On the complexity of computing Kostka numbers and Littlewood-Richardson coefficients

Random Walks on Polytopes and an Affine Interior Point Method for Linear Programming