We will review some of the major results in random graphs and some of the more challenging open problems. We will cover algorithmic and structural questions. We will touch on newer models, including those related to the WWW.

Random graphs

tions. Bootstrap has found many applications in engineering field, including artificial neural networks, biomedical engineering, environmental engineering, image processing, and radar and sonar signal processing. Basic concepts of the bootstrap are summarized in each section as a step-by-step algorithm for ease of implementation. Most of the applications are taken from the signal processing literature. The principles of the bootstrap are introduced in Chapter 2. Both the nonparametric and parametric bootstrap procedures are explained. Babu and Singh (1984) have demonstrated that in general, these two procedures behave similarly for pivotal (Studentized) statistics. The fact that the bootstrap is not the solution for all of the problems has been known to statistics community for a long time; however, this fact is rarely touched on in the manuscripts meant for practitioners. It was first observed by Babu (1984) that the bootstrap does not work in the infinite variance case. Bootstrap Techniques for Signal Processing explains the limitations of bootstrap method with an example. I especially liked the presentation style. The basic results are stated without proofs; however, the application of each result is presented as a simple step-by-step process, easy for nonstatisticians to follow. The bootstrap procedures, such as moving block bootstrap for dependent data, along with applications to autoregressive models and for estimation of power spectral density, are also presented in Chapter 2. Signal detection in the presence of noise is generally formulated as a testing of hypothesis problem. Chapter 3 introduces principles of bootstrap hypothesis testing. The topics are introduced with interesting real life examples. Flow charts, typical in engineering literature, are used to aid explanations of the bootstrap hypothesis testing procedures. The bootstrap leads to second-order correction due to pivoting; this improvement in the results due to pivoting is also explained. In the second part of Chapter 3, signal processing is treated as a regression problem. The performance of the bootstrap for matched filters as well as constant false-alarm rate matched filters is also illustrated. Chapters 2 and 3 focus on estimation problems. Chapter 4 introduces bootstrap methods used in model selection. Due to the inherent structure of the subject matter, this chapter may be difficult for nonstatisticians to follow. Chapter 5 is the most impressive chapter in the book, especially from the standpoint of statisticians. It provides real data bootstrap applications to illustrate the theory covered in the earlier chapters. These include applications to optimal sensor placement for knock detection and land-mine detection. The authors also provide a MATLAB toolbox comprising frequently used routines. Overall, this is a very useful handbook for engineers, especially those working in signal processing.

Probability and Random Processes

Probability and Measure

High-dimensional probability offers insight into the behavior of random vectors, random matrices, random subspaces, and objects used to quantify uncertainty in high dimensions Drawing on ideas from probability, analysis, and geometry, it lends itself to applications in mathematics, statistics, theoretical computer science, signal processing, optimization, and more It is the first to integrate theory, key tools, and modern applications of high-dimensional probability Concentration inequalities form the core, and it covers both classical results such as Hoeffding's and Chernoff's inequalities and modern developments such as the matrix Bernstein's inequality It then introduces the powerful methods based on stochastic processes, including such tools as Slepian's, Sudakov's, and Dudley's inequalities, as well as generic chaining and bounds based on VC dimension A broad range of illustrations is embedded throughout, including classical and modern results for covariance estimation, clustering, networks, semidefinite programming, coding, dimension reduction, matrix completion, machine learning, compressed sensing, and sparse regression

High-Dimensional Probability: An Introduction with Applications in Data Science

From social networks such as Facebook, the World Wide Web and the Internet, to the complex interactions between proteins in the cells of our bodies, we constantly face the challenge of understanding the structure and development of networks. The theory of random graphs provides a framework for this understanding, and in this book the authors give a gentle introduction to the basic tools for understanding and applying the theory. Part I includes sufficient material, including exercises, for a one semester course at the advanced undergraduate or beginning graduate level. The reader is then well prepared for the more advanced topics in Parts II and III. A final part provides a quick introduction to the background material needed. All those interested in discrete mathematics, computer science or applied probability and their applications will find this an ideal introduction to the subject.

/pdf/introduction-to-random-graphs-3cfvwg09nu.pdf

Introduction to random graphs

We prove new upper and lower bounds on the VC-dimension of deep neural networks with the ReLU activation function. These bounds are tight for almost the entire range of parameters. Letting $W$ be the number of weights and $L$ be the number of layers, we prove that the VC-dimension is $O(W L \log(W))$, and provide examples with VC-dimension $\Omega( W L \log(W/L) )$. This improves both the previously known upper bounds and lower bounds. In terms of the number $U$ of non-linear units, we prove a tight bound $\Theta(W U)$ on the VC-dimension. All of these bounds generalize to arbitrary piecewise linear activation functions, and also hold for the pseudodimensions of these function classes. 
Combined with previous results, this gives an intriguing range of dependencies of the VC-dimension on depth for networks with different non-linearities: there is no dependence for piecewise-constant, linear dependence for piecewise-linear, and no more than quadratic dependence for general piecewise-polynomial.

/pdf/nearly-tight-vc-dimension-and-pseudodimension-bounds-for-25svr7hkwn.pdf

Nearly-tight VC-dimension and Pseudodimension Bounds for Piecewise Linear Neural Networks

We prove a lower bound and an upper bound for the total variation distance between two high-dimensional Gaussians, which are within a constant factor of one another.

The total variation distance between high-dimensional Gaussians

We prove new upper and lower bounds on the VC-dimension of deep neural
networks with the ReLU activation function. These bounds are tight for almost
the entire range of parameters. Letting $W$ be the number of weights and $L$ be
the number of layers, we prove that the VC-dimension is $O(W L \log(W))$ and
$\Omega( W L \log(W/L) )$. This improves both the previously known upper bounds
and lower bounds. In terms of the number $U$ of non-linear units, we prove a
tight bound $\Theta(W U)$ on the VC-dimension. All of these results generalize
to arbitrary piecewise linear activation functions.

Nearly-tight VC-dimension bounds for piecewise linear neural networks

Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks

Let A be an isotropic, sub-gaussian m × n matrix. We prove that the process $Z_{x}\,:=\,\left \|Ax\right \|_{2} -\sqrt{m}\left \|x\right \|_{2}$ has sub-gaussian increments, that is, $\|Z_{x} - Z_{y}\|_{\psi _{2}} \leq C\|x - y\|_{2}$ for any $x,y \in \mathbb{R}^{n}$. Using this, we show that for any bounded set $T \subseteq \mathbb{R}^{n}$, the deviation of ∥ Ax ∥ 2 around its mean is uniformly bounded by the Gaussian complexity of T. We also prove a local version of this theorem, which allows for unbounded sets. These theorems have various applications, some of which are reviewed in this paper. In particular, we give a new result regarding model selection in the constrained linear model.

/pdf/a-simple-tool-for-bounding-the-deviation-of-random-matrices-324lzpxl57.pdf

Abbas Mehrabian

Papers

Nearly-tight VC-dimension and Pseudodimension Bounds for Piecewise Linear Neural Networks

The total variation distance between high-dimensional Gaussians

Nearly-tight VC-dimension bounds for piecewise linear neural networks

Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks

A simple tool for bounding the deviation of random matrices on geometric sets