We will review some of the major results in random graphs and some of the more challenging open problems. We will cover algorithmic and structural questions. We will touch on newer models, including those related to the WWW.

Random graphs

国際会議開催報告:2013 IEEE International Symposium on Information Theory

We consider the fundamental problem of solving quadratic systems of equations in $n$ variables, where $y_i = |\langle \boldsymbol{a}_i, \boldsymbol{x} \rangle|^2$, $i = 1, \ldots, m$ and $\boldsymbol{x} \in \mathbb{R}^n$ is unknown. We propose a novel method, which starting with an initial guess computed by means of a spectral method, proceeds by minimizing a nonconvex functional as in the Wirtinger flow approach. There are several key distinguishing features, most notably, a distinct objective functional and novel update rules, which operate in an adaptive fashion and drop terms bearing too much influence on the search direction. These careful selection rules provide a tighter initial guess, better descent directions, and thus enhanced practical performance. On the theoretical side, we prove that for certain unstructured models of quadratic systems, our algorithms return the correct solution in linear time, i.e. in time proportional to reading the data $\{\boldsymbol{a}_i\}$ and $\{y_i\}$ as soon as the ratio $m/n$ between the number of equations and unknowns exceeds a fixed numerical constant. We extend the theory to deal with noisy systems in which we only have $y_i \approx |\langle \boldsymbol{a}_i, \boldsymbol{x} \rangle|^2$ and prove that our algorithms achieve a statistical accuracy, which is nearly un-improvable. We complement our theoretical study with numerical examples showing that solving random quadratic systems is both computationally and statistically not much harder than solving linear systems of the same size---hence the title of this paper. For instance, we demonstrate empirically that the computational cost of our algorithm is about four times that of solving a least-squares problem of the same size.

/pdf/solving-random-quadratic-systems-of-equations-is-nearly-as-4qugf00r6a.pdf

Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems

Low-rank matrices play a fundamental role in modeling and computational methods for signal processing and machine learning. In many applications where low-rank matrices arise, these matrices cannot be fully sampled or directly observed, and one encounters the problem of recovering the matrix given only incomplete and indirect observations. This paper provides an overview of modern techniques for exploiting low-rank structure to perform matrix recovery in these settings, providing a survey of recent advances in this rapidly-developing field. Specific attention is paid to the algorithms most commonly used in practice, the existing theoretical guarantees for these algorithms, and representative practical applications of these techniques.

An Overview of Low-Rank Matrix Recovery From Incomplete Observations

Electronics is approaching a major paradigm shift because silicon transistor scaling no longer yields historical energy-efficiency benefits, spurring research towards beyond-silicon nanotechnologies. In particular, carbon nanotube field-effect transistor (CNFET)-based digital circuits promise substantial energy-efficiency benefits, but the inability to perfectly control intrinsic nanoscale defects and variability in carbon nanotubes has precluded the realization of very-large-scale integrated systems. Here we overcome these challenges to demonstrate a beyond-silicon microprocessor built entirely from CNFETs. This 16-bit microprocessor is based on the RISC-V instruction set, runs standard 32-bit instructions on 16-bit data and addresses, comprises more than 14,000 complementary metal–oxide–semiconductor CNFETs and is designed and fabricated using industry-standard design flows and processes. We propose a manufacturing methodology for carbon nanotubes, a set of combined processing and design techniques for overcoming nanoscale imperfections at macroscopic scales across full wafer substrates. This work experimentally validates a promising path towards practical beyond-silicon electronic systems. A 16-bit microprocessor built from over 14,000 carbon nanotube transistors may enable energy efficiency advances in electronics technologies beyond silicon.

Modern microprocessor built from complementary carbon nanotube transistors

The problem of recovering a matrix from an incomplete sampling of its entries—also known as matrix completion—arises in a wide variety of practical situations, including collaborative filtering, system identification, sensor localization, rank aggregation, and many more. While many of these applications have a relatively long history, recent advances in the closely related field of compressed sensing have enabled a burst of progress in the last few years, and we now have a strong base of theoretical results concerning matrix completion. A typical result from this literature is that a generic d × d matrix of rank r can be exactly recovered from O(r dpolylog(d)) randomly chosen entries. Similar results can be established in the case of noisy observations and approximately low-rank matrices. See [1] and references therein for further details. Although these results are quite impressive, there is an important gap between the statement of the problem as considered in the matrix completion literature and many of the most common applications discussed therein. As an example, consider collaborative filtering and the now-famous “Netflix problem.” In this setting, we assume that there is some unknown matrix whose entries each represent a rating for a particular user on a particular movie. Since any user will rate only a small subset of possible movies, we are only able to observe a small fraction of the total entries in the matrix, and our goal is to infer the unseen ratings from the observed ones. If the rating matrix is low-rank, then this would seem to be the exact problem studied in the matrix completion literature. However, there is a subtle difference: the theory developed in this literature generally assumes that observations consist of (possibly noisy) continuous-valued entries of the matrix, whereas in the Netflix problem the observations are “quantized” to the set of integers between 1 and 5. If we believe that it is possible for a user’s true rating for a particular movie to be, for example, 4.5, then we must account for the impact of this “quantization noise” on our recovery. Of course, one could potentially treat quantization simply as a form of bounded noise, but this is somewhat unsatisfying because the ratings aren’t just quantized — there are also hard limits placed on the minimum and maximum allowable ratings. (Why should we suppose that a movie given a rating of 5 could not have a true underlying rating of 6 or 7 or 10?) The inadequacy of standard matrix completion techniques in dealing with this effect is particularly pronounced when we consider recommender systems where each rating consists of a single bit representing a positive or negative rating (consider for example rating music on Pandora, the relevance of advertisements on Hulu, or posts on Reddit or MathOverflow). Similar situations arise in nearly every application that has been proposed for matrix completion, including the analysis of incomplete survey data, the recovery of pairwise distance matrices (multidimensional scaling), quantum state tomography, and many others. In such cases, the assumptions made in the existing theory of matrix completion do not apply, standard algorithms are ill-posed, and a new theory is required. In this work we describe the approach we take in [1] to extend the theory of matrix completion to the case of 1-bit observations. We consider a statistical model for such data where a binary output is generated according to a probability distribution which is parameterized by the corresponding entry of the unknown matrix M . The central question we ask is: “Given observations of this form, can we recover the underlying matrix?” Several new challenges arise when trying to develop a theory for 1-bit matrix completion. First, matrix completion is in some sense a more challenging problem than compressed sensing. Specifically, some additional difficulty arises because the set of low-rank matrices is “coherent” with single entry measurements—there will always be certain (sparse) low-rank matrices that we cannot hope to recover without essentially sampling every entry of the matrix. The typical way to deal with this possibility is to consider a reduced set of lowrank matrices by placing restrictions on the entry-wise maximum of the matrix or its singular vectors—informally, we require that the matrix is not too “spiky”. However, we introduce an entirely new dimension of ill-posedness by restricting ourselves to 1-bit observations. An example of this is described in detail in [1] and shows that in the case where we simply observe Y = sign(M), the problem of recovering M is illposed. To see this, let M = uv∗ for any vectors u,v ∈ R, and for simplicity assume that there are no zero entries in u or v. Now let ũ and ṽ be any vectors with the same sign pattern as u and v respectively. It is apparent that either M or M = ũṽ will yield the same observations Y , and thus M and M are indistinguishable. Note that while it is obvious that this 1-bit measurement process will destroy any information we have regarding the scaling of M , this ill-posedness remains even if we knew something about the scaling a priori (such as the Frobenius norm of M ). For any given set of observations, there will always be radically different possible matrices that are all consistent with observed measurements. After considering this example, the problem might seem hopeless. However, an interesting surprise is that when we add noise to the problem (that is, when we observe a subset of the matrix Y = sign(M + Z) where Z 6= 0 is an appropriate stochastic matrix) the picture completely changes—this noise has a “dithering” effect and the problem becomes well-posed. In fact, we will show that in this setting we can sometimes recover M to the same degree of accuracy that is possible when given access to completely unquantized measurements! In particular, under appropriate conditions, O(rd) measurements are sufficient to accurately recover M . We will provide an overview of these results and discuss a number of practical applications.

1-Bit matrix completion

Machine learning relies on the assumption that unseen test instances of a classification problem follow the same distribution as observed training data. However, this principle can break down when machine learning is used to make important decisions about the welfare (employment, education, health) of strategic individuals. Knowing information about the classifier, such individuals may manipulate their attributes in order to obtain a better classification outcome. As a result of this behavior -- often referred to as gaming -- the performance of the classifier may deteriorate sharply. Indeed, gaming is a well-known obstacle for using machine learning methods in practice; in financial policy-making, the problem is widely known as Goodhart's law. In this paper, we formalize the problem, and pursue algorithms for learning classifiers that are robust to gaming.We model classification as a sequential game between a player named "Jury" and a player named "Contestant." Jury designs a classifier, and Contestant receives an input to the classifier drawn from a distribution. Before being classified, Contestant may change his input based on Jury's classifier. However, Contestant incurs a cost for these changes according to a cost function. Jury's goal is to achieve high classification accuracy with respect to Contestant's original input and some underlying target classification function, assuming Contestant plays best response. Contestant's goal is to achieve a favorable classification outcome while taking into account the cost of achieving it.For a natural class of "separable" cost functions, and certain generalizations, we obtain computationally efficient learning algorithms which are near optimal, achieving a classification error that is arbitrarily close to the theoretical minimum. Surprisingly, our algorithms are efficient even on concept classes that are computationally hard to learn. For general cost functions, designing an approximately optimal strategy-proof classifier, for inverse-polynomial approximation, is NP-hard.

Strategic Classification

We study the performance of Reed–Solomon (RS) codes for the exact repair problem in distributed storage. Our main result is that, in some parameter regimes, Reed–Solomon codes are optimal regenerating codes, among maximum distance separable (MDS) codes with linear repair schemes. Moreover, we give a characterization of MDS codes with linear repair schemes, which holds in any parameter regime, and which can be used to give non-trivial repair schemes for RS codes in other settings. More precisely, we show that for $k$ -dimensional RS codes whose evaluation points are a finite field of size $n$ , there are exact repair schemes with bandwidth $(n-1)\log ((n-1)/(n-k))$ bits, and that this is optimal for any MDS code with a linear repair scheme. In contrast, the naive (commonly implemented) repair algorithm for this RS code has bandwidth $k\log (n)$ bits. When the entire field is used as evaluation points, the number of nodes $n$ is much larger than the number of bits per node (which is $O(\log (n))$ ), and so this result holds only when the degree of sub-packetization is small. However, our method applies in any parameter regime, and to illustrate this for high levels of sub-packetization, we give an improved repair scheme for a specific (14,10)-RS code used in the facebook hadoop analytics cluster.

Repairing Reed-Solomon Codes

Binary measurements arise naturally in a variety of statistics and engineering applications. They may be inherent to the problem—for example, in determining the relationship between genetics and the presence or absence of a disease—or they may be a result of extreme quantization. A recent influx of literature has suggested that using prior signal information can greatly improve the ability to reconstruct a signal from binary measurements. This is exemplified by one-bit compressed sensing , which takes the compressed sensing model but assumes that only the sign of each measurement is retained. It has recently been shown that the number of one-bit measurements required for signal estimation mirrors that of unquantized compressed sensing. Indeed, $s$ -sparse signals in $ {\mathbb R}^{n}$ can be estimated (up to normalization) from $\Omega (s \log ~(n/s))$ one-bit measurements. Nevertheless, controlling the precise accuracy of the error estimate remains an open challenge. In this paper, we focus on optimizing the decay of the error as a function of the oversampling factor $\lambda := m/(s \log (n/s))$ , where $m$ is the number of measurements. It is known that the error in reconstructing sparse signals from standard one-bit measurements is bounded below by $\Omega (\lambda ^{-1})$ . Without adjusting the measurement procedure, reducing this polynomial error decay rate is impossible. However, we show that an adaptive choice of the thresholds used for quantization can lower the error rate to $e^{-\Omega (\lambda )}$ . This improves upon guarantees for other methods of adaptive thresholding, such as sigma–delta quantization. We develop a general recursive strategy to achieve this exponential decay and two specific polynomial-time algorithms, which fall into this framework, one based on convex programming and one on hard thresholding. Our work bridges the one-bit compressed sensing model, in which the engineer controls the measurement procedure, to sigma–delta and successive approximation quantization. Moreover, the principle is extendable to signal reconstruction problems in a variety of binary statistical models as well as statistical estimation problems like logistic regression.

/pdf/exponential-decay-of-reconstruction-error-from-binary-3rvo4h0jn8.pdf

Exponential Decay of Reconstruction Error From Binary Measurements of Sparse Signals

Machine learning relies on the assumption that unseen test instances of a classification problem follow the same distribution as observed training data. However, this principle can break down when machine learning is used to make important decisions about the welfare (employment, education, health) of strategic individuals. Knowing information about the classifier, such individuals may manipulate their attributes in order to obtain a better classification outcome. As a result of this behavior---often referred to as gaming---the performance of the classifier may deteriorate sharply. Indeed, gaming is a well-known obstacle for using machine learning methods in practice; in financial policy-making, the problem is widely known as Goodhart's law. In this paper, we formalize the problem, and pursue algorithms for learning classifiers that are robust to gaming. We model classification as a sequential game between a player named "Jury" and a player named "Contestant." Jury designs a classifier, and Contestant receives an input to the classifier, which he may change at some cost. Jury's goal is to achieve high classification accuracy with respect to Contestant's original input and some underlying target classification function. Contestant's goal is to achieve a favorable classification outcome while taking into account the cost of achieving it. For a natural class of cost functions, we obtain computationally efficient learning algorithms which are near-optimal. Surprisingly, our algorithms are efficient even on concept classes that are computationally hard to learn. For general cost functions, designing an approximately optimal strategy-proof classifier, for inverse-polynomial approximation, is NP-hard.

Mary Wootters

Papers

1-Bit matrix completion

Strategic Classification

Repairing Reed-Solomon Codes

Exponential Decay of Reconstruction Error From Binary Measurements of Sparse Signals

Strategic Classification