We will review some of the major results in random graphs and some of the more challenging open problems. We will cover algorithmic and structural questions. We will touch on newer models, including those related to the WWW.

Random graphs

Many ``real-world'' networks are clearly defined while most ``social'' networks are to some extent subjective. Indeed, the accuracy of empirically-determined social networks is a question of some concern because individuals may have distinct perceptions of what constitutes a social link. One unambiguous type of connection is sexual contact. Here we analyze data on the sexual behavior of a random sample of individuals, and find that the cumulative distributions of the number of sexual partners during the twelve months prior to the survey decays as a power law with similar exponents $\alpha \approx 2.4$ for females and males. The scale-free nature of the web of human sexual contacts suggests that strategic interventions aimed at preventing the spread of sexually-transmitted diseases may be the most efficient approach.

The Web of Human Sexual Contacts

In this paper, we analyze the convergence of the alternating direction method of multipliers (ADMM) for minimizing a nonconvex and possibly nonsmooth objective function, $$\phi (x_0,\ldots ,x_p,y)$$
 , subject to coupled linear equality constraints. Our ADMM updates each of the primal variables $$x_0,\ldots ,x_p,y$$
 , followed by updating the dual variable. We separate the variable y from $$x_i$$
 ’s as it has a special role in our analysis. The developed convergence guarantee covers a variety of nonconvex functions such as piecewise linear functions, $$\ell _q$$
 quasi-norm, Schatten-q quasi-norm (
 $$0<q<1$$
 ), minimax concave penalty (MCP), and smoothly clipped absolute deviation penalty. It also allows nonconvex constraints such as compact manifolds (e.g., spherical, Stiefel, and Grassman manifolds) and linear complementarity constraints. Also, the $$x_0$$
 -block can be almost any lower semi-continuous function. By applying our analysis, we show, for the first time, that several ADMM algorithms applied to solve nonconvex models in statistical learning, optimization on manifold, and matrix decomposition are guaranteed to converge. Our results provide sufficient conditions for ADMM to converge on (convex or nonconvex) monotropic programs with three or more blocks, as they are special cases of our model. ADMM has been regarded as a variant to the augmented Lagrangian method (ALM). We present a simple example to illustrate how ADMM converges but ALM diverges with bounded penalty parameter $$\beta $$
 . Indicated by this example and other analysis in this paper, ADMM might be a better choice than ALM for some nonconvex nonsmooth problems, because ADMM is not only easier to implement, it is also more likely to converge for the concerned scenarios.

http://www.optimization-online.org/DB_FILE/2015/11/5222.pdf

Global Convergence of ADMM in Nonconvex Nonsmooth Optimization

We consider the problem of estimating multiple related Gaussian graphical models from a high-dimensional data set with observations belonging to distinct classes. We propose the joint graphical lasso, which borrows strength across the classes in order to estimate multiple graphical models that share certain characteristics, such as the locations or weights of nonzero edges. Our approach is based upon maximizing a penalized log likelihood. We employ generalized fused lasso or group lasso penalties, and implement a fast ADMM algorithm to solve the corresponding convex optimization problems. The performance of the proposed method is illustrated through simulated and real data examples.

/pdf/the-joint-graphical-lasso-for-inverse-covariance-estimation-2ar558in3i.pdf

The joint graphical lasso for inverse covariance estimation across multiple classes

We consider the problem of estimating the frequency components of a mixture of s complex sinusoids from a random subset of n regularly spaced samples. Unlike previous work in compressed sensing, the frequencies are not assumed to lie on a grid, but can assume any values in the normalized frequency domain [0,1]. We propose an atomic norm minimization approach to exactly recover the unobserved samples. We reformulate this atomic norm minimization as an exact semidefinite program. Even with this continuous dictionary, we show that most sampling sets of size O(s log s log n) are sufficient to guarantee the exact frequency estimation with high probability, provided the frequencies are well separated. Numerical experiments are performed to illustrate the effectiveness of the proposed method.

Compressed Sensing off the Grid

The problem of minimizing the rank of a matrix subject to affine constraints has applications in several areas including machine learning, and is known to be NP-hard. A tractable relaxation for this problem is nuclear norm (or trace norm) minimization, which is guaranteed to find the minimum rank matrix under suitable assumptions. In this paper, we propose a family of Iterative Reweighted Least Squares algorithms IRLS-p (with 0 ≤ p ≤ 1), as a computationally efficient way to improve over the performance of nuclear norm minimization. The algorithms can be viewed as (locally) minimizing certain smooth approximations to the rank function. When p = 1, we give theoretical guarantees similar to those for nuclear norm minimization, that is, recovery of low-rank matrices under certain assumptions on the operator defining the constraints. For p < 1, IRLS-p shows better empirical performance in terms of recovering low-rank matrices than nuclear norm minimization. We provide an efficient implementation for IRLS-p, and also present a related family of algorithms, sIRLS-p. These algorithms exhibit competitive run times and improved recovery when compared to existing algorithms for random instances of the matrix completion problem, as well as on the MovieLens movie recommendation data set.

/pdf/iterative-reweighted-algorithms-for-matrix-rank-minimization-3g8j07x35l.pdf

Iterative reweighted algorithms for matrix rank minimization

We consider the problem of estimating high-dimensional Gaussian graphical models corresponding to a single set of variables under several distinct conditions. This problem is motivated by the task of recovering transcriptional regulatory networks on the basis of gene expression data containing heterogeneous samples, such as different disease states, multiple species, or different developmental stages. We assume that most aspects of the conditional dependence networks are shared, but that there are some structured differences between them. Rather than assuming that similarities and differences between networks are driven by individual edges, we take a node-based approach, which in many cases provides a more intuitive interpretation of the network differences. We consider estimation under two distinct assumptions: (1) differences between the K networks are due to individual nodes that are perturbed across conditions, or (2) similarities among the K networks are due to the presence of common hub nodes that are shared across all K networks. Using a row-column overlap norm penalty function, we formulate two convex optimization problems that correspond to these two assumptions. We solve these problems using an alternating direction method of multipliers algorithm, and we derive a set of necessary and sufficient conditions that allows us to decompose the problem into independent subproblems so that our algorithm can be scaled to high-dimensional settings. Our proposal is illustrated on synthetic data, a webpage data set, and a brain cancer gene expression data set.

/pdf/node-based-learning-of-multiple-gaussian-graphical-models-4f2lj5brgn.pdf

Node-based learning of multiple Gaussian graphical models

The matrix rank minimization problem consists of finding a matrix of minimum rank that satisfies given convex constraints. It is NP-hard in general and has applications in control, system identification, and machine learning. Reweighted trace minimization has been considered as an iterative heuristic for this problem. In this paper, we analyze the convergence of this iterative heuristic, showing that the difference between successive iterates tends to zero. Then, after reformulating the heuristic as reweighted nuclear norm minimization, we propose an efficient gradient-based implementation that takes advantage of the new formulation and opens the way to solving large-scale problems. We apply this algorithm to the problem of low-order system identification from input-output data. Numerical examples demonstrate that the reweighted nuclear norm minimization makes model order selection easier and results in lower order models compared to nuclear norm minimization without weights.

/pdf/reweighted-nuclear-norm-minimization-with-application-to-29ptmfsz8c.pdf

Reweighted nuclear norm minimization with application to system identification

The classical compressed sensing problem is to find the sparsest solution to an underdetermined system of linear equations. A good convex approximation to this problem is to minimize the l 1 norm subject to affine constraints. The Iterative Reweighted Least Squares (IRLSp) algorithm (0 &#60; p ≤ 1), has been proposed as a method to solve the l p (p ≤ 1) minimization problem with affine constraints. Recently Chartrand et al observed that IRLS-p with p &#60; 1 has better empirical performance than l 1 minimization, and Daubechies et al gave ‘local’ linear and super-linear convergence results for IRLS-p with p = 1 and p &#60; 1 respectively. In this paper we extend IRLS-p as a family of algorithms for the matrix rank minimization problem and we also present a related family of algorithms, sIRLS-p. We present guarantees on recovery of low-rank matrices for IRLS-1 under the Null Space Property (NSP). We also establish that the difference between the successive iterates of IRLS-p and sIRLS-p converges to zero and that the IRLS-0 algorithm converges to the stationary point of a non-convex rank-surrogate minimization problem. On the numerical side, we give a few efficient implementations for IRLS-0 and demonstrate that both sIRLS-0 and IRLS-0 perform better than algorithms such as Singular Value Thresholding (SVT) on a range of ‘hard’ problems (where the ratio of number of degrees of freedom in the variable to the number of measurements is large). We also observe that sIRLS-0 performs better than Iterative Hard Thresholding algorithm (IHT) when there is no apriori information on the low rank solution.

/pdf/iterative-reweighted-least-squares-for-matrix-rank-446b8frfu8.pdf

Iterative reweighted least squares for matrix rank minimization

Recovering sparse vectors and low-rank matrices from noisy linear measurements has been the focus of much recent research. Various reconstruction algorithms have been studied, including $\ell_1$ and nuclear norm minimization as well as $\ell_p$ minimization with $p<1$. These algorithms are known to succeed if certain conditions on the measurement map are satisfied. Proofs of robust recovery for matrices have so far been much more involved than in the vector case. 
In this paper, we show how several robust classes of recovery conditions can be extended from vectors to matrices in a simple and transparent way, leading to the best known restricted isometry and nullspace conditions for matrix recovery. Our results rely on the ability to "vectorize" matrices through the use of a key singular value inequality.

Karthik Mohan

Papers

Iterative reweighted algorithms for matrix rank minimization

Node-based learning of multiple Gaussian graphical models

Reweighted nuclear norm minimization with application to system identification

Iterative reweighted least squares for matrix rank minimization

A Simplified Approach to Recovery Conditions for Low Rank Matrices