Showing papers on "Convex optimization published in 2006"

Efficient sparse coding algorithms

Abstract: Sparse coding provides a class of algorithms for finding succinct representations of stimuli; given only unlabeled input data, it discovers basis functions that capture higher-level features in the data. However, finding sparse codes remains a very difficult computational problem. In this paper, we present efficient sparse coding algorithms that are based on iteratively solving two convex optimization problems: an L1-regularized least squares problem and an L2-constrained least squares problem. We propose novel algorithms to solve both of these optimization problems. Our algorithms result in a significant speedup for sparse coding, allowing us to learn larger sparse codes than possible with previously described algorithms. We apply these algorithms to natural images and demonstrate that the inferred sparse codes exhibit end-stopping and non-classical receptive field surround suppression and, therefore, may provide a partial explanation for these two phenomena in V1 neurons.

Just relax: convex programming methods for identifying sparse signals in noise

Abstract: This paper studies a difficult and fundamental problem that arises throughout electrical engineering, applied mathematics, and statistics. Suppose that one forms a short linear combination of elementary signals drawn from a large, fixed collection. Given an observation of the linear combination that has been contaminated with additive noise, the goal is to identify which elementary signals participated and to approximate their coefficients. Although many algorithms have been proposed, there is little theory which guarantees that these algorithms can accurately and efficiently solve the problem. This paper studies a method called convex relaxation, which attempts to recover the ideal sparse signal by solving a convex program. This approach is powerful because the optimization can be completed in polynomial time with standard scientific software. The paper provides general conditions which ensure that convex relaxation succeeds. As evidence of the broad impact of these results, the paper describes how convex relaxation can be used for several concrete signal recovery problems. It also describes applications to channel coding, linear regression, and numerical analysis

Multi-Task Feature Learning

Abstract: We present a method for learning a low-dimensional representation which is shared across a set of multiple related tasks. The method builds upon the well-known 1-norm regularization problem using a new regularizer which controls the number of learned features common for all the tasks. We show that this problem is equivalent to a convex optimization problem and develop an iterative algorithm for solving it. The algorithm has a simple interpretation: it alternately performs a supervised and an unsupervised step, where in the latter step we learn commonacross-tasks representations and in the former step we learn task-specific functions using these representations. We report experiments on a simulated and a real data set which demonstrate that the proposed method dramatically improves the performance relative to learning each task independently. Our algorithm can also be used, as a special case, to simply select – not learn – a few common features across the tasks.

Convexity, Classification, and Risk Bounds

Abstract: Many of the classification algorithms developed in the machine learning literature, including the support vector machine and boosting, can be viewed as minimum contrast methods that minimize a convex surrogate of the 0–1 loss function. The convexity makes these algorithms computationally efficient. The use of a surrogate, however, has statistical consequences that must be balanced against the computational virtues of convexity. To study these issues, we provide a general quantitative relationship between the risk as assessed using the 0–1 loss and the risk as assessed using any nonnegative surrogate loss function. We show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function—that it satisfies a pointwise form of Fisher consistency for classification. The relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications. We also present a refined version of this result in the...

Algorithms for finding global minimizers of image segmentation and denoising models

Abstract: We show how certain nonconvex optimization problems that arise in image processing and computer vision can be restated as convex minimization problems. This allows, in particular, the finding of global minimizers via standard convex minimization schemes.

The scenario approach to robust control design

Abstract: This paper proposes a new probabilistic solution framework for robust control analysis and synthesis problems that can be expressed in the form of minimization of a linear objective subject to convex constraints parameterized by uncertainty terms. This includes the wide class of NP-hard control problems representable by means of parameter-dependent linear matrix inequalities (LMIs). It is shown in this paper that by appropriate sampling of the constraints one obtains a standard convex optimization problem (the scenario problem) whose solution is approximately feasible for the original (usually infinite) set of constraints, i.e., the measure of the set of original constraints that are violated by the scenario solution rapidly decreases to zero as the number of samples is increased. We provide an explicit and efficient bound on the number of samples required to attain a-priori specified levels of probabilistic guarantee of robustness. A rich family of control problems which are in general hard to solve in a deterministically robust sense is therefore amenable to polynomial-time solution, if robustness is intended in the proposed risk-adjusted sense.

Convex Approximations of Chance Constrained Programs

Abstract: We consider a chance constrained problem, where one seeks to minimize a convex objective over solutions satisfying, with a given close to one probability, a system of randomly perturbed convex constraints. This problem may happen to be computationally intractable; our goal is to build its computationally tractable approximation, i.e., an efficiently solvable deterministic optimization program with the feasible set contained in the chance constrained problem. We construct a general class of such convex conservative approximations of the corresponding chance constrained problem. Moreover, under the assumptions that the constraints are affine in the perturbations and the entries in the perturbation vector are independent-of-each-other random variables, we build a large deviation-type approximation, referred to as “Bernstein approximation,” of the chance constrained problem. This approximation is convex and efficiently solvable. We propose a simulation-based scheme for bounding the optimal value in the chance constrained problem and report numerical experiments aimed at comparing the Bernstein and well-known scenario approximation approaches. Finally, we extend our construction to the case of ambiguous chance constrained problems, where the random perturbations are independent with the collection of distributions known to belong to a given convex compact set rather than to be known exactly, while the chance constraint should be satisfied for every distribution given by this set.

Disciplined Convex Programming

Abstract: A new methodology for constructing convex optimization models called disciplined convex programming is introduced. The methodology enforces a set of conventions upon the models constructed, in turn allowing much of the work required to analyze and solve the models to be automated.

H/sub /spl infin// control for networked systems with random communication delays

Abstract: This note is concerned with a new controller design problem for networked systems with random communication delays. Two kinds of random delays are simultaneously considered: i) from the controller to the plant, and ii) from the sensor to the controller, via a limited bandwidth communication channel. The random delays are modeled as a linear function of the stochastic variable satisfying Bernoulli random binary distribution. The observer-based controller is designed to exponentially stabilize the networked system in the sense of mean square, and also achieve the prescribed H/sub /spl infin// disturbance attenuation level. The addressed controller design problem is transformed to an auxiliary convex optimization problem, which can be solved by a linear matrix inequality (LMI) approach. An illustrative example is provided to show the applicability of the proposed method.

Radial distribution load flow using conic programming

Abstract: This paper shows that the load flow problem of a radial distribution system can be modeled as a convex optimization problem, particularly a conic program. The implications of the conic programming formulation are threefold. First, the solution of the distribution load flow problem can be obtained in polynomial time using interior-point methods. Second, numerical ill-conditioning can be automatically alleviated by the use of scaling in the interior-point algorithm. Third, the conic formulation facilitates the inclusion of the distribution power flow equations in radial system optimization problems. A state-of-the-art implementation of an interior-point method for conic programming is used to obtain the solution of nine different distribution systems. Comparisons are carried out with a previously published radial load flow program by R. Cespedes

Cross-Layer Congestion Control, Routing and Scheduling Design in Ad Hoc Wireless Networks

Abstract: This paper considers jointly optimal design of crosslayer congestion control, routing and scheduling for ad hoc wireless networks. We first formulate the rate constraint and scheduling constraint using multicommodity flow variables, and formulate resource allocation in networks with fixed wireless channels (or single-rate wireless devices that can mask channel variations) as a utility maximization problem with these constraints. By dual decomposition, the resource allocation problem naturally decomposes into three subproblems: congestion control, routing and scheduling that interact through congestion price. The global convergence property of this algorithm is proved. We next extend the dual algorithm to handle networks with timevarying channels and adaptive multi-rate devices. The stability of the resulting system is established, and its performance is characterized with respect to an ideal reference system which has the best feasible rate region at link layer. We then generalize the aforementioned results to a general model of queueing network served by a set of interdependent parallel servers with time-varying service capabilities, which models many design problems in communication networks. We show that for a general convex optimization problem where a subset of variables lie in a polytope and the rest in a convex set, the dual-based algorithm remains stable and optimal when the constraint set is modulated by an irreducible finite-state Markov chain. This paper thus presents a step toward a systematic way to carry out cross-layer design in the framework of “layering as optimization decomposition” for time-varying channel models.

An introduction to convex optimization for communications and signal processing

Abstract: Convex optimization methods are widely used in the design and analysis of communication systems and signal processing algorithms. This tutorial surveys some of recent progress in this area. The tutorial contains two parts. The first part gives a survey of basic concepts and main techniques in convex optimization. Special emphasis is placed on a class of conic optimization problems, including second-order cone programming and semidefinite programming. The second half of the survey gives several examples of the application of conic programming to communication problems. We give an interpretation of Lagrangian duality in a multiuser multi-antenna communication problem; we illustrate the role of semidefinite relaxation in multiuser detection problems; we review methods to formulate robust optimization problems via second-order cone programming techniques

EfficientL 1 regularized logistic regression

Abstract: L1 regularized logistic regression is now a workhorse of machine learning: it is widely used for many classification problems, particularly ones with many features. L1 regularized logistic regression requires solving a convex optimization problem. However, standard algorithms for solving convex optimization problems do not scale well enough to handle the large datasets encountered in many practical settings. In this paper, we propose an efficient algorithm for L1 regularized logistic regression. Our algorithm iteratively approximates the objective function by a quadratic approximation at the current point, while maintaining the L1 constraint. In each iteration, it uses the efficient LARS (Least Angle Regression) algorithm to solve the resulting L1 constrained quadratic optimization problem. Our theoretical results show that our algorithm is guaranteed to converge to the global optimum. Our experiments show that our algorithm significantly outperforms standard algorithms for solving convex optimization problems. Moreover, our algorithm outperforms four previously published algorithms that were specifically designed to solve the L1 regularized logistic regression problem.

Interior Gradient and Proximal Methods for Convex and Conic Optimization

Abstract: Interior gradient (subgradient) and proximal methods for convex constrained minimization have been much studied, in particular for optimization problems over the nonnegative octant. These methods are using non-Euclidean projections and proximal distance functions to exploit the geometry of the constraints. In this paper, we identify a simple mechanism that allows us to derive global convergence results of the produced iterates as well as improved global rates of convergence estimates for a wide class of such methods, and with more general convex constraints. Our results are illustrated with many applications and examples, including some new explicit and simple algorithms for conic optimization problems. In particular, we derive a class of interior gradient algorithms which exhibits an $O(k^{-2})$ global convergence rate estimate.

A robust maximin approach for MIMO communications with imperfect channel state information based on convex optimization

Abstract: This paper considers a wireless communication system with multiple transmit and receive antennas, i.e., a multiple-input-multiple-output (MIMO) channel. The objective is to design the transmitter according to an imperfect channel estimate, where the errors are explicitly taken into account to obtain a robust design under the maximin or worst case philosophy. The robust transmission scheme is composed of an orthogonal space-time block code (OSTBC), whose outputs are transmitted through the eigenmodes of the channel estimate with an appropriate power allocation among them. At the receiver, the signal is detected assuming a perfect channel knowledge. The optimization problem corresponding to the design of the power allocation among the estimated eigenmodes, whose goal is the maximization of the signal-to-noise ratio (SNR), is transformed to a simple convex problem that can be easily solved. Different sources of errors are considered in the channel estimate, such as the Gaussian noise from the estimation process and the errors from the quantization of the channel estimate, among others. For the case of Gaussian noise, the robust power allocation admits a closed-form expression. Finally, the benefits of the proposed design are evaluated and compared with the pure OSTBC and nonrobust approaches.

An introduction to nonlinear dimensionality reduction by maximum variance unfolding

Abstract: Many problems in AI are simplified by clever representations of sensory or symbolic input. How to discover such representations automatically, from large amounts of unlabeled data, remains a fundamental challenge. The goal of statistical methods for dimensionality reduction is to detect and discover low dimensional structure in high dimensional data. In this paper, we review a recently proposed algorithm-- maximum, variance unfolding--for learning faithful low dimensional representations of high dimensional data. The algorithm relies on modem tools in convex optimization that are proving increasingly useful in many areas of machine learning.

Sparse reconstruction by convex relaxation: Fourier and Gaussian measurements

Abstract: This paper proves best known guarantees for exact reconstruction of a sparse signal f from few non-adaptive universal linear measurements. We consider Fourier measurements (random sample of frequencies of f) and random Gaussian measurements. The method for reconstruction that has recently gained momentum in the sparse approximation theory is to relax this highly non-convex problem to a convex problem, and then solve it as a linear program. What are best guarantees for the reconstruction problem to be equivalent to its convex relaxation is an open question. Recent work shows that the number of measurements k(r,n) needed to exactly reconstruct any r-sparse signal f of length n from its linear measurements with convex relaxation is usually O(r poly log (n)). However, known guarantees involve huge constants, in spite of very good performance of the algorithms in practice. In attempt to reconcile theory with practice, we prove the first guarantees for universal measurements (i.e. which work for all sparse functions) with reasonable constants. For Gaussian measurements, k(r,n) lsim 11.7 r [1.5 + log(n/r)], which is optimal up to constants. For Fourier measurements, we prove the best known bound k(r, n) = O(r log(n) middot log2(r) log(r log n)), which is optimal within the log log n and log3 r factors. Our arguments are based on the technique of geometric functional analysis and probability in Banach spaces.

Efficient Structure Learning of Markov Networks using L_1-Regularization

Abstract: Markov networks are commonly used in a wide variety of applications, ranging from computer vision, to natural language, to computational biology. In most current applications, even those that rely heavily on learned models, the structure of the Markov network is constructed by hand, due to the lack of effective algorithms for learning Markov network structure from data. In this paper, we provide a computationally efficient method for learning Markov network structure from data. Our method is based on the use of L1 regularization on the weights of the log-linear model, which has the effect of biasing the model towards solutions where many of the parameters are zero. This formulation converts the Markov network learning problem into a convex optimization problem in a continuous space, which can be solved using efficient gradient methods. A key issue in this setting is the (unavoidable) use of approximate inference, which can lead to errors in the gradient computation when the network structure is dense. Thus, we explore the use of different feature introduction schemes and compare their performance. We provide results for our method on synthetic data, and on two real world data sets: pixel values in the MNIST data, and genetic sequence variations in the human HapMap data. We show that our L1 -based method achieves considerably higher generalization performance than the more standard L2-based method (a Gaussian parameter prior) or pure maximum-likelihood learning. We also show that we can learn MRF network structure at a computational cost that is not much greater than learning parameters alone, demonstrating the existence of a feasible method for this important problem.

Cross-Layer Design for Lifetime Maximization in Interference-Limited Wireless Sensor Networks

Abstract: We consider the joint optimal design of the physical, medium access control (MAC), and routing layers to maximize the lifetime of energy-constrained wireless sensor networks. The problem of computing lifetime-optimal routing flow, link schedule, and link transmission powers for all active time slots is formulated as a non-linear optimization problem. We first restrict the link schedules to the class of interference-free time division multiple access (TDMA) schedules. In this special case, we formulate the optimization problem as a mixed integerconvex program, which can be solved using standard techniques. Moreover, when the slots lengths are variable, the optimization problem is convex and can be solved efficiently and exactly using interior point methods. For general non-orthogonal link schedules, we propose an iterative algorithm that alternates between adaptive link scheduling and computation of optimal link rates and transmission powers for a fixed link schedule. The performance of this algorithm is compared to other design approaches for several network topologies. The results illustrate the advantages of load balancing, multihop routing, frequency reuse, and interference mitigation in increasing the lifetime of energy-constrained networks. We also briefly discuss computational approaches to extend this algorithm to large networks

Convex optimization of graph Laplacian eigenvalues

Abstract: We consider the problem of choosing the edge weights of an undirected graph so as to maximize or minimize some function of the eigenvalues of the associated Laplacian matrix, subject to some constraints on the weights, such as nonnegativity, or a given total value. In many interesting cases this problem is convex, i.e., it involves minimizing a convex function (or maximizing a concave function) over a convex set. This allows us to give simple necessary and sufficient optimality conditions, derive interesting dual problems, find analytical solutions in some cases, and efficiently compute numerical solutions in all cases. In this overview we briefly describe some more specific cases of this general problem, which have been addressed in a series of recent papers. � Fastest mixing Markov chain. Find edge transition probabilities that give the fastest mixing (symmetric, discrete-time) Markov chain on the graph. � Fastest mixing Markov process. Find the edge transition rates that give the fastest mixing (symmetric, continuous-time) Markov process on the graph. � Absolute algebraic connectivity. Find edge weights that maximize the algebraic connectivity of the graph (i.e., the smallest positive eigenvalue of its Laplacian matrix). The optimal value is called the absolute algebraic connectivity by Fiedler. � Minimum total effective resistance. Find edge weights that minimize the total effective resistance of the graph. This is same as minimizing the average commute time from any node to any other, in the associated Markov chain. � Fastest linear averaging. Find weights in a distributed averaging network that yield fastest convergence. � Least steady-state mean-square deviation. Find weights in a distributed averaging network, driven by random noise, that minimizes the steady-state mean-square deviation of the node values.

Design of variable–stiffness laminates using lamination parameters

Abstract: Benefits of the directional properties of fiber reinforced composites could be fully utilized by proper placement of the fibers in their optimal spatial orientations. This paper investigates the optimal design of fiber reinforced rectangular composite plates for minimum compliance. The classical minimum compliance design problem is formulated in the lamination parameters space. The use of lamination parameters guarantees that the obtained solutions represent the best possible performance of the structure since there are no restrictions on the possible lamination sequence. Two types of designs are considered: constant–stiffness designs where the lamination parameters are constant over the plate domain, and variable-stiffness designs where the laminations parameters are allowed to vary in a continuous manner over the domain. The optimality conditions for the problem are reformulated as a local design rule. The local design rule assumes the form of a convex optimization problem and is solved using a feasible sequential quadratic programming. Optimal designs for both constant and variable–stiffness plates are obtained. It is shown that significant improvements in stiffness can be gained by using variable–stiffness design.

Robust stability analysis of generalized neural networks with discrete and distributed time delays

Abstract: This paper is concerned with the problem of robust global stability analysis for generalized neural networks (GNNs) with both discrete and distributed delays. The parameter uncertainties are assumed to be time-invariant and bounded, and belong to given compact sets. The existence of the equilibrium point is first proved under mild conditions, assuming neither differentiability nor strict monotonicity for the activation function. Then, by employing a Lyapunov–Krasovskii functional, the addressed stability analysis problem is converted into a convex optimization problem, and a linear matrix inequality (LMI) approach is utilized to establish the sufficient conditions for the globally robust stability for the GNNs, with and without parameter uncertainties. These conditions can be readily checked by utilizing the Matlab LMI toolbox. A numerical example is provided to demonstrate the usefulness of the proposed global stability condition.