Graph Implementations for Nonsmooth
Convex Programs
Michael C. Grant I and Stephen P. Boyd 2
1 Stanford University
mcgrant9 ord. edu
2 Stanford University
boydOstanf ord. edu
Summary. We describe graph implementations, a generic method for representing
a convex function via its epigraph, described in a disciplined convex programming
framework. This simple and natural idea allows a very wide variety of smooth and
nonsmooth convex programs to be easily specified and efficiently solved, using interior-
point methods for smooth or cone convex programs.
Keywords: Convex optimization, nonsmooth optimization, disciplined convex pro-
gramming, optimization modeling languages, semidefinite programming, second-order
cone programming, conic optimization, nondifferentiable functions.
1 Introduction
It is well known that convex programs have many attractive properties, includ-
ing the proven existence of efficient methods to solve them. What is not as
widely appreciated, however, is that nonsmooth convex programs--i.e., models
with nondifferentiable constraints or objectives--can, in theory, be solved just
as efficiently as their smooth counterparts. But here, as is often the case, theory
and practice do not coincide. Methods that are theoretically efficient for general
nondifferentiable problems, such as the ellipsoid method [3], are notoriously slow
in practice.
In contrast, there are many solvers available for smooth convex programs, as
well as for certain standard forms such as semidefinite programs (SDPs), that are
efficient in both theory and practice; e.g., [13, 16, 17, 18]. These solvers can often
be used to solve a general nonsmooth problem as well--not directly of course,
but by first transforming it into an equivalent form supported by the solver. The
equivalent problem is usually larger than the original, but the superior efficiency
of the solver more than compensates for the increase in size, especially if problem
structure is taken into account.
The transformation approach dates back to the very first days of linear pro-
graming [7]. It is usually taught as a collection of tricks that a modeler can use
to (hopefully) reformulate problems by hand. The versatility of the approach,
of course, depends upon the variety of transformations known by the modeler.
But while some transformations are fairly obvious and widely known, others are
V.D. Blondel et al. (Eds.) Recent Advances in Learning and Control, LNCIS 371, pp. 95-110, 2008.
springerlink.com (~) Springer-Verlag Berlin Heidelberg 2008
96 M.C. Grant and S.P. Boyd
neither obvious nor well known, even to some experts in convex optimization.
Furthermore, even if a transformation is identified, the reformulation process is
often time consuming and error prone, for both experts and applications-oriented
modelers alike.
We propose to enable modeling frameworks to largely
automate
the process
of identifying and applying these transformations, so that a much wider variety
of models--smooth and nonsmooth alike--can be both easily specified and era-
ciently solved. Our approach depends upon two distinct but interrelated devel-
opments. The first is a methodology for constructing convex optimization models
called
disciplined convex programming.
The methodology imposes a set of rules
or conventions that must be followed when constructing convex programs. The
rules are simple and teachable, drawn from basic principles of convex analy-
sis, and follow the practices of those who regularly use convex optimization.
Conforming problems are called, appropriately,
disciplined convex programs,
or
DCPs.
The DCP ruleset does not limit generality, but it does require that the
modeler explicitly provide just enough structure to allow further analysis and
solution of a problem to be automated.
The second development is a new method for defining or implementing a func-
tion in a modeling framework, as as the optimal value of a parameterized convex
program (specifically, a DCP). We call such a function definition a
graph ira-
plementation
because it exploits the relationship between convex and concave
functions and their epigraphs and hypographs, respectively. A graph implemen-
tation encapsulates a method for transforming instances of a specific function
in a constraint or objective into a form compatible with the underlying solver's
standard form. The conditions imposed by the DCP ruleset ensure that these
transformations always preserve equivalence and convexity. The most significant
benefit of graph implementations is their ability to efficiently implement non-
differentiable functions. But in fact, graph implementations can also be used
to implement many
smooth
functions as well when the target standard form is
nonsmooth
(e.g.,
an SDP).
We have created a modeling framework called cvx [8] that supports disciplined
convex programming and graph implementations, cvx uses the object-oriented
features of MATLAB~to turn it into an optimization modeling language: opti-
mization variables can be declared and constraints and objectives specified using
natural MATLAB| syntax, cvx verifies compliance with the DCP ruleset, trans-
forms conforming models to solvable form, calls an appropriate numerical solver,
and translates the numerical results back to the original problem--all without
user intervention. The framework includes a large library of common convex and
concave functions, both smooth and nonsmooth, and more can be added.
To an applications-oriented user, the conceptual model presented by cvx is
very simple: cvx solves any problem (up to some practical size limits, of course)
constructed according to the DCP ruleset from functions found in the cvx library.
The modeler need not know what transformations are taking place, or even
that a transformation is necessary. That is, graph implementations are entirely
opaque
or hidden from a standard cvx user. On the other hand, expert users
Graph Implementations for Nonsmooth Convex Programs 97
can use graph implementations to add new transformations to the system that
general users can exploit simply by calling the new functions in their models.
This division of labor allows modelers to focus on building convex programs, and
experts to focus on solving them.
In what follows, we will describe disciplined convex programming and graph
implementations in detail. Both are abstract, language-independent concepts;
nevertheless, it will be easier to explain both using examples from an actual
modeling framework such as cvx. So we begin by i~roducing cvx with a few
simple examples. A basic familiarity with MATLAB~is assumed throughout.
2 A Brief Introduction to cvx
To begin, consider the simple linear program
minimize
c T x
subject to
Ax < b,
(1)
with variable x C R n and data
A C R mxn, b C R m,
and c C R n. The following
(MATLAB@/cvx) code generates and solves a random instance of (1)"
m = 16; n = 8;
A = randn(m,n) ; b = randn(m,1) ; c = randn(n,1);
cvx_begin
variable x (n)
minimize( c' * x )
subject to
A.x<=b;
cvx end
The indentation is purely for stylistic reasons and is optional. The code is rela-
tively self-explanatory, but a few notes are in order:
9 The cvx begin and cvx end commands mark the beginning and end, re-
spectively, of any cvx model.
9 Variables must be declared before their first use. For example, the variable
statement above declares x to be a vector of length n.
9 The subject to statement is optional--it is provided by cvx only to make
models more readable and has no mathematical effect.
9 Objectives and constraints may be placed in any order.
When cvx end is reached, cvx will complete the conversion of the model to
solvable form and call the underlying numerical solver. It will replace the
MATLAB| variable x, which up to that point was a special cvx variable object,
with a numeric vector representing an optimal value.
Now consider a norm minimization problem with box constraints:
minimize
IIAy- bll 2
(2)
subject to g __ y _ u
98 M.C. Grant and S.P. Boyd
The following cvx/MATLAB@ code constructs and solves a version of (2),
reusing A and b from above and generating random bounds:
1 = -abs(randn(n,2)) ; u = +abs(randn(n,2)) ;
cvx_begin
variable y(n)
minimize(norm(A,y-b,2) )
subject
to
y<=u;
y>=l;
cvx_end
It is well known that (2) can be reformulated as a (convex) quadratic program
(QP) or a second-order cone program (SOCP). (cvx, in fact, converts this prob-
lem to an SOCP.) The transformation in this case is not particularly complicated;
still, it is nice to have it completely automated.
cvx supports a variety of other norms and penalties for this model simply by
replacing the objective function; for example:
minimize(norm(A,y-b,2) + O. l,norm(y, I) )
minimize(norm(A,y-b,3.5) )
minimize(sum(huber(A,y-b)) )
All of these examples can be reformulated as SOCPs as well. Here, however, the
transformations are not at all obvious, even to experts; and in all three cases
working out the transformation by hand would be tedious and error prone, cvx,
however, can solve all three problems automatically.
As a final example, consider the task of determining the minimum volume
ellipsoid (also known as the LSwner-John ellipsoid) g containing a finite set of
points Zl, z2, 9 9 9 zn c R d"
minimize vol(g) (3)
subject tozicg,
i-1,...,n.
The parameterization we will use for g is
-~ {u I IIPu + qll2 -< 1 }, (4)
where
(P, q) E R dxd
x R d, and P is symmetric positive semidefinite. In this case,
vol(g) is proportional to det(P -1) (see [6, w With this parametrization we
can cast the problem above as
minimize det P-1
subject tollPzi+ql[2<l, i-l,2,...,n, (5)
with variables
P - pT E a dxd
and q C R d. We have written the objective
informally as det P- 1; a more precise description is faet_inv (P), where
A [det(P) -1
P - pT ~_ 0
f~_~v(P)
+oc otherwise. (6)
Graph Implementations for Nonsmooth Convex Programs 99
This function implicitly constrains P to be symmetric and positive definite. The
function faet_inv is convex, so the,problem above is a convex problem.
The following cvx/MATLAB~ code generates a random set of points and
computes the optimal ellipsoid by solving (5)"
d=2;
z = randn(d,n) ;
cvx_begin
variables P(d,d) q(d)
minimize (det_inv(P) )
subject to
for i = i 9 n,
norm( P,z(',i)+q,2 ) <= I;
end
cvx_end
The function det_inv represents faet_inv('), including the implicit constraint that
its argument be symmetric and positive definite. It is known that this problem
can be cast as a semidefinite program (SDP), but the required conversion is
quite complicated. Fortunately, that conversion is buried inside cvx's definition
of det_inv and performed automatically.
This is, of course, a considerably abbreviated introduction to cvx, intended
only to give the reader an idea of the basic syntax and structure of cvx models.
The reader is encouraged to read the user's guide [8] for a more thorough treat-
ment, or to download the software and try it. The examples presented here can
be entered exactly as listed.
3 Disciplined Convex Programming
Disciplined convex programming was first named and described by Grant, Boyd,
and Ye in [9] and Grant in [10]. It was modeled on the methods used by those who
regularly construct convex optimization models. Such modelers do not simply
construct arbitrary nonlinear programs and attempt to verify convexity after the
fact; rather, they begin with a mental library of functions and sets with known
geometries, and combine them in ways which convex analysis guarantees will
preserve convexity.
Disciplined convex programming is an attempt to formalize and this practice
and codify its techniques. It consists of two key components:
9 an atom library--a collection of functions or sets with known properties of
curvature (convexity and concavity) and monotonicity; and
9 the DCP ruleset--a finite enumeration of ways in which atoms may be com-
bined in objectives and constraints while preserving convexity.
The rules are drawn from basic principles of convex analysis, and are easy to
learn, once you have had an exposure to convex analysis and convex optimization.
They constitute a set of sufficient but not necessary conditions for convexity,