The formalism of probabilistic graphical models provides a unifying framework for capturing complex dependencies among random variables, and building large-scale multivariate statistical models. Graphical models have become a focus of research in many statistical, computational and mathematical fields, including bioinformatics, communication theory, statistical physics, combinatorial optimization, signal and image processing, information retrieval and statistical machine learning. Many problems that arise in specific instances — including the key problems of computing marginals and modes of probability distributions — are best studied in the general setting. Working with exponential family representations, and exploiting the conjugate duality between the cumulant function and the entropy for exponential families, we develop general variational representations of the problems of computing likelihoods, marginal probabilities and most probable configurations. We describe how a wide variety of algorithms — among them sum-product, cluster variational methods, expectation-propagation, mean field methods, max-product and linear programming relaxation, as well as conic programming relaxations — can all be understood in terms of exact or approximate forms of these variational representations. The variational approach provides a complementary alternative to Markov chain Monte Carlo as a general source of approximation methods for inference in large-scale statistical models.

/pdf/graphical-models-exponential-families-and-variational-3i6dwlfta1.pdf

Graphical Models, Exponential Families, and Variational Inference

In the first part of this thesis, we introduce a specific class of Linear Matrix Inequalities (LMI) whose optimal solution can be characterized exactly. This family corresponds to the case where the associated linear operator maps the cone of positive semidefinite matrices onto itself. In this case, the optimal value equals the spectral radius of the operator. It is shown that some rank minimization problems, as well as generalizations of the structured singular value ($mu$) LMIs, have exactly this property.

In the same spirit of exploiting structure to achieve computational efficiency, an algorithm for the numerical solution of a special class of frequency-dependent LMIs is presented. These optimization problems arise from robustness analysis questions, via the Kalman-Yakubovich-Popov lemma. The procedure is an outer approximation method based on the algorithms used in the computation of hinf norms for linear, time invariant systems. The result is especially useful for systems with large state dimension.

The other main contribution in this thesis is the formulation of a convex optimization framework for semialgebraic problems, i.e., those that can be expressed by polynomial equalities and inequalities. The key element is the interaction of concepts in real algebraic geometry (Positivstellensatz) and semidefinite programming.

To this end, an LMI formulation for the sums of squares decomposition for multivariable polynomials is presented. Based on this, it is shown how to construct sufficient Positivstellensatz-based convex tests to prove that certain sets are empty. Among other applications, this leads to a nonlinear extension of many LMI based results in uncertain linear system analysis.

Within the same framework, we develop stronger criteria for matrix copositivity, and generalizations of the well-known standard semidefinite relaxations for quadratic programming.

Some applications to new and previously studied problems are presented. A few examples are Lyapunov function computation, robust bifurcation analysis, structured singular values, etc. It is shown that the proposed methods allow for improved solutions for very diverse questions in continuous and combinatorial optimization.

Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization

A hierarchy of convex relaxations for semialgebraic problems is introduced. For questions reducible to a finite number of polynomial equalities and inequalities, it is shown how to construct a complete family of polynomially sized semidefinite programming conditions that prove infeasibility. The main tools employed are a semidefinite programming formulation of the sum of squares decomposition for multivariate polynomials, and some results from real algebraic geometry. The techniques provide a constructive approach for finding bounded degree solutions to the Positivstellensatz, and are illustrated with examples from diverse application fields.

/pdf/semidefinite-programming-relaxations-for-semialgebraic-54kejoyrbk.pdf

Semidefinite programming relaxations for semialgebraic problems

Constraint programming is a powerful paradigm for solving combinatorial search problems that draws on a wide range of techniques from artificial intelligence, computer science, databases, programming languages, and operations research. Constraint programming is currently applied with success to many domains, such as scheduling, planning, vehicle routing, configuration, networks, and bioinformatics.

The aim of this handbook is to capture the full breadth and depth of the constraint programming field and to be encyclopedic in its scope and coverage. While there are several excellent books on constraint programming, such books necessarily focus on the main notions and techniques and cannot cover also extensions, applications, and languages. The handbook gives a reasonably complete coverage of all these lines of work, based on constraint programming, so that a reader can have a rather precise idea of the whole field and its potential. Of course each line of work is dealt with in a survey-like style, where some details may be neglected in favor of coverage. However, the extensive bibliography of each chapter will help the interested readers to find suitable sources for the missing details. Each chapter of the handbook is intended to be a self-contained survey of a topic, and is written by one or more authors who are leading researchers in the area.

The intended audience of the handbook is researchers, graduate students, higher-year undergraduates and practitioners who wish to learn about the state-of-the-art in constraint programming. No prior knowledge about the field is necessary to be able to read the chapters and gather useful knowledge. Researchers from other fields should find in this handbook an effective way to learn about constraint programming and to possibly use some of the constraint programming concepts and techniques in their work, thus providing a means for a fruitful cross-fertilization among different research areas.

The handbook is organized in two parts. The first part covers the basic foundations of constraint programming, including the history, the notion of constraint propagation, basic search methods, global constraints, tractability and computational complexity, and important issues in modeling a problem as a constraint problem. The second part covers constraint languages and solver, several useful extensions to the basic framework (such as interval constraints, structured domains, and distributed CSPs), and successful application areas for constraint programming.

- Covers the whole field of constraint programming
- Survey-style chapters
- Five chapters on applications

Table of Contents

Foreword (Ugo Montanari) 
Part I : Foundations
Chapter 1. Introduction (Francesca Rossi, Peter van Beek, Toby Walsh) 
Chapter 2. Constraint Satisfaction: An Emerging Paradigm (Eugene C. Freuder, Alan K. Mackworth) 
Chapter 3. Constraint Propagation (Christian Bessiere) 
Chapter 4. Backtracking Search Algorithms (Peter van Beek) 
Chapter 5. Local Search Methods (Holger H. Hoos, Edward Tsang) 
Chapter 6. Global Constraints (Willem-Jan van Hoeve, Irit Katriel)
Chapter 7. Tractable Structures for CSPs (Rina Dechter)
Chapter 8. The Complexity of Constraint Languages
(David Cohen, Peter Jeavons) 
Chapter 9. Soft Constraints (Pedro Meseguer, Francesca Rossi, Thomas Schiex) 
Chapter 10. Symmetry in Constraint Programming
(Ian P. Gent, Karen E. Petrie, Jean-Francois Puget)
Chapter 11. Modelling (Barbara M. Smith) 
Part II : Extensions, Languages, and Applications

Chapter 12. Constraint Logic Programming (Kim Marriott, Peter J. Stuckey, Mark Wallace) 
Chapter 13. Constraints in Procedural and Concurrent Languages (Thom Fruehwirth, Laurent Michel, Christian Schulte)
Chapter 14. Finite Domain Constraint Programming Systems (Christian Schulte, Mats Carlsson) 
Chapter 15. Operations Research Methods in Constraint Programming (John Hooker) 
Chapter 16. Continuous and Interval Constraints(Frederic Benhamou, Laurent Granvilliers) 
Chapter 17. Constraints over Structured Domains
(Carmen Gervet) 
Chapter 18. Randomness and Structure (Carla Gomes, Toby Walsh)
Chapter 19. Temporal CSPs (Manolis Koubarakis)
Chapter 20. Distributed Constraint Programming
(Boi Faltings) 
Chapter 21. Uncertainty and Change (Kenneth N. Brown, Ian Miguel)
Chapter 22. Constraint-Based Scheduling and Planning
(Philippe Baptiste, Philippe Laborie, Claude Le Pape, Wim Nuijten)
Chapter 23. Vehicle Routing (Philip Kilby, Paul Shaw) 
Chapter 24. Configuration (Ulrich Junker)
Chapter 25. Constraint Applications in Networks
(Helmut Simonis)
Chapter 26. Bioinformatics and Constraints (Rolf Backofen, David Gilbert)

/pdf/handbook-of-constraint-programming-53srutguo2.pdf

Handbook of Constraint Programming

http://cepac.cheme.cmu.edu/pasilectures/sahinidis/NVSpaperthree.pdf

Optimization under uncertainty: state-of-the-art and opportunities

In this paper a reformulation technique is presented that takes a given linear zero-one programming problem, converts it into a zero-one polynomial programming problem, and then relinearizes it into an extended linear program. It is shown that the strength of the resulting reformulation depends on the degree of the terms used to produce the polynomial program at the intermediate step of this method. In fact, as this degree varies from one up to the number of variables in the problem, a hierarchy of sharper representations is obtained with the final relaxation representing the convex hull of feasible solutions. The reformulation technique readily extends to produce a similar hierarchy of linear relaxations for zero-one polynomial programming problems. A characterization of the convex hull in the original variable space is also available through a projection process. The structure of this convex hull characterization (or its other relaxations) can be exploited to generate strong or facetial valid inequaliti...

A hierarchy of relaxation between the continuous and convex hull representations

Preface. 1. Introduction. Part I: Discrete Nonconvex Programs. 2. RLT Hierarchy for Mixed-Integer Zero-One Problems. 3. Generalized Hierarchy for Exploiting Special Structures in Mixed-Integer Zero-One Problems. 4. RLT Hierarchy for General Discrete Mixed-Integer Problems. 5. Generating Valid Inequalities and Facets Using RLT. 6. Persistency in Discrete Optimization. Part II: Continuous Nonconvex Programs. 7. RLT-Based Global Optimization Algorithms for Nonconvex Polynomial Programming Problems. 8. Reformulation-Convexification Technique for Quadratic Programs and Some Convex Envelope Characterizations. 9. Reformulation-Convexification Technique for Polynomial Programs: Design and Implementation. Part III: Special Applications to Discrete and Continuous Nonconvex Programs. 10. Applications to Discrete Problems. 11. Applications to Continuous Problems. References.

/pdf/a-reformulation-linearization-technique-for-solving-discrete-1wai0rmms0.pdf

A Reformulation-Linearization Technique for Solving Discrete and Continuous Nonconvex Problems

A hierarchy of relaxations and convex hull characterizations for mixed-integer zero-one programming problems

This paper is concerned with the solution of linearly constrained zero-one quadratic programming problems. Problems of this kind arise in numerous economic, location decision, and strategic planning situations, including capital budgeting, facility location, quadratic assignment, media selection, and dynamic set covering. A new linearization technique is presented for this problem which is demonstrated to yield a tighter continuous or linear programming relaxation than is available through other methods. An implicit enumeration algorithm which uses Lagrangian relaxation, Benders' cutting planes, and local explorations is designed to exploit the strength of this linearization. Computational experience is provided to demonstrate the usefulness of the proposed linearization and algorithm.

A Tight Linearization and an Algorithm for Zero-One Quadratic Programming Problems

This paper is concerned with a new linearization strategy for a class of zero-one mixed integer programming problems that contains quadratic cross-product terms between continuous and binary variables, and between the binary variables themselves. This linearization scheme provides an equivalent mixed integer linear programming problem which yields a tighter continuous relaxation than that obtainable via the alternative linearization techniques available in the literature. Moreover, the proposed technique provides a unifying framework in the sense that all the alternate methods lead to formulations that are accessible through appropriate surrogates of the constraints of the new linearized formulation. Extensions to various other types of mixed integer nonlinear programming problems are also discussed.

Warren P. Adams

Papers

A hierarchy of relaxation between the continuous and convex hull representations

A Reformulation-Linearization Technique for Solving Discrete and Continuous Nonconvex Problems

A hierarchy of relaxations and convex hull characterizations for mixed-integer zero-one programming problems

A Tight Linearization and an Algorithm for Zero-One Quadratic Programming Problems

Linearization strategies for a class of zero-one mixed integer programming problems