Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.

Deep Learning

There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Preface 1. Introduction Overview A Brief History of LMIs in Control Theory Notes on the Style of the Book Origin of the Book 2. Some Standard Problems Involving LMIs. Linear Matrix Inequalities Some Standard Problems Ellipsoid Algorithm Interior-Point Methods Strict and Nonstrict LMIs Miscellaneous Results on Matrix Inequalities Some LMI Problems with Analytic Solutions 3. Some Matrix Problems. Minimizing Condition Number by Scaling Minimizing Condition Number of a Positive-Definite Matrix Minimizing Norm by Scaling Rescaling a Matrix Positive-Definite Matrix Completion Problems Quadratic Approximation of a Polytopic Norm Ellipsoidal Approximation 4. Linear Differential Inclusions. Differential Inclusions Some Specific LDIs Nonlinear System Analysis via LDIs 5. Analysis of LDIs: State Properties. Quadratic Stability Invariant Ellipsoids 6. Analysis of LDIs: Input/Output Properties. Input-to-State Properties State-to-Output Properties Input-to-Output Properties 7. State-Feedback Synthesis for LDIs. Static State-Feedback Controllers State Properties Input-to-State Properties State-to-Output Properties Input-to-Output Properties Observer-Based Controllers for Nonlinear Systems 8. Lure and Multiplier Methods. Analysis of Lure Systems Integral Quadratic Constraints Multipliers for Systems with Unknown Parameters 9. Systems with Multiplicative Noise. Analysis of Systems with Multiplicative Noise State-Feedback Synthesis 10. Miscellaneous Problems. Optimization over an Affine Family of Linear Systems Analysis of Systems with LTI Perturbations Positive Orthant Stabilizability Linear Systems with Delays Interpolation Problems The Inverse Problem of Optimal Control System Realization Problems Multi-Criterion LQG Nonconvex Multi-Criterion Quadratic Problems Notation List of Acronyms Bibliography Index.

/pdf/linear-matrix-inequalities-in-system-and-control-theory-2q5cf0pqm2.pdf

Linear Matrix Inequalities in System and Control Theory

http://control.disp.uniroma2.it/galeani/CO/materiale/automatica2000.pdf

Survey Constrained model predictive control: Stability and optimality

In the first part of this thesis, we introduce a specific class of Linear Matrix Inequalities (LMI) whose optimal solution can be characterized exactly. This family corresponds to the case where the associated linear operator maps the cone of positive semidefinite matrices onto itself. In this case, the optimal value equals the spectral radius of the operator. It is shown that some rank minimization problems, as well as generalizations of the structured singular value ($mu$) LMIs, have exactly this property.

In the same spirit of exploiting structure to achieve computational efficiency, an algorithm for the numerical solution of a special class of frequency-dependent LMIs is presented. These optimization problems arise from robustness analysis questions, via the Kalman-Yakubovich-Popov lemma. The procedure is an outer approximation method based on the algorithms used in the computation of hinf norms for linear, time invariant systems. The result is especially useful for systems with large state dimension.

The other main contribution in this thesis is the formulation of a convex optimization framework for semialgebraic problems, i.e., those that can be expressed by polynomial equalities and inequalities. The key element is the interaction of concepts in real algebraic geometry (Positivstellensatz) and semidefinite programming.

To this end, an LMI formulation for the sums of squares decomposition for multivariable polynomials is presented. Based on this, it is shown how to construct sufficient Positivstellensatz-based convex tests to prove that certain sets are empty. Among other applications, this leads to a nonlinear extension of many LMI based results in uncertain linear system analysis.

Within the same framework, we develop stronger criteria for matrix copositivity, and generalizations of the well-known standard semidefinite relaxations for quadratic programming.

Some applications to new and previously studied problems are presented. A few examples are Lyapunov function computation, robust bifurcation analysis, structured singular values, etc. It is shown that the proposed methods allow for improved solutions for very diverse questions in continuous and combinatorial optimization.

Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization

In this paper, we present new conditions ensuring existence, uniqueness, and Global Asymptotic Stability (GAS) of the equilibrium point for a large class of neural networks. The results are applicable to both symmetric and nonsymmetric interconnection matrices and allow for the consideration of all continuous nondecreasing neuron activation functions. Such functions may be unbounded (but not necessarily surjective), may have infinite intervals with zero slope as in a piece-wise-linear model, or both. The conditions on GAS rely on the concept of Lyapunov Diagonally Stable (or Lyapunov Diagonally Semi-Stable) matrices and are proved by employing a class of Lyapunov functions of the generalized Lur'e-Postnikov type. Several classes of interconnection matrices of applicative interest are shown to satisfy our conditions for GAS. In particular, the results are applied to analyze GAS for the class of neural circuits introduced for solving linear and quadratic programming problems. In this application, the principal result here obtained is that these networks are GAS also when the constraint amplifiers are dynamical, as it happens in any practical implementation. >

New conditions for global stability of neural networks with application to linear and quadratic programming problems

The authors propose a theoretical framework for backpropagation (BP) in order to identify some of its limitations as a general learning procedure and the reasons for its success in several experiments on pattern recognition. The first important conclusion is that examples can be found in which BP gets stuck in local minima. A simple example in which BP can get stuck during gradient descent without having learned the entire training set is presented. This example guarantees the existence of a solution with null cost. Some conditions on the network architecture and the learning environment that ensure the convergence of the BP algorithm are proposed. It is proven in particular that the convergence holds if the classes are linearly separable. In this case, the experience gained in several experiments shows that multilayered neural networks (MLNs) exceed perceptrons in generalization to new examples. >

On the problem of local minima in backpropagation

Harmonic balance methods for the analysis of chaotic dynamics in nonlinear systems

In this note, robust stability of state-space models with respect to real parametric uncertainty is considered. Specifically, a new class of parameter-dependent quadratic Lyapunov functions for establishing stability of a polytope of matrices is introduced, i.e., the homogeneous polynomially parameter-dependent quadratic Lyapunov functions (HPD-QLFs). The choice of this class, which contains parameter-dependent quadratic Lyapunov functions whose dependence on the uncertain parameters is expressed as a polynomial homogeneous form, is motivated by the property that a polytope of matrices is stable if and only there exists an HPD-QLF. The main result of the note is a sufficient condition for determining the sought HPD-QLF, which amounts to solving linear matrix inequalities (LMIs) derived via the complete square matricial representation (CSMR) of homogeneous matricial forms and the Lyapunov matrix equation. Numerical examples are provided to demonstrate the effectiveness of the proposed approach.

Polynomially parameter-dependent Lyapunov functions for robust stability of polytopic systems: an LMI approach

Positive Forms.- Positivity Gap.- Robustness with Time-varying Uncertainty.- Robustness with Time-invariant Uncertainty.- Robustness with Bounded-rate Time-varying Uncertainty.- Distance Problems with Applications to Robust Control.

Alberto Tesi

Papers

New conditions for global stability of neural networks with application to linear and quadratic programming problems

On the problem of local minima in backpropagation

Harmonic balance methods for the analysis of chaotic dynamics in nonlinear systems

Polynomially parameter-dependent Lyapunov functions for robust stability of polytopic systems: an LMI approach

Homogeneous Polynomial Forms for Robustness Analysis of Uncertain Systems