We review machine learning methods employing positive definite kernels. These methods formulate learning and estimation problems in a reproducing kernel Hilbert space (RKHS) of functions defined on the data domain, expanded in terms of a kernel. Working in linear spaces of function has the benefit of facilitating the construction and analysis of learning algorithms while at the same time allowing large classes of functions. The latter include nonlinear functions as well as functions defined on nonvectorial data. We cover a wide range of methods, ranging from binary classifiers to sophisticated methods for estimation with structured data.

/pdf/kernel-methods-in-machine-learning-2d4c18y756.pdf

Kernel methods in machine learning

While classical kernel-based learning algorithms are based on a single kernel, in practice it is often desirable to use multiple kernels. Lanckriet et al. (2004) considered conic combinations of kernel matrices for classification, leading to a convex quadratically constrained quadratic program. We show that it can be rewritten as a semi-infinite linear program that can be efficiently solved by recycling the standard SVM implementations. Moreover, we generalize the formulation and our method to a larger class of problems, including regression and one-class classification. Experimental results show that the proposed algorithm works for hundred thousands of examples or hundreds of kernels to be combined, and helps for automatic model selection, improving the interpretability of the learning result. In a second part we discuss general speed up mechanism for SVMs, especially when used with sparse feature maps as appear for string kernels, allowing us to train a string kernel SVM on a 10 million real-world splice data set from computational biology. We integrated multiple kernel learning in our machine learning toolbox SHOGUN for which the source code is publicly available at http://www.fml.tuebingen.mpg.de/raetsch/projects/shogun .

/pdf/large-scale-multiple-kernel-learning-b4x3bns83a.pdf

Large Scale Multiple Kernel Learning

We consider the problem of choosing a set of k sensor measurements, from a set of m possible or potential sensor measurements, that minimizes the error in estimating some parameters. Solving this problem by evaluating the performance for each of the (m k) possible choices of sensor measurements is not practical unless m and k are small. In this paper, we describe a heuristic, based on convex optimization, for approximately solving this problem. Our heuristic gives a subset selection as well as a bound on the best performance that can be achieved by any selection of k sensor measurements. There is no guarantee that the gap between the performance of the chosen subset and the performance bound is always small; but numerical experiments suggest that the gap is small in many cases. Our heuristic method requires on the order of m 3 operations; for m= 1000 possible sensors, we can carry out sensor selection in a few seconds on a 2-GHz personal computer.

/pdf/sensor-selection-via-convex-optimization-2nryt3q469.pdf

Sensor Selection via Convex Optimization

In this paper we give an explicit description of the cones of polynomials represent able as a sum of squared functions. We prove that such cones can be always seen as a linear image of the cone of positive semidefinite matrices. As a consequence of the result, we get a description of the cones of univariate polynomials, which are non-negative on a ray and on an interval, and a description of non-negative trigonometric polynomials. We discuss some applications of the results to multi-variate polynomials.

Squared Functional Systems and Optimization Problems

We provide an introduction to theoretical and practical aspects of Boosting and Ensemble learning, providing a useful reference for researchers in the field of Boosting as well as for those seeking to enter this fascinating area of research. We begin with a short background concerning the necessary learning theoretical foundations of weak learners and their linear combinations. We then point out the useful connection between Boosting and the Theory of Optimization, which facilitates the understanding of Boosting and later on enables us to move on to new Boosting algorithms, applicable to a broad spectrum of problems. In order to increase the relevance of the paper to practitioners, we have added remarks, pseudo code, "tricks of the trade", and algorithmic considerations where appropriate. Finally, we illustrate the usefulness of Boosting algorithms by giving an overview of some existing applications. The main ideas are illustrated on the problem of binary classification, although several extensions are discussed.

/pdf/an-introduction-to-boosting-and-leveraging-36mfqregqi.pdf

An introduction to boosting and leveraging

Starting from a number of motivating and abundant applications in §2, including control of robots, eigenvalue computations, mechanical stress of materials, and statistical design, the authors describe a class of optimization problems which are referred to as semi-infinite, because their constraints bound functions of a finite number of variables on a whole region. In §§3–5, first- and second-order optimality conditions are derived for general nonlinear problems as well as a procedure for reducing the problem locally to one with only finitely many constraints. Another main effort for achieving simplification is through duality in §6. There, algebraic properties of finite linear programming are brought to bear on duality theory in semi-infinite programming. Section 7 treats numerical methods based on either discretization or local reduction with the emphasis on the design of superlinearly convergent (SQP-type) methods. Taking this differentiable point of view, this paper can be considered to be complementar...

K. O. Kortanek

Papers

Semi-infinite programming: theory, methods, and applications