Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

LIBSVM is a library for Support Vector Machines (SVMs). We have been actively developing this package since the year 2000. The goal is to help users to easily apply SVM to their applications. LIBSVM has gained wide popularity in machine learning and many other areas. In this article, we present all implementation details of LIBSVM. Issues such as solving SVM optimization problems theoretical convergence multiclass classification probability estimates and parameter selection are discussed in detail.

/pdf/libsvm-a-library-for-support-vector-machines-2gdxasg9zv.pdf

LIBSVM: A library for support vector machines

The increasing volume of data in modern business and science calls for more complex and sophisticated tools. Although advances in data mining technology have made extensive data collection much easier, it's still always evolving and there is a constant need for new techniques and tools that can help us transform this data into useful information and knowledge. Since the previous edition's publication, great advances have been made in the field of data mining. Not only does the third of edition of Data Mining: Concepts and Techniques continue the tradition of equipping you with an understanding and application of the theory and practice of discovering patterns hidden in large data sets, it also focuses on new, important topics in the field: data warehouses and data cube technology, mining stream, mining social networks, and mining spatial, multimedia and other complex data. Each chapter is a stand-alone guide to a critical topic, presenting proven algorithms and sound implementations ready to be used directly or with strategic modification against live data. This is the resource you need if you want to apply today's most powerful data mining techniques to meet real business challenges. * Presents dozens of algorithms and implementation examples, all in pseudo-code and suitable for use in real-world, large-scale data mining projects. * Addresses advanced topics such as mining object-relational databases, spatial databases, multimedia databases, time-series databases, text databases, the World Wide Web, and applications in several fields. *Provides a comprehensive, practical look at the concepts and techniques you need to get the most out of real business data

/pdf/data-mining-concepts-and-techniques-4dtvdfkvmi.pdf

Data Mining: Concepts and Techniques

This paper describes a machine learning approach for visual object detection which is capable of processing images extremely rapidly and achieving high detection rates. This work is distinguished by three key contributions. The first is the introduction of a new image representation called the "integral image" which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features from a larger set and yields extremely efficient classifiers. The third contribution is a method for combining increasingly more complex classifiers in a "cascade" which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. The cascade can be viewed as an object specific focus-of-attention mechanism which unlike previous approaches provides statistical guarantees that discarded regions are unlikely to contain the object of interest. In the domain of face detection the system yields detection rates comparable to the best previous systems. Used in real-time applications, the detector runs at 15 frames per second without resorting to image differencing or skin color detection.

/pdf/rapid-object-detection-using-a-boosted-cascade-of-simple-4crej3dxmd.pdf

Rapid object detection using a boosted cascade of simple features

The tutorial starts with an overview of the concepts of VC dimension and structural risk minimization. We then describe linear Support Vector Machines (SVMs) for separable and non-separable data, working through a non-trivial example in detail. We describe a mechanical analogy, and discuss when SVM solutions are unique and when they are global. We describe how support vector training can be practically implemented, and discuss in detail the kernel mapping technique which is used to construct SVM solutions which are nonlinear in the data. We show how Support Vector machines can have very large (even infinite) VC dimension by computing the VC dimension for homogeneous polynomial and Gaussian radial basis function kernels. While very high VC dimension would normally bode ill for generalization performance, and while at present there exists no theory which shows that good generalization performance is guaranteed for SVMs, there are several arguments which support the observed high accuracy of SVMs, which we review. Results of some experiments which were inspired by these arguments are also presented. We give numerous examples and proofs of most of the key theorems. There is new material, and I hope that the reader will find that even old material is cast in a fresh light.

/pdf/a-tutorial-on-support-vector-machines-for-pattern-5cn6x20psd.pdf

A Tutorial on Support Vector Machines for Pattern Recognition

Given a set of circles C e lc1, …, cnr on the Euclidean plane with centers l(a1, b1), …, (an, bn)r and radii lr1, …, rnr, the smallest enclosing circle (of fixed circles) problem is to find the circle of minimum radius that encloses all circles in C. We survey four known approaches for this problem, including a second order cone reformulation, a subgradient approach, a quadratic programming scheme, and a randomized incremental algorithm. For the last algorithm we also give some implementation details. It turns out the quadratic programming scheme outperforms the other three in our computational experiment.

/pdf/solution-methodologies-for-the-smallest-enclosing-circle-3gwj9d2tre.pdf

Solution Methodologies for the Smallest Enclosing Circle Problem

This article provides a synopsis of the major developments in interior point methods for mathematical programming in the last thirteen years, and discusses current and future research directions in interior point methods, with a brief selective guide to the research literature.1

/pdf/interior-point-methods-current-status-and-future-directions-2rlwoc1uff.pdf

Interior point methods : current status and future directions

This paper presents extensions and further analytical properties of algorithms for linear programming based only on primal scaling and projected gradients of a potential function. The paper contains extensions and analysis of two polynomial-time algorithms for linear programming. We first present an extension of Gonzaga's O(nL) iteration algorithm, that computes dual variables and does not assume a known optimal objective function value. This algorithm uses only affine scaling, and is based on computing the projected gradient of the potential function
 
$$q\ln (x^T s) - \sum\limits_{j = 1}^n {\ln (x_j )} $$

 wherex is the vector of primal variables ands is the vector of dual slack variables, and q = n +
 
$$\sqrt n $$

 . The algorithm takes either a primal step or recomputes dual variables at each iteration. We next present an alternate form of Ye's O(
 
$$\sqrt n $$

 
L) iteration algorithm, that is an extension of the first algorithm of the paper, but uses the potential function
 
$$q\ln (x^T s) - \sum\limits_{j = 1}^n {\ln (x_j ) - \sum\limits_{j - 1}^n {\ln (s_j )} } $$

 where q = n +
 
$$\sqrt n $$

 . We use this alternate form of Ye's algorithm to show that Ye's algorithm is optimal with respect to the choice of the parameterq in the following sense. Suppose thatq = n + n

 t
 wheret⩾0. Then the algorithm will solve the linear program in O(n

 r
 
L) iterations, wherer = max{t, 1 − t}. Thus the value oft that minimizes the complexity bound ist = 1/2, yielding Ye's O(
 
$$\sqrt n $$

 
L) iteration bound.

Polynomial-time algorithms for linear programming based only on primal scaling and projected gradients of a potential function

A constructive proof of Tucker's combinatorial lemma

http://www.optimization-online.org/DB_FILE/2009/07/2341.pdf

Robert M. Freund

Papers

Solution Methodologies for the Smallest Enclosing Circle Problem

Interior point methods : current status and future directions

Polynomial-time algorithms for linear programming based only on primal scaling and projected gradients of a potential function

A constructive proof of Tucker's combinatorial lemma

Bandgap optimization of two-dimensional photonic crystals using semidefinite programming and subspace methods