Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.

Gradient-based learning applied to document recognition

LIBSVM is a library for Support Vector Machines (SVMs). We have been actively developing this package since the year 2000. The goal is to help users to easily apply SVM to their applications. LIBSVM has gained wide popularity in machine learning and many other areas. In this article, we present all implementation details of LIBSVM. Issues such as solving SVM optimization problems theoretical convergence multiclass classification probability estimates and parameter selection are discussed in detail.

/pdf/libsvm-a-library-for-support-vector-machines-2gdxasg9zv.pdf

LIBSVM: A library for support vector machines

The increasing volume of data in modern business and science calls for more complex and sophisticated tools. Although advances in data mining technology have made extensive data collection much easier, it's still always evolving and there is a constant need for new techniques and tools that can help us transform this data into useful information and knowledge. Since the previous edition's publication, great advances have been made in the field of data mining. Not only does the third of edition of Data Mining: Concepts and Techniques continue the tradition of equipping you with an understanding and application of the theory and practice of discovering patterns hidden in large data sets, it also focuses on new, important topics in the field: data warehouses and data cube technology, mining stream, mining social networks, and mining spatial, multimedia and other complex data. Each chapter is a stand-alone guide to a critical topic, presenting proven algorithms and sound implementations ready to be used directly or with strategic modification against live data. This is the resource you need if you want to apply today's most powerful data mining techniques to meet real business challenges. * Presents dozens of algorithms and implementation examples, all in pseudo-code and suitable for use in real-world, large-scale data mining projects. * Addresses advanced topics such as mining object-relational databases, spatial databases, multimedia databases, time-series databases, text databases, the World Wide Web, and applications in several fields. *Provides a comprehensive, practical look at the concepts and techniques you need to get the most out of real business data

/pdf/data-mining-concepts-and-techniques-4dtvdfkvmi.pdf

Data Mining: Concepts and Techniques

This paper describes a machine learning approach for visual object detection which is capable of processing images extremely rapidly and achieving high detection rates. This work is distinguished by three key contributions. The first is the introduction of a new image representation called the "integral image" which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features from a larger set and yields extremely efficient classifiers. The third contribution is a method for combining increasingly more complex classifiers in a "cascade" which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. The cascade can be viewed as an object specific focus-of-attention mechanism which unlike previous approaches provides statistical guarantees that discarded regions are unlikely to contain the object of interest. In the domain of face detection the system yields detection rates comparable to the best previous systems. Used in real-time applications, the detector runs at 15 frames per second without resorting to image differencing or skin color detection.

/pdf/rapid-object-detection-using-a-boosted-cascade-of-simple-4crej3dxmd.pdf

Rapid object detection using a boosted cascade of simple features

The tutorial starts with an overview of the concepts of VC dimension and structural risk minimization. We then describe linear Support Vector Machines (SVMs) for separable and non-separable data, working through a non-trivial example in detail. We describe a mechanical analogy, and discuss when SVM solutions are unique and when they are global. We describe how support vector training can be practically implemented, and discuss in detail the kernel mapping technique which is used to construct SVM solutions which are nonlinear in the data. We show how Support Vector machines can have very large (even infinite) VC dimension by computing the VC dimension for homogeneous polynomial and Gaussian radial basis function kernels. While very high VC dimension would normally bode ill for generalization performance, and while at present there exists no theory which shows that good generalization performance is guaranteed for SVMs, there are several arguments which support the observed high accuracy of SVMs, which we review. Results of some experiments which were inspired by these arguments are also presented. We give numerous examples and proofs of most of the key theorems. There is new material, and I hope that the reader will find that even old material is cast in a fresh light.

/pdf/a-tutorial-on-support-vector-machines-for-pattern-5cn6x20psd.pdf

A Tutorial on Support Vector Machines for Pattern Recognition

For a conic linear system of the form Ax ∈ K, K a convex cone, several condition measures have been extensively studied in the last dozen years. Among these, Renegar's condition number C(A) is arguably the most prominent for its relation to data perturbation, error bounds, problem geometry, and computational complexity of algorithms. Nonetheless, C(A) is a representation-dependent measure which is usually difficult to interpret and may lead to overly-conservative bounds of computational complexity and/or geometric quantities associated with the set of feasible solutions. Herein we show that Renegar's condition number is bounded from above and below by certain purely geometric quantities associated with A and K, and highlights the role of the singular values of A and their relationship with the condition number. Moreover, by using the notion of conic curvature, we show how Renegar's condition number can be used to provide both lower and upper bounds on the width of the set of feasible solutions. This complements the literature where only lower bounds have heretofore been developed.

/pdf/a-geometric-analysis-of-renegar-s-condition-number-and-its-1fdldjq5sg.pdf

A Geometric Analysis of Renegar's Condition Number, and its Interplay with Conic Curvature

The classical perceptron algorithm is an elementary row-action/relaxation algorithm for solving a homogeneous linear inequality system Ax > 0. A natural condition measure associated with this algorithm is the Euclidean width T of the cone of feasible solutions, and the iteration complexity of the perceptron algorithm is bounded by 1/T^2, see Rosenblatt 1962. Dunagan and Vempala have developed a re-scaled version of the perceptron algorithm with an improved complexity of O(n ln(1/T)) iterations (with high probability), which is theoretically efficient in T, and in particular is polynomial-time in the bit-length model. We explore extensions of the concepts of these perceptron methods to the general homogeneous conic system Ax is an element of a set int K where K is a regular convex cone. We provide a conic extension of the re-scaled perceptron algorithm based on the notion of a deep-separation oracle of a cone, which essentially computes a certificate of strong separation. We give a general condition under which the re-scaled perceptron algorithm is itself theoretically efficient; this includes the cases when K is the cross-product of half-spaces, second-order cones, and the positive semi-definite cone.

/pdf/an-efficient-re-scaled-perceptron-algorithm-for-conic-2sfofut5hp.pdf

An Efficient Re-Scaled Perceptron Algorithm for Conic Systems

We perform full 3D topology optimization (in which "every voxel" of the unit cell is a degree of freedom) of photonic-crystal structures in order to find optimal omnidirectional band gaps for various symmetry groups, including fcc (including diamond), bcc, and simple-cubic lattices. Even without imposing the constraints of any fabrication process, the resulting optimal gaps are only slightly larger than previous hand designs, suggesting that current photonic crystals are nearly optimal in this respect. However, optimization can discover new structures, e.g. a new fcc structure with the same symmetry but slightly larger gap than the well known inverse opal, which may offer new degrees of freedom to future fabrication technologies. Furthermore, our band-gap optimization is an illustration of a computational approach to 3D dispersion engineering which is applicable to many other problems in optics, based on a novel semidefinite-program formulation for nonconvex eigenvalue optimization combined with other techniques such as a simple approach to impose symmetry constraints. We also demonstrate a technique for \emph{robust} topology optimization, in which some uncertainty is included in each voxel and we optimize the worst-case gap, and we show that the resulting band gaps have increased robustness to systematic fabrication errors.

/pdf/robust-topology-optimization-of-three-dimensional-photonic-4pmzbo7gxs.pdf

Robust topology optimization of three-dimensional photonic-crystal band-gap structures

We evaluate the practical relevance of two measures of conic convex problem complexity as applied to second-order cone problems solved using the homogeneous self-dual (HSD) embedding model in the software SeDuMi. The first measure we evaluate is Renegar's data-based condition measure C(d), and the second measure is a combined measure of the optimal solution size and the initial infeasibility/optimality residuals denoted by S (where the solution size is measured in a norm that is naturally associated with the HSD model). We constructed a set of 144 secondorder cone test problems with widely distributed values of C(d) and S and solved these problems using SeDuMi. For each problem instance in the test set, we also computed estimates of C(d) (using Pena's method) and computed S directly. Our computational experience indicates that SeDuMi iteration counts and log(C(d)) are fairly highly correlated (sample correlation R = 0.676), whereas SeDuMi iteration counts are not quite as highly correlated with S (R = 0.600). Furthermore, the experimental evidence indicates that the average rate of convergence of SeDuMi iterations is affected by the condition number C(d) of the problem instance, a phenomenon that makes some intuitive sense yet is not directly implied by existing theory.

On Two Measures of Problem Instance Complexity and their Correlation with the Performance of SeDuMi on Second-Order Cone Problems

The design of metamaterials is currently a fertile research domain. However, most of the metamaterial designs described in the literature arise from physical intuition, and often assume infinite periodicity. There is therefore a need for a design methodology capable of computing patterns and designs involving two different materials where the underlying design variables correspond to a finite set of pixels in a 2-dimensional mesh, and where the goal is a design with prescribed metamaterial properties. This naturally leads to the consideration of binary optimization models in contrast to classical (continuous) gradient-based methods which generically provide continuous solutions that then need to be “rounded” to binary values. While the potential drawback of binary optimization is that its computational complexity is usually NP-hard and hence theoretically unattractive, we show herein that binary optimization combined with a reduced basis approach can relatively efficiently produce very good solutions to metamaterial design problems of interest.

Robert M. Freund

Papers

A Geometric Analysis of Renegar's Condition Number, and its Interplay with Conic Curvature

An Efficient Re-Scaled Perceptron Algorithm for Conic Systems

Robust topology optimization of three-dimensional photonic-crystal band-gap structures

On Two Measures of Problem Instance Complexity and their Correlation with the Performance of SeDuMi on Second-Order Cone Problems

A Binary Optimization Method for Linear Metamaterial Design Optimization