Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

Numerical Optimization

Today's Web-enabled deluge of electronic data calls for automated methods of data analysis. Machine learning provides these, developing methods that can automatically detect patterns in data and then use the uncovered patterns to predict future data. This textbook offers a comprehensive and self-contained introduction to the field of machine learning, based on a unified, probabilistic approach. The coverage combines breadth and depth, offering necessary background material on such topics as probability, optimization, and linear algebra as well as discussion of recent developments in the field, including conditional random fields, L1 regularization, and deep learning. The book is written in an informal, accessible style, complete with pseudo-code for the most important algorithms. All topics are copiously illustrated with color images and worked examples drawn from such application domains as biology, text processing, computer vision, and robotics. Rather than providing a cookbook of different heuristic methods, the book stresses a principled model-based approach, often using the language of graphical models to specify models in a concise and intuitive way. Almost all the models described have been implemented in a MATLAB software package--PMTK (probabilistic modeling toolkit)--that is freely available online. The book is suitable for upper-level undergraduates with an introductory-level college math background and beginning graduate students.

Machine Learning : A Probabilistic Perspective

A digital computer is generally believed to be an efficient universal computing device; that is, it is believed able to simulate any physical computing device with an increase in computation time by at most a polynomial factor. This may not be true when quantum mechanics is taken into consideration. This paper considers factoring integers and finding discrete logarithms, two problems which are generally thought to be hard on a classical computer and which have been used as the basis of several proposed cryptosystems. Efficient randomized algorithms are given for these two problems on a hypothetical quantum computer. These algorithms take a number of steps polynomial in the input size, e.g., the number of digits of the integer to be factored.

Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer

We present a novel approach to measuring similarity between shapes and exploit it for object recognition. In our framework, the measurement of similarity is preceded by: (1) solving for correspondences between points on the two shapes; (2) using the correspondences to estimate an aligning transform. In order to solve the correspondence problem, we attach a descriptor, the shape context, to each point. The shape context at a reference point captures the distribution of the remaining points relative to it, thus offering a globally discriminative characterization. Corresponding points on two similar shapes will have similar shape contexts, enabling us to solve for correspondences as an optimal assignment problem. Given the point correspondences, we estimate the transformation that best aligns the two shapes; regularized thin-plate splines provide a flexible class of transformation maps for this purpose. The dissimilarity between the two shapes is computed as a sum of matching errors between corresponding points, together with a term measuring the magnitude of the aligning transform. We treat recognition in a nearest-neighbor classification framework as the problem of finding the stored prototype shape that is maximally similar to that in the image. Results are presented for silhouettes, trademarks, handwritten digits, and the COIL data set.

Shape matching and object recognition using shape contexts

Planning algorithms are impacting technical disciplines and industries around the world, including robotics, computer-aided design, manufacturing, computer graphics, aerospace applications, drug design, and protein folding. This coherent and comprehensive book unifies material from several sources, including robotics, control theory, artificial intelligence, and algorithms. The treatment is centered on robot motion planning but integrates material on planning in discrete spaces. A major part of the book is devoted to planning under uncertainty, including decision theory, Markov decision processes, and information spaces, which are the “configuration spaces” of all sensor-based planning problems. The last part of the book delves into planning under differential constraints that arise when automating the motions of virtually any mechanical system. Developed from courses taught by the author, the book is intended for students, engineers, and researchers in robotics, artificial intelligence, and control theory as well as computer graphics, algorithms, and computational biology.

Planning Algorithms: Introductory Material

Kopec, Oppenheim, and Tribolet have described a homomorphic technique for producing, from a speech signal, a minimum-phase estimate of the vocal-tract impulse response. Once such an estimate has been obtained, the problem of modeling the vocal tract with a pole-zero model is a classical one in nonlinear estimation theory. It is shown in this paper that Shanks' method, Kalman's method, and the iterative prefiltering method are all different linearizations of the same nonlinear problem, and the iterative prefiltering method is proposed as an approach to estimating the poles and zeros of the vocal-tract transfer function simultaneously. A simulation is described which shows the advantage of estimating poles and zeros simultaneously rather than sequentially as in Shanks' method. A preliminary example of application to real speech is also given.

On the simultaneous estimation of poles and zeros in speech analysis

Full-duplex data communication over a multi-input/multi-output linear time-invariant channel is considered. The minimum mean square error (MMSE) linear equalizer is derived in the presence of both near- and far-end crosstalk and independent additive noise. The MMSE equalizer is completely specified in terms of the channel and crosstalk transfer functions by using a generalization of previous work due to Salz (1985). Conditions are given under which the equalizer can completely eliminate both near- and far-end crosstalk and intersymbol interference. The MMSE transmitter filter, subject to a transmitted power constraint, is specified when the channel and crosstalk transfer functions are bandlimited to the Nyquist frequency. Also considered is the design of MMSE transmitter and receiver filters when the data signals are arbitrary wide-sense stationary continuous or discrete-time signals, corresponding to the situation where the crosstalk is not phase-synchronous with the desired signal. >

/pdf/suppression-of-near-and-far-end-crosstalk-by-linear-pre-and-5gfua9exy3.pdf

Suppression of near- and far-end crosstalk by linear pre- and post-filtering

A practical method is described for designing recursive digital filters with arbitrary, prescribed magnitude characteristics. The method uses the Fletcher-Powell optimization algorithm to minimize a square-error criterion in the frequency domain. A strategy is described whereby stability and minimum-phase constraints are observed, while still using the unconstrained optimization algorithm. The cascade canonic form is used, so that the resultant filters can be realized accurately and simply. Design examples are given of low-pass, wide-band differentiator, linear discriminator, and vowel formant filters.

Computer-aided design of recursive digital filters

Soliton-like behavior in automata

We consider the problem of picking (buying) an inexpensive s -- t path in a graph where edges are owned by independent (selfish) agents, and the cost of an edge is known to its owner only. We study the problem of finding frugal mechanisms for this task, i.e. we investigate the payments the buyer must make in order to buy a path.First, we show that any mechanism with (weakly) dominant strategies (or, equivalently, any truthful mechanism) for the agents can force the buyer to make very large payments. Namely, for every such mechanism, the buyer can be forced to pay c(P) + 1/2k(c(Q) -- c(P)), where c(P) is the cost of the shortest path, c(Q) is the cost of the second-shortest path, and k is the number of edges in P. This extends the previous work of Archer and Tardos [1], who showed a similar lower bound for a subclass of truthful mechanisms called min-function mechanisms. Our lower bounds have no such limitations on the mechanism.Motivated by this lower bound, we study mechanisms for this problem providing Bayes-Nash equilibrium strategies for the agents. In this class, we identify the optimal mechanism with regard to total payment. We then demonstrate a separation in terms of average overpayments between the classical VCG mechanism and the optimal mechanism showing that under various natural distributions of edge costs, the optimal mechanism pays at most logarithmic factor more than the actual cost, whereas VCG pays √k times the actual cost. On the other hand, we also show that the optimal mechanism does incur at least a constant factor overpayment in natural distributions of edge costs. Since our mechanism is optimal, this gives a lower bound on all mechanisms with Bayes--Nash equilibria.

/pdf/frugality-in-path-auctions-15lvxn57xg.pdf

Kenneth Steiglitz

Papers

On the simultaneous estimation of poles and zeros in speech analysis

Suppression of near- and far-end crosstalk by linear pre- and post-filtering

Computer-aided design of recursive digital filters

Soliton-like behavior in automata

Frugality in path auctions