Numerical Optimization presents a comprehensive and up-to-date description of the most effective methods in continuous optimization. It responds to the growing interest in optimization in engineering, science, and business by focusing on the methods that are best suited to practical problems. For this new edition the book has been thoroughly updated throughout. There are new chapters on nonlinear interior methods and derivative-free methods for optimization, both of which are used widely in practice and the focus of much current research. Because of the emphasis on practical methods, as well as the extensive illustrations and exercises, the book is accessible to a wide audience. It can be used as a graduate text in engineering, operations research, mathematics, computer science, and business. It also serves as a handbook for researchers and practitioners in the field. The authors have strived to produce a text that is pleasant to read, informative, and rigorous - one that reveals both the beautiful nature of the discipline and its practical side.

Numerical Optimization

ion makes it possible to build non-recursive functions directly inside Calculus of Constructions terms, without giving a name to them, but for recursive functions the Fixpoint command always gives a name to the defined function. It mixes the two operations: first the description of a recursive function, second the definition of a constant having this function as value. With the fix construct, we can have only the first operation; in this sense, it is similar to the abstraction construct. Here is the syntax for this construct: As with the Fixpoint command, the {struct ai} is not mandatory if p = l. In the particular case where one defines only one recursive function the two occurrences of the identifier f must coincide. This identifier is used to denote the recursive function being defined but it can only be used inside the expression expr. The similarity between Fixpoint and fix makes it easier to understand the need for the various parts of this construct. For instance, the mul t2 function could also have been declared in the following manner: Definition mult2' : nat~nat := fix f (n:nat) : nat .= match n with 0 =} 0 I S p =} S (S (f p)) end. Here we have willingly changed the name given to the recursive function inside the fix construct to underline the fact that f is bound only inside the construct. Thus, this identifier has no relation with the name under which the function will be known. 6.4 Polymorphic Types 175 6.4 Polymorphic Types Among the operations that one can perform on binary trees carrying integer values, many rely only on the tree structure, but are independent of the fact that the values are integer values. For instance, we can compute the size or the height of a tree without looking at the values. It is sensible to define a general type of tree, in which the type of elements is left as a parameter, and use instances of this general type according to the needs of our algorithms. This is similar to the polymorphic data structures available in conventional functional languages, or the generic data structures of Ada. We describe this notion of polymorphism on lists, pairs, etc. 6.4.1 Polymorphic Lists The Coq system provides a theory of polymorphic lists in the package List. Require Import List. Print list. Inductive list (A : Set) : Set := nil : list A / cons : A -+ list A -+ list A For nil: Argument A is implicit For cons: Argument A is implicit For list: Argument scope is [type_scope] For nil: Argument scope is [type_scope] For cons: Argument scopes are [type_scope __ ] The Coq system provides a notation for lists, so that the expression" cons a l" is actually denoted "a: : l." We see here that the inductive type being defined does not occur as a simple identifier, but as a dependent type with one argument. The value of this argument is always A, the parameter of the definition, as given in the first line of the definition. This definition behaves as if we were actually defining a whole family of inductive types, indexed over the sort Set. This illustrates the construction of higher-order types that we saw in Chap. 4. There may be several parameters in an inductive definition. When parameters are provided, they must appear at every use of the type being defined. Everything happens as if the inductive type had been declared in a section, with a context where A is bound as a variable. Thus, the definition above is equivalent to a definition of the following form: Section define_lists. Variable A : Set. Inductive list' : Set := I nil' : list' I cons' : A -+ list' -+ list'. End define_lists. 176 6 Inductive Data Types This analogy between polymorphic inductive types and simple inductive types defined inside a section is helpful to understand the form of the induction principle. Let us first study the type of the induction principle as it would have been constructed inside the section: list'_indO : 'v'P : list' --+Prop, P nil' --+ ('v'(x:A)(l:list'), P 1 --+ P (cons' x 1»--+ 'v'x:list', P x. When the section is closed, the variable A is discharged, the type list' must be abstracted over A, the constructors, too, and the induction principle must take these changes into account: Check list'. list' : Set--+Set Check nil'. nil' : 'v' A:Set, list' A Check cons'. cons' : 'v' A:Set, A --+ list' A --+ list' A Check list'_ind. list' ind: 'v' (A:Set)(P:list' A --+ Prop), P (nil' A) --+{V (a:A)(l:list' A), P l--+ P (cons' A a l)) --+ 'v' l:list' A, P l From a practical point of view, an important characteristic of parametric inductive definitions is that the universal quantification appears before the universal quantification over the property that is the object of the proof by induction. This characteristic will be important in comparison with the inductive principles for inductive definitions of variably dependent types (see Sect. 6.5.2) Recursive functions and pattern matching on polymorphic types can be performed in the same manner as for the inductive types of the previous sections. However, there is an important difference; the parameters must not appear in the left-hand side of pattern matching clauses. For instance, the function to concatenate polymorphic lists is defined by an expression of this form: Fixpoint app (A:Set) (1 m:list A){struct I} : list A := match 1 with I nil :::} m I cons a 11 :::} cons a (app A 11 m) end. In this pattern matching construct, nil appears in the left-hand side of its clause without its Set argument. The same occurs for the cons constructor, 6.4 Polymorphic Types 177 even though cons normally has three arguments; the pattern only has two. The reasoh for removing the parameter arguments from the constructors is that these parameters cannot be bound in the pattern. The type A for the values is already fixed because an expression of type "list A" is being analyzed by the pattern matching construct. In the right-hand side of the second clause, cons also appears with two arguments, but this is because the function is defined with the first argument being implicit (see Sect. 4.2.3.1). Use of implicit arguments for functions manipulating polymorphic types is frequent. For instance, the function app also has its first argument as an implicit argument. For this function, the Coq system also provides an infix notation, where "app h h" is actually denoted "h++h." Exercise 6.34 Build a polymorphic function that takes a list as argument and returns a list containing the first two elements when they exist. Exercise 6.35 Build a function that takes a natural number, n, and a list as arguments and returns the list containing the first n elements of the list when they exist. Exercise 6.36 Build a function that takes a list of integers as argument and returns the sum of these numbers. Exercise 6.37 Build a function that takes a natural number n as argument and builds a list containing n occurrences of the number one. Exercise 6.38 Build a function that takes a number n and returns the list containing the integers from 1 to n, in this order. 6.4.2 The option Type Polymorphic types need not be truly recursive. A frequent example is the option type that is well-adapted to describe a large class of partial functions. This type is also present in conventional functional programming languages. Its inductive definition has the following form: Print option. Inductive option (A:Set) : Set := Some: A-+option A / None: option A For Some: Argument A is implicit For None: Argument A is implicit For option: Argument scope is [type_scope] For Some: Argument scopes are [type_scope _] For None: Argument scope is [type_scope] 178 6 Inductive Data Types When we need to define a function that is not total from a type A to a type B, it is often possible to describe it as a total function from A to "option B," with the convention that the value "None" is the result when the function is not defined and the value is "Some y" when the function would have been defined with value y. For instance, the Coq library contains a pred function of type nat---+nat that maps any natural number to its predecessor (when it exists) and maps zero to itself. A partial function could have been defined that does not have a value for the number zero. The definition would have been as follows: Definition pred_option (n:nat) : option nat := match n with 0 => None I S P => Some pend. To use a value of the option type, a case analysis is necessary, to express explicitly how the computation proceeds when no true value is given. For instance, the function that returns the predecessor's predecessor can be defined as follows: Definition pred2_option (n:nat) : option nat := match pred_option n with I None => None I Some p => pred_option p end. As a second example, we can consider the function that returns the nth element of a list. The Coq library provides a function called nth for this requirement but that function takes an extra argument which is used for the result when the list has less than n elements. An alternative could be to define a function whose result belongs in the option type. This function can be defined using simultaneous pattern matching on the number and the list. Both arguments decrease at each recursive call, so that the principal recursion argument could be either of them. Here is one of the two possible versions: Fixpoint nth_option (A:Set) (n:nat) (l:list A){struct l} : option A := match n, 1 with I 0, cons a tl => Some a I S p, cons a tl => nth_option A p tl I n, nil => None end. Exercise 6.39 Define the other variant "nth_option'." The arguments are given in the same order, but the principal argument is the number n. Prove that both functions always give the same result when applied on the same input. Exercise 6.40 * Prove V(A:Set)(n:nat)(l:list A), nth_option A n 1 = None ---+ length 1 < n. 6.4 Polymorphic Types 179 Exercise 6.41 * Define a function that maps a type A in sort Set, a function f of type A---+booL, and a list L to the first element :z: in L such that "f x" is true. 6.4.3 The Type of Pairs Pairs prov

Interactive Theorem Proving and Program Development

In its basic form, the reverse mode of computational differentiation yields the gradient of a scalar-valued function at a cost that is a small multiple of the computational work needed to evaluate the function itself. However, the corresponding memory requirement is proportional to the run-time of the evaluation program. Therefore, the practical applicability of the reverse mode in its original formulation is limited despite the availability of ever larger memory systems. This observation leads to the development of checkpointing schedules to reduce the storage requirements. This article presents the function revolve, which generates checkpointing schedules that are provably optimal with regard to a primary and a secondary criterion. This routine is intended to be used as an explicit “controller” for running a time-dependent applications program.

Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation

Numerical codes that calculate not only a result, but also the derivatives of the variables with respect to each other, facilitate sensitivity analysis, inverse problem solving, and optimization. The paper considers how Adifor 2.0, which won the 1995 Wilkinson Prize for Numerical Software, can automatically differentiate complicated Fortran code much faster than a programmer can do it by hand. The Adifor system has three main components: the AdiFor preprocessor, the ADIntrinsics exception-handling system, and the SparsLinC library.

Adifor 2.0: automatic differentiation of Fortran 77 programs

Automatic Differentiation (AD) is a maturing computational technology. It has become a mainstream tool used by practicing scientists and computer engineers. The rapid advance of hardware computing power and AD tools has enabled practitioners to generate derivative enhanced versions of their code for a broad range of applications in applied research and development. Automatic Differentiation of Algorithms provides a comprehensive and authoritative survey of all recent developments, new techniques, and tools for AD use.

Automatic Differentiation Of Algorithms: From Simulation To Optimization

We give new bounds and algorithms for minimal solutions of linear diophantine systems. These bounds are simply exponential, while previous known bounds were, at least until recently, doubly exponential.

Minimal solutions of linear diophantine systems: bounds and algorithms

This paper describes current results of Ofr, a system for extracting and understanding mathematical expressions in documents. Such a tool could be really useful to be able to re-use knowledge in scientific books which are not available in electronic form. We currently also study use of this system for direct input of formulas with a graphical tablet for computer algebra system softwares. Existing solutions for mathematical recognition have problems to analyze 2D expressions like vectors and matrices. This is because they often try to use extended classical grammar to analyze formulas, relatively to baseline. But a lot of mathematical notations do not respect rules for such a parsing and that is the reason why they fail to extend text parsing technic. We investigate graph grammar and graph rewriting as a solution to recognize 2D mathematical notations. Graph grammar provide a powerful formalism to describe structural manipulations of multi-dimensional data. The main two problems to solve are ambiguities between rules of grammar and construction of graph.© (1998) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

https://hal.archives-ouvertes.fr/hal-01349210/document

Mathematical formula recognition using graph grammar

This paper presents an approach for the recognition of on-line handwritten mathematical expressions. The hidden Markov model (HMM) based system makes use of simultaneous segmentation and recognition capabilities, avoiding a crucial segmentation during pre-processing. With the segmentation and recognition results, obtained from the HMM recognizer it is possible to analyze and interpret the spatial two-dimensional arrangement of the symbols. We use a graph grammar approach for the structure recognition, also used in off-line recognition process, resulting in a general tree-structure of the underlying input-expression. The resulting constructed tree can be translated to any desired syntax (for example: Lisp, KT/sub E/X, and OpenMath).

/pdf/on-line-handwritten-formula-recognition-using-hidden-markov-5asjiucru6.pdf

On-line handwritten formula recognition using hidden Markov models and context dependent graph grammars

The paper describes the design and the first steps of implementation of OFR (optical formula recognition), a system for extracting and understanding mathematical expressions in printed documents. Our approach clearly separate OCR step, geometrical treatments and syntactic analysis. We focus on the third part: we define a class of context sensitive graph grammars for mathematical formulas, study their properties and show how to remove their ambiguities (by adding contexts in rules) to define efficient parsing. This method is based on a "critical pairs" approach in the sense of Knuth-Bendix algorithm.

/pdf/optical-formula-recognition-41gde57kp8.pdf

Optical formula recognition

This report concerns time/space trade-offs for the reverse mode of automatic differentiation on the straight-line programs with nested loops class of programs. In the first part we consider the problem of reversing a finite sequence given by $u_{n+1}=3Df(u_n)$, which can mode lize a certain class of finite loops. We show an optimal time strategy for this problem, the number of available registers being fixed, and a lower bound on the time-space product equal to $\frac{p (\ln p)^2}{(\ln 4)^2}$. We then present an optimal strategy on nested loops with the objective of taking care of the program structure. Finally we consider an application of this storage/recomputation strategy to compute in reverse mode the derivatives of a function represented as a {\sc fortran} program.

https://hal.inria.fr/inria-00073896/document

Loïc Pottier

Papers

Minimal solutions of linear diophantine systems: bounds and algorithms

Mathematical formula recognition using graph grammar

On-line handwritten formula recognition using hidden Markov models and context dependent graph grammars

Optical formula recognition

Optimal Time and Minimum Space-Time Product for Reversing a Certain Class of Programs