Showing papers on "Dynamic programming published in 2002"

Adaptive dynamic programming

Abstract: Unlike the many soft computing applications where it suffices to achieve a "good approximation most of the time," a control system must be stable all of the time. As such, if one desires to learn a control law in real-time, a fusion of soft computing techniques to learn the appropriate control law with hard computing techniques to maintain the stability constraint and guarantee convergence is required. The objective of the paper is to describe an adaptive dynamic programming algorithm (ADPA) which fuses soft computing techniques to learn the optimal cost (or return) functional for a stabilizable nonlinear system with unknown dynamics and hard computing techniques to verify the stability and convergence of the algorithm. Specifically, the algorithm is initialized with a (stabilizing) cost functional and the system is run with the corresponding control law (defined by the Hamilton-Jacobi-Bellman equation), with the resultant state trajectories used to update the cost functional in a soft computing mode. Hard computing techniques are then used to show that this process is globally convergent with stepwise stability to the optimal cost functional/control law pair for an (unknown) input affine system with an input quadratic performance measure (modulo the appropriate technical conditions). Three specific implementations of the ADPA are developed for 1) the linear case, 2) for the nonlinear case using a locally quadratic approximation to the cost functional, and 3) the nonlinear case using a radial basis function approximation of the cost functional; illustrated by applications to flight control.

Dynamic Programming Algorithm Optimization for Spoken Word Recognition

Abstract: This paper reports on an optimum dynamic programming (DP) based time-normalization algorithm for spoken word recognition First, a general principle of time-normalization i s given using timewarping function Then, two time-normalized distance definitions, d e d symmetric and asymmetric forms, are derived from the principle These two forms are compared with each other through theoretical discussions and experimental studies The symmetric form algorithm superiority is established A new technique, called slope constraint, is successfully introduced, in which the warping function slope is restricted so as to improve discrimination between words in different categories The effective slope constraint characteristic is qualitatively analyzed, and the optimum slope constraint condition is determined through experiments The optimized algorithm is then extensively subjected to experimentat comparison with various DP-algorithms, previously applied to spoken word recognition by different research groups The experiment shows that the present algorithm gives no more than about twothirds errors, even compared to the best conventional algorithm I

Synthesizing Monitors for Safety Properties

Abstract: The problem of testing a linear temporal logic (LTL) formula on a finite execution trace of events, generated by an executing program, occurs naturally in runtime analysis of software. An algorithm which takes a past time LTL formula and generates an efficient dynamic programming algorithm is presented. The generated algorithm tests whether the formula is satisfied by a finite trace of events given as input and runs in linear time, its constant depending on the size of the LTL formula. The memory needed is constant, also depending on the size of the formula. Further optimizations of the algorithm are suggested. Past time operators suitable for writing succinct specifications are introduced and shown definitionally equivalent to the standard operators. This work is part of the PathExplorer project, the objective of which it is to construct a flexible framework for monitoring and analyzing program executions.

Adaptive particle swarm optimization: detection and response to dynamic systems

Abstract: This paper introduces an adaptive PSO, which automatically tracks various changes in a dynamic system. Different environment detection and response techniques are tested on the parabolic and Rosenbrock benchmark functions, and re-randomization is introduced to respond to the dynamic changes. Performance on the benchmark functions with various severities is analyzed.

Algorithms for optimal scheduling and management of hidden Markov model sensors

Abstract: The author considers a hidden Markov model (HMM) where a single Markov chain is observed by a number of noisy sensors. Due to computational or communication constraints, at each time instant, one can select only one of the noisy sensors. The sensor scheduling problem involves designing algorithms for choosing dynamically at each time instant which sensor to select to provide the next measurement. Each measurement has an associated measurement cost. The problem is to select an optimal measurement scheduling policy to minimize a cost function of estimation errors and measurement costs. The optimal measurement policy is solved via stochastic dynamic programming. Sensor management issues and suboptimal scheduling algorithms are also presented. A numerical example that deals with the aircraft identification problem is presented.

A memory-efficient dynamic programming algorithm for optimal alignment of a sequence to an RNA secondary structure.

Abstract: Covariance models (CMs) are probabilistic models of RNA secondary structure, analogous to profile hidden Markov models of linear sequence. The dynamic programming algorithm for aligning a CM to an RNA sequence of length N is O(N3) in memory. This is only practical for small RNAs. I describe a divide and conquer variant of the alignment algorithm that is analogous to memory-efficient Myers/Miller dynamic programming algorithms for linear sequence alignment. The new algorithm has an O(N2 log N) memory complexity, at the expense of a small constant factor in time. Optimal ribosomal RNA structural alignments that previously required up to 150 GB of memory now require less than 270 MB.

Performance analysis of a dynamic programming track before detect algorithm

Abstract: We analyze a dynamic programming (DP)-based track before detect (TBD) algorithm. By using extreme value theory we obtain explicit expressions for various performance measures of the algorithm such as probability of detection and false alarm. Our analysis has two advantages. First the unrealistic Gaussian and independence assumptions used in previous works are not required. Second, the probability of detection and false alarm curves obtained fit computer simulated performance results significantly more accurately than previously proposed analyses of the TBD algorithm.

An Adaptive Dynamic Programming Algorithm for Dynamic Fleet Management, I: Single Period Travel Times

Abstract: In a companion paper (Godfrey and Powell 2002) we introduced an adaptive dynamic programming algorithm for stochastic dynamic resource allocation problems, which arise in the context of logistics and distribution, fleet management, and other allocation problems. The method depends on estimating separable nonlinear approximations of value functions, using a dynamic programming framework. That paper considered only the case in which the time to complete an action was always a single time period. Experiments with this technique quickly showed that when the basic algorithm was applied to problems with multiperiod travel times, the results were very poor. In this paper, we illustrate why this behavior arose, and propose a modified algorithm that addresses the issue. Experimental work demonstrates that the modified algorithm works on problems with multiperiod travel times, with results that are almost as good as the original algorithm applied to single period travel times.

Applying Population Based ACO to Dynamic Optimization Problems

Abstract: Population based ACO algorithms for dynamic optimization problems are studied in this paper. In the population based approach a set of solutions is transferred from one iteration of the algorithm to the next instead of transferring pheromone information as in most ACO algorithms. The set of solutions is then used to compute the pheromone information for the ants of the next iteration. The population based approach can be used to solve dynamic optimization problems when a good solution of the old instance can be modified after a change of the problem instance so that it represents a reasonable solution for the new problem instance. This is tested experimentally for a dynamic TSP and dynamic QAP problem. Moreover the behavior of different strategies for updating the population of solutions are compared.

Dynamic programming for stochastic target problems and geometric flows

Abstract: Given a controlled stochastic process, the reachability set is the collection of all initial data from which the state process can be driven into a target set at a specified time. Differential properties of these sets are studied by the dynamic programming principle which is proved by the Jankov-von Neumann measurable selection theorem. This principle implies that the reachability sets satisfy a geometric partial differential equation, which is the analogue of the Hamilton-Jacobi-Bellman equation for this problem. By appropriately choosing the controlled process, this connection provides a stochastic representation for mean curvature type geometric flows. Another application is the super-replication problem in financial mathematics. Several applications in this direction are also discussed.

Application of dynamic programming to the correlation of paleoclimate records

Abstract: [1] Signal matching is a powerful tool frequently used in paleoclimate research, but it is enormously time-consuming when performed by hand. Previously proposed automatic correlation techniques require very good initial fits to find the correct alignment of two records. A new technique presented in this paper utilizes dynamic programming to find the globally optimal alignment of two records. Geological realism is instilled in the solution through the definition of penalty functions for undesirable behavior such as unlikely changes in accumulation rate. Examples with synthetic and real data demonstrate that the dynamic programming technique produces accurate, high-resolution results with much less effort than hand tuning or preexisting automated correlation techniques.

Optimization of powertrain operating policy for feasibility assessment and calibration: stochastic dynamic programming approach

Abstract: An approach based on stochastic dynamic programming is proposed to develop optimal operating policies for automotive powertrain systems. The goal is to minimize fuel consumption and tailpipe emissions. Unlike in the conventional approach, the minimization is performed not for a predetermined drive cycle, but in a stochastic "average" sense over a class of trajectories from an underlying Markov chain drive cycle generator. The objective of this paper is to introduce the approach and illustrate its applications. with several examples.

Smooth Solutions to Optimal Investment Models with Stochastic Volatilities and Portfolio Constraints

Abstract: . This paper deals with an extension of Merton's optimal investment problem to a multidimensional model with stochastic volatility and portfolio constraints. The classical dynamic programming approach leads to a characterization of the value function as a viscosity solution of the highly nonlinear associated Bellman equation. A logarithmic transformation expresses the value function in terms of the solution to a semilinear parabolic equation with quadratic growth on the derivative term. Using a stochastic control representation and some approximations, we prove the existence of a smooth solution to this semilinear equation. An optimal portfolio is shown to exist, and is expressed in terms of the classical solution to this semilinear equation. This reduction is useful for studying numerical schemes for both the value function and the optimal portfolio. We illustrate our results with several examples of stochastic volatility models popular in the financial literature.

An Approximate Dynamic Programming Approach to Multidimensional Knapsack Problems

Abstract: We present an Approximate Dynamic Programming ADP approach for the multidimensional knapsack problem MKP. We approximate the value function a using parametric and nonparametric methods and busing a base-heuristic. We propose a new heuristic which adaptively rounds the solution of the linear programming relaxation. Our computational study suggests: athe new heuristic produces high quality solutions fast and robustly, bstate of the art commercial packages like CPLEX require significantly larger computational time to achieve the same quality of solutions, c the ADP approach using the new heuristic competes successfully with alternative heuristic methods such as genetic algorithms, dthe ADP approach based on parametric and nonparametric approximations, while producing reasonable solutions, is not competitive. Overall, this research illustrates that the base-heuristic approach is a promising computational approach for MKPs worthy of further investigation.

A Hierarchical Symmetric Stereo Algorithm Using Dynamic Programming

Abstract: In this paper, a new hierarchical stereo algorithm is presented. The algorithm matches individual pixels in corresponding scanlines by minimizing a cost function. Several cost functions are compared. The algorithm achieves a tremendous gain in speed and memory requirements by implementing it hierarchically. The images are downsampled an optimal number of times and the disparity map of a lower level is used as ‘offset’ disparity map at a higher level. An important contribution consists of the complexity analysis of the algorithm. It is shown that this complexity is independent of the disparityrange. This result is also used to determine the optimal number of downsample levels. This speed gain results in the ability to use more complex (compute intensive) cost functions that deliver high quality disparity maps. Another advantage of this algorithm is that cost functions can be chosen independent of the optimisation algorithm. The algorithm in this paper is symmetric, i.e. exactly the same matches are found if left and right image are swapped. Finally, the algorithm was carefully implemented so that a minimal amount of memory is used. It has proven its efficiency on large images with a high disparity range as well as its quality. Examples are given in this paper.

A parallel repair genetic algorithm to solve the unit commitment problem

Abstract: This paper addresses the unit commitment problem of thermal units. This optimization problem is large-scale, combinatorial, mixed-integer, and nonlinear. Exact solution techniques to solve it are not currently available. This paper proposes a novel repair genetic algorithm conducted through heuristics to achieve a near optimal solution to this problem. This optimization technique is directly parallelizable. Three different parallel approaches have been developed. The modeling framework provided by genetic algorithms is less restrictive than the frameworks provided by other approaches such as dynamic programming or Lagrangian relaxation. A state-of-the-art Lagrangian relaxation algorithm is used to appraise the behavior of the proposed parallel genetic algorithm. The computing time requirement to solve problems of realistic size is moderate. The developed genetic algorithm has been successfully applied to realistic case studies.

Convex dynamic programming for hybrid systems

Abstract: A classical linear programming approach to optimization of flow or transportation in a discrete graph is extended to hybrid systems. The problem is finite dimensional if the state space is discrete and finite, but becomes infinite dimensional for a continuous or hybrid state space. It is shown how strict lower bounds on the optimal loss function can be computed by gridding the continuous state space and restricting the linear program to a finite-dimensional subspace. Upper bounds can be obtained by evaluation of the corresponding control laws.

On the Robust Single Machine Scheduling Problem

Abstract: The single machine scheduling problem with sum of completion times criterion (SS) can be solved easily by the Shortest Processing Time (SPT) rule. In the case of significant uncertainty of the processing times, a robustness approach is appropriate. In this paper, we show that the robust version of the (SS) problem is NP-complete even for very restricted cases. We present an algorithm for finding optimal solutions for the robust (SS) problem using dynamic programming. We also provide two polynomial time heuristics and demonstrate their effectiveness.

Dynamic programming in stochastic control of systems with delay

Abstract: We consider optimal control problems for systems described by stochastic differential equations with delay (SDDE). We prove a version of Bellman's principle of optimality (the dynamic programming principle) for a general class of such problems. That the class in general means that both the dynamics and the cost depends on the past in a general way. As an application, we study systems where the value function depends on the past only through some weighted average. For such systems we obtain a Hamilton-Jacobi-Bellman partial differential equation that the value function must solve if it is smooth enough. The weak uniqueness of the SDDEs we consider is our main tool in proving the result. Notions of strong and weak uniqueness for SDDEs are introduced, and we prove that strong uniqueness implies weak uniqueness, just as for ordinary stochastic differential equations.

LQR optimization of linear system switching

Abstract: Considers offline optimization of a switching sequence for a given finite set of linear control systems, together with joint optimization of control laws. A linear quadratic full information criterion is optimized and dynamic programming is used to find an optimal switching sequence and control law. The main result is a method for efficient pruning of the search tree to avoid combinatoric explosion. A method to prove optimality of a found candidate switch sequence and corresponding control laws is presented.

Power management in a hydro-thermal system under uncertainty by Lagrangian relaxation

Abstract: We present a dynamic multistage stochastic programming model for the cost-optimal generation of electric power in a hydro-thermal system under uncertainty in load, inflow to reservoirs and prices for fuel and delivery contracts. The stochastic load process is approximated by a scenario tree obtained by adapting a SARIMA model to historical data, using empirical means and variances of simulated scenarios to construct an initial tree, and reducing it by a scenario deletion procedure based on a suitable probability distance. Our model involves many mixed-integer variables and individual power unit constraints, but relatively few coupling constraints. Hence we employ stochastic Lagrangian relaxation that assigns stochastic multipliers to the coupling constraints. Solving the Lagrangian dual by a proximal bundle method leads to successive decomposition into single thermal and hydro unit subproblems that are solved by dynamic programming and a specialized descent algorithm, respectively. The optimal stochastic multipliers are used in Lagrangian heuristics to construct approximately optimal first stage decisions. Numerical results are presented for realistic data from a German power utility, with a time horizon of one week and scenario numbers ranging from 5 to 100. The corresponding optimization problems have up to 200,000 binary and 350,000 continuous variables, and more than 500,000 constraints.

Dynamic Optimal Groundwater Management with Inclusion of Fixed Costs

Abstract: Obtaining optimal solutions for groundwater resources planning problems, while simultaneously considering both fixed costs and time-varying pumping rates, is a challenging task. Application of conventional optimization algorithms such as linear and nonlinear programming is difficult due to the discontinuity of the fixed cost function in the objective function and the combinatorial nature of assigning discrete well locations. Use of conventional discrete algorithms such as integer programming or discrete dynamic programming is hampered by the large computational burden caused by varying pumping rates over time. A novel procedure that integrates a genetic algorithm ~GA! with constrained differential dynamic programming ~CDDP! calculates optimal solutions for a groundwater resources planning problem while simultaneously considering fixed costs and time-varying pumping rates. The GA determines the number and locations of pumping wells with operating costs then evaluated using CDDP. This study demonstrates that fixed costs associated with installing wells significantly impact the optimal number and locations of wells.

A sub-quadratic sequence alignment algorithm for unrestricted cost matrices

Abstract: The classical algorithm for computing the similarity between two sequences [36, 39] uses a dynamic programming matrix, and compares two strings of size n in O(n2) time. We address the challenge of computing the similarity of two strings in sub-quadratic time, for metrics which use a scoring matrix of unrestricted weights. Our algorithm applies to both local and global alignment computations.The speed-up is achieved by dividing the dynamic programming matrix into variable sized blocks, as induced by Lempel-Ziv parsing of both strings, and utilizing the inherent periodic nature of both strings. This leads to an O(n2/log n) algorithm for an input of constant alphabet size. For most texts, the time complexity is actually O(hn2/log n) where h ≤ 1 is the entropy of the text.

A new integrated approach to dynamic schedule synchronization and energy-saving train control

Abstract: This paper presents a new approach to fulfill conflicting goals of dynamic schedule synchronization and energy saving in rapid rail transit systems. For public transport operators, passenger transfers between mass rapid transit systems and road-bound means of transport are unavoidable to ensure efficient operation. But for passengers transfering between different modes of transport, it can be a major annoyance when connections are missed. This work aims to propose an algorithm for the dynamic modification of train running times in such a way that the probability of getting connections to other means of public transport can be increased and overall energy consumption of train operation remains low. An optimal train timetable can be computed in real-time using the dynamic programming method (Bellman). Energy consumption and waiting time due to missing a connection form the bicriterion optimization function. Simulation with MATLAB/SIMULINK has shown the feasibility of the algorithm within an assumed automatic train operation environment. A driver-assistance system will be developed to ensure the feasibility of the proposed method for manually driven trains. This system is being tested in a train simulator at Dresden University of Technology and will be used on demonstrator trains of the Dresden suburban railway by 2004.

Optimal and Hierarchical Controls in Dynamic Stochastic Manufacturing Systems: A Survey

Abstract: Most manufacturing systems are large and complex and operate in an uncertain environment. One approach to managing such systems is that of hierarchical decomposition. This paper reviews the research devoted to proving that a hierarchy based on the frequencies of occurrence of different types of events in the systems results in decisions that are asymptotically optimal as the rates of some events become large compared to those of others. The paper also reviews the research on stochastic optimal control problems associated with manufacturing systems, their dynamic programming equations, existence of solutions of these equations, and verification theorems of optimality for the systems. Manufacturing systems that are addressed include single-machine systems, dynamic flowshops, and dynamic jobshops producing multiple products. These systems may also incorporate random production capacity and demands, and decisions such as production rates, capacity expansion, and promotional campaigns. Related computational results and areas of applications are also presented. A more detailed survey is available at .

Dynamic programming for parsing and estimation of stochastic unification-based grammars

Abstract: Stochastic unification-based grammars (SUBGs) define exponential distributions over the parses generated by a unification-based grammar (UBG). Existing algorithms for parsing and estimation require the enumeration of all of the parses of a string in order to determine the most likely one, or in order to calculate the statistics needed to estimate a grammar from a training corpus. This paper describes a graph-based dynamic programming algorithm for calculating these statistics from the packed UBG parse representations of Maxwell and Kaplan (1995) which does not require enumerating all parses. Like many graphical algorithms, the dynamic programming algorithm's complexity is worst-case exponential, but is often polynomial. The key observation is that by using Maxwell and Kaplan packed representations, the required statistics can be rewritten as either the max or the sum of a product of functions. This is exactly the kind of problem which can be solved by dynamic programming over graphical models.