Showing papers on "Dynamic programming published in 2008"

A Novel Infinite-Time Optimal Tracking Control Scheme for a Class of Discrete-Time Nonlinear Systems via the Greedy HDP Iteration Algorithm

Abstract: In this paper, we aim to solve the infinite-time optimal tracking control problem for a class of discrete-time nonlinear systems using the greedy heuristic dynamic programming (HDP) iteration algorithm. A new type of performance index is defined because the existing performance indexes are very difficult in solving this kind of tracking problem, if not impossible. Via system transformation, the optimal tracking problem is transformed into an optimal regulation problem, and then, the greedy HDP iteration algorithm is introduced to deal with the regulation problem with rigorous convergence analysis. Three neural networks are used to approximate the performance index, compute the optimal control policy, and model the nonlinear system for facilitating the implementation of the greedy HDP iteration algorithm. An example is given to demonstrate the validity of the proposed optimal tracking control scheme.

On the Choice-Based Linear Programming Model for Network Revenue Management

Abstract: Gallego et al. [Gallego, G., G. Iyengar, R. Phillips, A. Dubey. 2004. Managing flexible products on a network. CORC Technical Report TR-2004-01, Department of Industrial Engineering and Operations Research, Columbia University, New York.] recently proposed a choice-based deterministic linear programming model (CDLP) for network revenue management (RM) that parallels the widely used deterministic linear programming (DLP) model. While they focused on analyzing “flexible products”---a situation in which the provider has the flexibility of using a collection of products (e.g., different flight times and/or itineraries) to serve the same market demand (e.g., an origin-destination connection)---their approach has broader implications for understanding choice-based RM on a network. In this paper, we explore the implications in detail. Specifically, we characterize optimal offer sets (sets of available network products) by extending to the network case a notion of “efficiency” developed by Talluri and van Ryzin [Talluri, K. T., G. J. van Ryzin. 2004. Revenue management under a general discrete choice model of consumer behavior. Management Sci.50 15--33.] for the single-leg, choice-based RM problem. We show that, asymptotically, as demand and capacity are scaled up, only these efficient sets are used in an optimal policy. This analysis suggests that efficiency is a potentially useful approach for identifying “good” offer sets on networks, as it is in the case of single-leg problems. Second, we propose a practical decomposition heuristic for converting the static CDLP solution into a dynamic control policy. The heuristic is quite similar to the familiar displacement-adjusted virtual nesting (DAVN) approximation used in traditional network RM, and it significantly improves on the performance of the static LP solution. We illustrate the heuristic on several numerical examples.

Optimal and approximate Q-value functions for decentralized POMDPs

Abstract: Decision-theoretic planning is a popular approach to sequential decision making problems, because it treats uncertainty in sensing and acting in a principled way. In single-agent frameworks like MDPs and POMDPs, planning can be carried out by resorting to Q-value functions: an optimal Q-value function Q* is computed in a recursive manner by dynamic programming, and then an optimal policy is extracted from Q*. In this paper we study whether similar Q-value functions can be defined for decentralized POMDP models (Dec-POMDPs), and how policies can be extracted from such value functions. We define two forms of the optimal Q-value function for Dec-POMDPs: one that gives a normative description as the Q-value function of an optimal pure joint policy and another one that is sequentially rational and thus gives a recipe for computation. This computation, however, is infeasible for all but the smallest problems. Therefore, we analyze various approximate Q-value functions that allow for efficient computation. We describe how they relate, and we prove that they all provide an upper bound to the optimal Q-value function Q*. Finally, unifying some previous approaches for solving Dec-POMDPs, we describe a family of algorithms for extracting policies from such Q-value functions, and perform an experimental evaluation on existing test problems, including a new firefighting benchmark problem.

Optimizing Hydropower Reservoir Operation Using Hybrid Genetic Algorithm and Chaos

Abstract: Genetic algorithms (GA) have been widely applied to solve water resources system optimization. With the increase of the complexity and the larger problem scale of water resources system, GAs are most frequently faced with the problems of premature convergence, slow iterations to reach the global optimal solution and getting stuck at a local optimum. A novel chaos genetic algorithm (CGA) based on the chaos optimization algorithm (COA) and genetic algorithm (GA), which makes use of the ergodicity and internal randomness of chaos iterations, is presented to overcome premature local optimum and increase the convergence speed of genetic algorithm. CGA integrates powerful global searching capability of the GA with that of powerful local searching capability of the COA. Two measures are adopted in order to improve the performance of the GA. The first one is the adoption of chaos optimization of the initialization to improve species quality and to maintain the population diversity. The second is the utilization of annealing chaotic mutation operation to replace standard mutation operator in order to avoid the search being trapped in local optimum. The Rosenbrock function and Schaffer function, which are complex and global optimum functions and often used as benchmarks for contemporary optimization algorithms for GAs and Evolutionary computation, are first employed to examine the performance of the GA and CGA. The test results indicate that CGA can improve convergence speed and solution accuracy. Furthermore, the developed model is applied for the monthly operation of a hydropower reservoir with a series of monthly inflow of 38 years. The results show that the long term average annual energy based CGA is the best and its convergent speed not only is faster than dynamic programming largely, but also overpasses the standard GA. Thus, the proposed approach is feasible and effective in optimal operations of complex reservoir systems.

New dynamic programming algorithms for the resource constrained elementary shortest path problem

Abstract: The resource constrained elementary shortest path problem (RCESPP) arises as a pricing subproblem in branch-and-price algorithms for vehicle-routing problems with additional constraints. We address the optimization of the RCESPP and we present and compare three methods. The first method is a well-known exact dynamic-programming algorithm improved by new ideas, such as bidirectional search with resource-based bounding. The second method consists in a branch-and-bound algorithm, where lower bounds are computed by dynamic-programming with state-space relaxation; we show how bounded bidirectional search can be adapted to state-space relaxation and we present different branching strategies and their hybridization. The third method, called decremental state-space relaxation, is a new one; exact dynamic-programming and state-space relaxation are two special cases of this new method. The experimental comparison of the three methods is definitely favorable to decrement state-space relaxation. Computational results are given for different kinds of resources, arising from the capacitated vehicle-routing problem, the vehicle-routing problem with distribution and collection, and the vehicle-routing problem with capacities and time windows. © 2007 Wiley Periodicals, Inc. NETWORKS, 2008

Free Space Computation Using Stochastic Occupancy Grids and Dynamic Programming

Abstract: The computation of free space available in an environment is an essential task for many intelligent automotive and robotic applications. This paper proposes a new approach, which builds a stochastic occupancy grid to address the free space problem as a dynamic programming task. Stereo measurements are integrated over time reducing disparity uncertainty. These integrated measurements are entered into an occupancy grid, taking into account the noise properties of the measurements. In order to cope with real-time requirements of the application, three occupancy grid types are proposed. Their applicabilities and implementations are also discussed. Experimental results with real stereo sequences show the robustness and accuracy of the method. The current implementation of the method runs on off-the-shelf hardware at 20 Hz.

Relaxations of Weakly Coupled Stochastic Dynamic Programs

Abstract: We consider a broad class of stochastic dynamic programming problems that are amenable to relaxation via decomposition. These problems comprise multiple subproblems that are independent of each other except for a collection of coupling constraints on the action space. We fit an additively separable value function approximation using two techniques, namely, Lagrangian relaxation and the linear programming (LP) approach to approximate dynamic programming. We prove various results comparing the relaxations to each other and to the optimal problem value. We also provide a column generation algorithm for solving the LP-based relaxation to any desired optimality tolerance, and we report on numerical experiments on bandit-like problems. Our results provide insight into the complexity versus quality trade-off when choosing which of these relaxations to implement.

An improved dynamic programming algorithm for coalition structure generation

Abstract: Forming effective coalitions is a major research challenge in the field of multi-agent systems. Central to this endeavour is the problem of partitioning the set of agents into exhaustive and disjoint coalitions such that the social welfare is maximized. This coalition structure generation problem is extremely challenging due to the exponential number of partitions that need to be examined. Specifically, given n agents, there are O(nn) possible partitions. To date, the only algorithm that can find an optimal solution in O(3n) is the Dynamic Programming (DP) algorithm, due to Rothkopf et al. However, one of the main limitations of DP is that it requires a significant amount of memory. In this paper, we devise an Improved Dynamic Programming algorithm (IDP) that is proved to perform fewer operations than DP (e.g. 38.7% of the operations given 25 agents), and is shown to use only 33.3% of the memory in the best case, and 66.6% in the worst.

Algorithms for Itinerary Planning in Multimodal Transportation Networks

Abstract: The itinerary planning problem in an urban public transport system constitutes a common routing and scheduling decision faced by travelers. The objective of this paper is to present a new formulation and an algorithm for solving the itinerary planning problem, i.e., determination of the itinerary that lexicographically optimizes a set of criteria (i.e., total travel time, number of transfers, and total walking and waiting time) while departing from the origin and arriving at the destination within specified time windows. Based on the proposed formulation, the itinerary planning problem is expressed as a shortest path problem in a multimodal time-schedule network with time windows and time-dependent travel times. A dynamic programming-based algorithm has been developed for the solution of the emerging problem. The special case of the problem involving a mandatory visit at an intermediate stop within a given time window is formulated as two nested itinerary planning problems which are solved by the aforementioned algorithm. The proposed algorithm has been integrated in a Web-based journey planning system, whereas its performance has been assessed by solving real-life itinerary planning problems defined on the Athens urban public transport network, providing fast and accurate solutions.

Optimal Restoration of Distribution Systems Using Dynamic Programming

Abstract: This paper solves the distribution system restoration problem using dynamic programming with state reduction. The algorithm is an operator-permissive, automated approach to the restoration of distribution systems after a blackout. The timing and selection of feeders to be energized are represented as states in a dynamic programming formulation. An enhanced dynamic programming method reduces the number of states by grouping states that are close to each other and selecting the best state. The algorithm was tested on an 8 feeder/32 load distribution system. The method is applicable to radially configured systems.

Multi-Dimensional Transitional Dynamics: A Simple Numerical Procedure

Abstract: We propose the relaxation algorithm as a simple and powerful method for simulating the transition process in growth models. This method has a number of important advantages: (1) It can easily deal with a wide range of dynamic systems including stiff differential equations and systems giving rise to a continuum of stationary equilibria. (2) The application of the procedure is fairly user friendly. The only input required consists of the dynamic system. (3) The variant of the relaxation algorithm we propose exploits in a natural manner the infinite time horizon, which usually underlies optimal control problems in economics. As an illustrative application, we simulate the transition process of the Jones (1995) and the Lucas (1988) model.

Shortest path stochastic control for hybrid electric vehicles

Abstract: When a hybrid electric vehicle (HEV) is certified for emissions and fuel economy, its power management system must be charge sustaining over the drive cycle, meaning that the battery state of charge (SOC) must be at least as high at the end of the test as it was at the beginning of the test. During the test cycle, the power management system is free to vary the battery SOC so as to minimize a weighted combination of fuel consumption and exhaust emissions. This paper argues that shortest path stochastic dynamic programming (SP-SDP) offers a more natural formulation of the optimal control problem associated with the design of the power management system because it allows deviations of battery SOC from a desired setpoint to be penalized only at key off. This method is illustrated on a parallel hybrid electric truck model that had previously been analyzed using infinite-horizon stochastic dynamic programming with discounted future cost. Both formulations of the optimization problem yield a time-invariant causal state-feedback controller that can be directly implemented on the vehicle. The advantages of the shortest path formulation include that a single tuning parameter is needed to trade off fuel economy and emissions versus battery SOC deviation, as compared with two parameters in the discounted, infinite-horizon case, and for the same level of complexity as a discounted future-cost controller, the shortest-path controller demonstrates better fuel and emission minimization while also achieving better SOC control when the vehicle is turned off. Linear programming is used to solve both stochastic dynamic programs. Copyright © 2007 John Wiley & Sons, Ltd.

Direct Heuristic Dynamic Programming for Damping Oscillations in a Large Power System

Abstract: This paper applies a neural-network-based approximate dynamic programming method, namely, the direct heuristic dynamic programming (direct HDP), to a large power system stability control problem. The direct HDP is a learning- and approximation-based approach to addressing nonlinear coordinated control under uncertainty. One of the major design parameters, the controller learning objective function, is formulated to directly account for network-wide low-frequency oscillation with the presence of nonlinearity, uncertainty, and coupling effect among system components. Results include a novel learning control structure based on the direct HDP with applications to two power system problems. The first case involves static var compensator supplementary damping control, which is used to provide a comprehensive evaluation of the learning control performance. The second case aims at addressing a difficult complex system challenge by providing a new solution to a large interconnected power network oscillation damping control problem that frequently occurs in the China Southern Power Grid.

Pricing Asset Scheduling Flexibility using Optimal Switching

Abstract: We study the financial engineering aspects of operational flexibility of energy assets The current practice relies on a representation that uses strips of European spark‐spread options, ignoring the operational constraints Instead, we propose a new approach based on a stochastic impulse control framework The model reduces to a cascade of optimal stopping problems and directly demonstrates that the optimal dispatch policies can be described with the aid of ‘switching boundaries’, similar to the free boundaries of standard American options Our main contribution is a new method of numerical solution relying on Monte Carlo regressions The scheme uses dynamic programming to efficiently approximate the optimal dispatch policy along the simulated paths Convergence analysis is carried out and results are illustrated with a variety of concrete computational examples We benchmark and compare our scheme with alternative numerical methods

Design optimization of wastewater collection networks by PSO

Abstract: Optimal design of wastewater collection networks is addressed in this paper by making use of the so-called PSO (Particle Swarm Optimization) technique. This already popular evolutionary technique is adapted for dealing both with continuous and discrete variables as required by this problem. An example of a wastewater collection network is used to show the algorithm performance and the obtained results are compared with those given by using dynamic programming to solve the same problem under the same conditions. PSO is shown to be a promising method to solve optimal design problems regarding, in particular, wastewater collection networks, according to the results herein obtained.

Coalition structure generation: dynamic programming meets anytime optimization

Abstract: Coalition structure generation involves partitioning a set of agents into exhaustive and disjoint coalitions so as to maximize the social welfare. What makes this such a challenging problem is that the number of possible solutions grows exponentially as the number of agents increases. To date, two main approaches have been developed to solve this problem, each with its own strengths and weaknesses. The state of the art in the first approach is the Improved Dynamic Programming (IDP) algorithm, due to Rahwan and Jennings, that is guaranteed to find an optimal solution in O(3n), but which cannot generate a solution until it has completed its entire execution. The state of the art in the second approach is an anytime algorithm called IP, due to Rahwan et aI., that provides worst-case guarantees on the quality of the best solution found so far, but which is O(nn). In this paper, we develop a novel algorithm that combines both IDP and IP, resulting in a hybrid performance that exploits the strength of both algorithms and, at the same, avoids their main weaknesses. Our approach is also significantly faster (e.g. given 25 agents, it takes only 28% of the time required by IP, and 0.3% of the time required by IDP).

Dynamic programming strikes back

Abstract: Two highly efficient algorithms are known for optimally ordering joins while avoiding cross products: DPccp, which is based on dynamic programming, and Top-Down Partition Search, based on memoization. Both have two severe limitations: They handle only (1) simple (binary) join predicates and (2) inner joins. However, real queries may contain complex join predicates, involving more than two relations, and outer joins as well as other non-inner joins.Taking the most efficient known join-ordering algorithm, DPccp, as a starting point, we first develop a new algorithm, DPhyp, which is capable to handle complex join predicates efficiently. We do so by modeling the query graph as a (variant of a) hypergraph and then reason about its connected subgraphs. Then, we present a technique to exploit this capability to efficiently handle the widest class of non-inner joins dealt with so far. Our experimental results show that this reformulation of non-inner joins as complex predicates can improve optimization time by orders of magnitude, compared to known algorithms dealing with complex join predicates and non-inner joins. Once again, this gives dynamic programming a distinct advantage over current memoization techniques.

Multiple Method 2-D Trajectory Optimization Satisfying Waypoints and No-Fly Zone Constraints

Abstract: Minimum time to target is one of the primary goals of a global strike mission. The Hypersonic Cruise Vehicle and the Common Aero Vehicle are currently being investigated for mission effectiveness. Additional mission requirements include passage through intermediate waypoints and avoidance of no-fly zones. Thus, a real-time or near real-time autonomous trajectory generation technique is desired to minimize the flight time, satisfy terminal and multiple intermediate state constraints, and remain within specified control limitations. The research herein presents a baseline technique, an analytical geometric trajectory optimization technique, and a dynamic optimization technique. Numerical examples for constant speed trajectories as well as decelerating flight are used to demonstrate and compare the presented techniques. These results show the significant time savings achievable through optimization, the accuracy and computation efficiency of the geometric solution, and the robustness and application of the dynamic optimization technique.

Congestion-Constrained Layer Assignment for Via Minimization in Global Routing

Abstract: In this paper, we study the problem of layer assignment for via minimization, which arises during multilayer global routing. In addressing this problem, we take the total overflow and the maximum overflow as the congestion constraints from a given one-layer global routing solution and aim to find a layer assignment result for each net such that the via cost is minimized while the given congestion constraints are satisfied. To solve the problem, we propose a polynomial-time algorithm which first generates a net order and then performs layer assignment one net at a time according to the order using dynamic programming. Our algorithm is guaranteed to generate a layer assignment solution satisfying the given congestion constraints. We used the six-layer benchmarks released from the ISPD'07 global routing contest to test our algorithm. The experimental results show that our algorithm was able to improve the contest results of the top three winners MaizeRouter, BoxRouter, and FGR on each benchmark. As compared to BoxRouter 2.0 and FGR 1.1, which are newer versions of BoxRouter and FGR, our algorithm respectively produced smaller via costs on all benchmarks and half the benchmarks. Our algorithm can also be adapted to refine a given multilayer global routing solution in a net-by-net manner, and the experimental results show that this refinement approach improved the via costs on all benchmarks for FGR 1.1.

Hamilton–Jacobi–Bellman Equations and Approximate Dynamic Programming on Time Scales

Abstract: The time scales calculus is a key emerging area of mathematics due to its potential use in a wide variety of multidisciplinary applications. We extend this calculus to approximate dynamic programming (ADP). The core backward induction algorithm of dynamic programming is extended from its traditional discrete case to all isolated time scales. Hamilton-Jacobi-Bellman equations, the solution of which is the fundamental problem in the field of dynamic programming, are motivated and proven on time scales. By drawing together the calculus of time scales and the applied area of stochastic control via ADP, we have connected two major fields of research.

Optimization models for the dynamic facility location and allocation problem

Abstract: The design of logistic distribution systems is one of the most critical and strategic issues in industrial facility management. The aim of this study is to develop and apply innovative mixed integer programming optimization models to design and manage dynamic (i.e. multi-period) multi-stage and multi-commodity location allocation problems (LAP). LAP belong to the NP-hard complexity class of decision problems, and the generic occurrence requires simultaneous determination of the number of logistic facilities (e.g. production plants, warehousing systems, distribution centres), their locations, and assignment of customer demand to them. The proposed models use a mixed integer linear programming solver to find solutions in complex industrial applications even when several entities are involved (production plants, distribution centres, customers, etc.). Lastly, the application of the proposed models to a significant case study is presented and discussed.

An Improved Genetic Algorithm-Simulated Annealing Hybrid Algorithm for the Optimization of Multiple Reservoirs

Abstract: A hybrid evolutionary search algorithm is developed to optimize the classical single-criterion operation of multi-reservoir systems. The proposed improved genetic algorithm-simulated annealing (IGA-SA) which combines genetic algorithms (GAs) and the simulated annealing (SA) is a new global optimization algorithm. The algorithm is capable of overcoming the premature convergence of GAs and escaping from local optimal solutions. In addition, it is faster than a traditional unimproved GA-SA algorithm. A case study of optimization operation on generation electricity of a 3-reservoir system in series over 41-year (from May 1940 to April 1981) time periods in Wujiang River, one branch of Yangtze River in China, was performed. The objective is to maximize generation output from the system over each 12-month operating periods. Trade-off analyses on binary coding representation and real-value coding representation of GAs are performed. Sensitivity to some parameters of the GA, the SA and the IGA-SA is analyzed, respectively, and the appropriate values of parameters are suggested. The performance of the proposed algorithm is compared with that of the existing genetic algorithm, the simulated annealing and the dynamic programming (DP). Results demonstrate that the GA is better than the DP, the SA performs better than the GA and the IGA-SA is more efficient than SA. The IGA-SA produces higher quality solutions and costs less computation time compared with the traditional GA-SA. The results obtained from these applications have proved that the IGA-SA has the ability of addressing large and complex problems and is a new promising search algorithm for multi-reservoir optimization problems.