A unified view of metaheuristics This book provides a complete background on metaheuristics and shows readers how to design and implement efficient algorithms to solve complex optimization problems across a diverse range of applications, from networking and bioinformatics to engineering design, routing, and scheduling. It presents the main design questions for all families of metaheuristics and clearly illustrates how to implement the algorithms under a software framework to reuse both the design and code. Throughout the book, the key search components of metaheuristics are considered as a toolbox for: Designing efficient metaheuristics (e.g. local search, tabu search, simulated annealing, evolutionary algorithms, particle swarm optimization, scatter search, ant colonies, bee colonies, artificial immune systems) for optimization problems Designing efficient metaheuristics for multi-objective optimization problems Designing hybrid, parallel, and distributed metaheuristics Implementing metaheuristics on sequential and parallel machines Using many case studies and treating design and implementation independently, this book gives readers the skills necessary to solve large-scale optimization problems quickly and efficiently. It is a valuable reference for practicing engineers and researchers from diverse areas dealing with optimization or machine learning; and graduate students in computer science, operations research, control, engineering, business and management, and applied mathematics.

/pdf/metaheuristics-from-design-to-implementation-1vqdfv8nk9.pdf

Metaheuristics: From Design to Implementation

I What Are the Ages of My Three Sons?.- 1 Why Are Some Problems Difficult to Solve?.- II How Important Is a Model?.- 2 Basic Concepts.- III What Are the Prices in 7-11?.- 3 Traditional Methods - Part 1.- IV What Are the Numbers?.- 4 Traditional Methods - Part 2.- V What's the Color of the Bear?.- 5 Escaping Local Optima.- VI How Good Is Your Intuition?.- 6 An Evolutionary Approach.- VII One of These Things Is Not Like the Others.- 7 Designing Evolutionary Algorithms.- VIII What Is the Shortest Way?.- 8 The Traveling Salesman Problem.- IX Who Owns the Zebra?.- 9 Constraint-Handling Techniques.- X Can You Tune to the Problem?.- 10 Tuning the Algorithm to the Problem.- XI Can You Mate in Two Moves?.- 11 Time-Varying Environments and Noise.- XII Day of the Week of January 1st.- 12 Neural Networks.- XIII What Was the Length of the Rope?.- 13 Fuzzy Systems.- XIV Everything Depends on Something Else.- 14 Coevolutionary Systems.- XV Who's Taller?.- 15 Multicriteria Decision-Making.- XVI Do You Like Simple Solutions?.- 16 Hybrid Systems.- 17 Summary.- Appendix A: Probability and Statistics.- A.1 Basic concepts of probability.- A.2 Random variables.- A.2.1 Discrete random variables.- A.2.2 Continuous random variables.- A.3 Descriptive statistics of random variables.- A.4 Limit theorems and inequalities.- A.5 Adding random variables.- A.6 Generating random numbers on a computer.- A.7 Estimation.- A.8 Statistical hypothesis testing.- A.9 Linear regression.- A.10 Summary.- Appendix B: Problems and Projects.- B.1 Trying some practical problems.- B.2 Reporting computational experiments with heuristic methods.- References.

/pdf/how-to-solve-it-modern-heuristics-1pu40hrf57.pdf

How to Solve It: Modern Heuristics

To date, reinforcement learning has mostly been studied solving simple learning tasks. Reinforcement learning methods that have been studied so far typically converge slowly. The purpose of this work is thus two-fold: 1) to investigate the utility of reinforcement learning in solving much more complicated learning tasks than previously studied, and 2) to investigate methods that will speed up reinforcement learning.

This paper compares eight reinforcement learning frameworks: adaptive heuristic critic (AHC) learning due to Sutton, Q-learning due to Watkins, and three extensions to both basic methods for speeding up learning. The three extensions are experience replay, learning action models for planning, and teaching. The frameworks were investigated using connectionism as an approach to generalization. To evaluate the performance of different frameworks, a dynamic environment was used as a testbed. The environment is moderately complex and nondeterministic. This paper describes these frameworks and algorithms in detail and presents empirical evaluation of the frameworks.

/pdf/self-improving-reactive-agents-based-on-reinforcement-4bh4oqtu6n.pdf

Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching

Evolutionary algorithms often have to solve optimization problems in the presence of a wide range of uncertainties. Generally, uncertainties in evolutionary computation can be divided into the following four categories. First, the fitness function is noisy. Second, the design variables and/or the environmental parameters may change after optimization, and the quality of the obtained optimal solution should be robust against environmental changes or deviations from the optimal point. Third, the fitness function is approximated, which means that the fitness function suffers from approximation errors. Fourth, the optimum of the problem to be solved changes over time and, thus, the optimizer should be able to track the optimum continuously. In all these cases, additional measures must be taken so that evolutionary algorithms are still able to work satisfactorily. This paper attempts to provide a comprehensive overview of the related work within a unified framework, which has been scattered in a variety of research areas. Existing approaches to addressing different uncertainties are presented and discussed, and the relationship between the different categories of uncertainties are investigated. Finally, topics for future research are suggested.

/pdf/evolutionary-optimization-in-uncertain-environments-a-survey-1cgcnwsuec.pdf

Evolutionary optimization in uncertain environments-a survey

In many classifier systems, the classifier strength parameter serves as a predictor of future payoff and as the classifier's fitness for the genetic algorithm. We investigate a classifier system, XCS, in which each classifier maintains a prediction of expected payoff, but the classifier's fitness is given by a measure of the prediction's accuracy. The system executes the genetic algorithm in niches defined by the match sets, instead of panmictically. These aspects of XCS result in its population tending to form a complete and accurate mapping X × A → P from inputs and actions to payoff predictions. Further, XCS tends to evolve classifiers that are maximally general, subject to an accuracy criterion. Besides introducing a new direction for classifier system research, these properties of XCS make it suitable for a wide range of reinforcement learning situations where generalization over states is desirable.

/pdf/classifier-fitness-based-on-accuracy-2xkfr7ankf.pdf

Classifier fitness based on accuracy

The problem of learning decision rules for sequential tasks is addressed, focusing on the problem of learning tactical decision rules from a simple flight simulator. The learning method relies on the notion of competition and employs genetic algorithms to search the space of decision policies. Several experiments are presented that address issues arising from differences between the simulation model on which learning occurs and the target environment on which the decision rules are ultimately tested.

/pdf/learning-sequential-decision-rules-using-simulation-models-30uh67gazb.pdf

Learning Sequential Decision Rules Using Simulation Models and Competition

In this paper, we introduce a case-based method of initializing genetic algorithms that are used to guide search in changing environments. This is incorporated in an anytime learning system. Anytime learning is a general approach to continuous learning in a changing environment. The agent’s learning module continuously tests new strategies against a simulation model of the task environment, and dynamically updates the knowledge base used by the agent on the basis of the results. The execution module includes a monitor that can dynamically modify the simulation model based on its observations of the external environment; an update to the simulation model causes the learning system to restart learning. Previous work has shown that genetic algorithms provide an appropriate search mechanism for anytime learning. This paper extends the approach by including strategies, which are learned under similar environmental conditions, in the initial population of the genetic algorithm. Experiments show that case-based initialization of the population results in a significantly improved performance.

Case-Based Initialization of Genetic Algorithms

The majority of current genetic algorithms (GAs), while inspired by natural evolutionary systems, are seldom viewed as biologically plausible models. This is not a criticism of GAs, but rather a reflection of choices made regarding the level of abstraction at which biological mechanisms are modeled, and a reflection of the more engineering-oriented goals of the evolutionary computation community. Understanding better and reducing this gap between GAs and genetics has been a central issue in an interdisciplinary project whose goal is to build GA-based computational models of viral evolution. The result is a system called Virtual Virus (VIV). VIV incorporates a number of more biologically plausible mechanisms, including a more flexible genotype-to-phenotype mapping. In VIV the genes are independent of position, and genomes can vary in length and may contain noncoding regions, as well as duplicative or competing genes.

Initial computational studies with VIV have already revealed several emergent phenomena of both biological and computational interest. In the absence of any penalty based on genome length, VIV develops individuals with long genomes and also performs more poorly (from a problem-solving viewpoint) than when a length penalty is used. With a fixed linear length penalty, genome length tends to increase dramatically in the early phases of evolution and then decrease to a level based on the mutation rate. The plateau genome length (i.e., the average length of individuals in the final population) generally increases in response to an increase in the base mutation rate. When VIV converges, there tend to be many copies of good alternative genes within the individuals. We observed many instances of switching between active and inactive genes during the entire evolutionary process. These observations support the conclusion that noncoding regions serve as scratch space in which VIV can explore alternative gene values. These results represent a positive step in understanding how GAs might exploit more of the power and flexibility of biological evolution while simultaneously providing better tools for understanding evolving biological systems.

Putting more genetics into genetic algorithms

Anytime learning is a general approach to continuous learning in a changing environment. The agent's learning module continuously tests new strategies against a simulation model of the task environment, and dynamically updates the knowledge base used by the agent on the basis of the results. The execution module controls the agent's interaction with the environment, and includes a monitor that can dynamically modify the simulation model based on its observations of the environment. When the simulation model is modified, the learning process is restarted on the modified model. The learning system is assumed to operate indefinitely, and the execution system uses the results of learning as they become available. An experimental study tests one of the key aspects of this design using a two-agent cat-and-mouse game as the task environment.

An Approach to Anytime Learning

The problem of learning decision rules for sequential tasks is addressed, focusing on the problem of learning tactical plans from a simple flight simulator where a plane must avoid a missile. The learning method relies on the notion of competition and employs genetic algorithms to search the space of decision policies. Experiments are presented that address issues arising from differences between the simulation model on which learning occurs and the target environment on which the decision rules are ultimately tested. Specifically, either the model or the target environment may contain noise. These experiments examine the effect of learning tactical plans without noise and then testing the plans in a noisy environment, and the effect of learning plans in a noisy simulator and then testing the plans in a noise-free environment. Empirical results show that, while best result are obtained when the training model closely matches the target environment, using a training environment that is more noisy than the target environment is better than using using a training environment that has less noise than the target environment.

Connie Loggia Ramsey

Papers

Learning Sequential Decision Rules Using Simulation Models and Competition

Case-Based Initialization of Genetic Algorithms

Putting more genetics into genetic algorithms

An Approach to Anytime Learning

Simulation-assisted learning by competition: effects of noise differences between training model and target environment