We describe and evaluate the algorithmic techniques that are used in the FF planning system. Like the HSP system, FF relies on forward state space search, using a heuristic that estimates goal distances by ignoring delete lists. Unlike HSP's heuristic, our method does not assume facts to be independent. We introduce a novel search strategy that combines hill-climbing with systematic search, and we show how other powerful heuristic information can be extracted and used to prune the search space. FF was the most successful automatic planner at the recent AIPS-2000 planning competition. We review the results of the competition, give data for other benchmark domains, and investigate the reasons for the runtime performance of FF compared to HSP.

/pdf/the-ff-planning-system-fast-plan-generation-through-3vgjtylx0c.pdf

The FF planning system: fast plan generation through heuristic search

This paper presents a new approach to hierarchical reinforcement learning based on decomposing the target Markov decision process (MDP) into a hierarchy of smaller MDPs and decomposing the value function of the target MDP into an additive combination of the value functions of the smaller MDPs. The decomposition, known as the MAXQ decomposition, has both a procedural semantics--as a subroutine hierarchy--and a declarative semantics--as a representation of the value function of a hierarchical policy. MAXQ unifies and extends previous work on hierarchical reinforcement learning by Singh, Kaelbling, and Dayan and Hinton. It is based on the assumption that the programmer can identify useful subgoals and define subtasks that achieve these subgoals. By defining such subgoals, the programmer constrains the set of policies that need to be considered during reinforcement learning. The MAXQ value function decomposition can represent the value function of any policy that is consistent with the given hierarchy. The decomposition also creates opportunities to exploit state abstractions, so that individual MDPs within the hierarchy can ignore large parts of the state space. This is important for the practical application of the method. This paper defines the MAXQ hierarchy, proves formal results on its representational power, and establishes five conditions for the safe use of state abstractions. The paper presents an online model-free learning algorithm, MAXQ-Q, and proves that it converges with probability 1 to a kind of locally-optimal policy known as a recursively optimal policy, even in the presence of the five kinds of state abstraction. The paper evaluates the MAXQ representation and MAXQ-Q through a series of experiments in three domains and shows experimentally that MAXQ-Q (with state abstractions) converges to a recursively optimal policy much faster than flat Q learning. The fact that MAXQ learns a representation of the value function has an important benefit: it makes it possible to compute and execute an improved, non-hierarchical policy via a procedure similar to the policy improvement step of policy iteration. The paper demonstrates the effectiveness of this nonhierarchical execution experimentally. Finally, the paper concludes with a comparison to related work and a discussion of the design tradeoffs in hierarchical reinforcement learning.

/pdf/hierarchical-reinforcement-learning-with-the-maxq-value-2hsm3smvjf.pdf

Hierarchical reinforcement learning with the MAXQ value function decomposition

Intelligent agents are employed as the central characters in this new introductory text. Beginning with elementary reactive agents, Nilsson gradually increases their cognitive horsepower to illustrate the most important and lasting ideas in AI. Neural networks, genetic programming, computer vision, heuristic search, knowledge representation and reasoning, Bayes networks, planning, and language understanding are each revealed through the growing capabilities of these agents. The book provides a refreshing and motivating new synthesis of the field by one of AI's master expositors and leading researchers. Artificial Intelligence: A New Synthesis takes the reader on a complete tour of this intriguing new world of AI.

* An evolutionary approach provides a unifying theme 
* Thorough coverage of important AI ideas, old and new
* Frequent use of examples and illustrative diagrams
* Extensive coverage of machine learning methods throughout the text
* Citations to over 500 references
* Comprehensive index

Table of Contents

1 Introduction
2 Stimulus-Response Agents
3 Neural Networks 
4 Machine Evolution 
5 State Machines 
6 Robot Vision 
7 Agents that Plan 
8 Uninformed Search 
9 Heuristic Search 
10 Planning, Acting, and Learning 
11 Alternative Search Formulations and Applications 
12 Adversarial Search 
13 The Propositional Calculus 
14 Resolution in The Propositional Calculus 
15 The Predicate Calculus 
16 Resolution in the Predicate Calculus 
17 Knowledge-Based Systems 
18 Representing Commonsense Knowledge 
19 Reasoning with Uncertain Information 
20 Learning and Acting with Bayes Nets 
21 The Situation Calculus 
22 Planning 
23 Multiple Agents 
24 Communication Among Agents 
25 Agent Architectures

/pdf/artificial-intelligence-a-new-synthesis-1mm7uhqtx9.pdf

Artificial Intelligence: A New Synthesis

This is a comprehensive description of the Enterprise Ontology, a collection of terms and definitions relevant to business enterprises. We state its intended purposes, describe how we went about building it, define all the terms and describe our experiences in converting these into formal definitions. We then describe how we used the Enterprise Ontology and give an evaluation which compares the actual uses with original purposes. We conclude by summarising what we have learned. The Enterprise Ontology was developed within the Enterprise Project, a collaborative effort to provide a framework for enterprise modelling. The ontology was built to serve as a basis for this framework which includes methods and a computer tool set for enterprise modelling. We give an overview of the Enterprise Project, elaborate on the intended use of the ontology, and give a brief overview of the process we went through to build it. The scope of the Enterprise Ontology covers those core concepts required for the project, which will appeal to a wider audience. We present natural language definitions for all the terms, starting with the foundational concepts (e.g. entity, relationship, actor). These are used to define the main body of terms, which are divided into the following subject areas: activities, organisation, strategy and marketing. We review some of the things learned during the formalisation process of converting the natural language definitions into Ontolingua. We identify and propose solutions for what may be general problems occurring in the development of a wide range of ontologies in other domains. We then characterise in general terms the sorts of issues that will be faced when converting an informal ontology into a formal one. Finally, we describe our experiences in using the Enterprise Ontology. We compare these with the intended uses, noting our successes and failures. We conclude with an overall evaluation and summary of what we have learned.

/pdf/the-enterprise-ontology-2kx58to7pj.pdf

The Enterprise Ontology

The SHOP2 planning system received one of the awards for distinguished performance in the 2002 International Planning Competition. This paper describes the features of SHOP2 which enabled it to excel in the competition, especially those aspects of SHOP2 that deal with temporal and metric planning domains.

/pdf/shop2-an-htn-planning-system-20n5kqaph4.pdf

SHOP2: an HTN planning system

O-Plan: the open planning architecture

Partially ordered plans have not solved the goal ordering problem. Consider: a goal in a partially ordered plan is an operator precondition that is not yet achieved; operators, orderings and variable bindings are introduced to achieve such goals. While the planning community has known how to achieve individual goals for some time, there has been little work on the problem of which one of the many possible goals the planner should achieve next. This paper argues that partially ordered plans do not usefully address the goal-ordering problem and then presents a heuristic called temporal coherence which does. Temporal coherence is an admissible heuristic which provides goal-ordering guidance. Temporal coherence is admissible in the sense that if a solution exists in the planner's search space, then there will be a series of goal achievements permitted by the heuristic which can produce this solution.

/pdf/goal-ordering-in-partially-ordered-plans-3x3m8sp63o.pdf

Goal ordering in partially ordered plans

O-Plan: control in the open planning architecture

O-Plan: control in the Open Planning Architecture

Correct conventional nonlinear planners operate in accordance with Chapman's modal truth criterion (MTC). The MTC characterizes the conditions under which an assertion will be true at a point in a nonlinear plan. However, the MTC is not all one requires in order to build a realistic planning system: it merely sanctions the use of a number of plan modifications in order to achieve each assertion in a developing plan. The number of modifications that can be made is usually very large. To avoid breadth-first search a planner must have some idea of which plan modification to consider. We describe a domain-independent search heuristic called temporal coherence, which helps guide the search through the space of partial plans defined by the MTC. Temporal coherence works by suggesting certain orderings of goal achievement as more appealing than others, and thus by finding bindings for plan variables consistent with the planner's overall goals. Our experience with a real nonlinear planner has highlighted the need for such a heuristic. In this paper, we give an example planning problem and use it to illustrate how temporal coherence can speed the search for an acceptable plan. We also prove that if a solution exists in the partial plan search space defined by the MTC, then there exists a path to that solution which is sanctioned by temporal coherence.



Les planificateurs non lineaires traditionnels exacts fonctionnent en conformite avec le critere de verite modale (CVM) de Chapman. Le CVM caracterise les conditions selon lesquelles une assertion sera vraie en un point d'un plan non lineaire. Cependant, le CVM seul ne suffit pas pour elaborer un systeme de planification realiste: il ne fait que sanctionner l'utilisation d'un certain nombre de modifications du plan afin de realiser chaque assertion dans un plan en voie de developpement. Le nombre de modifications qui peuvent etre effectuees est habituellement tres eleve. Pour eviter la recherche largeur d'abord, un planificateur doit avoir une idee de la modification du plan aetudier. Nous decrivons une recherche independante du domaine appelee coherence temporelle qui aide a orienter la recherche a travers l'espace des plans partiels definis par le CVM. La coherence temporelle suggere certains classements de realisation d'objectifs plus interessants que d'autres, et trouve des liens de variables de plans qui sont conformes aux objectifs globaux du planificateur. Notre experience avec un planificateur non lineaire reel a mis en relief la necessite d'un tel heuristique. Dans cet article, nous donnons un exemple de probleme de planification et l'utilisons pour illustrer comment la coherence temporelle peut accelerer la recherche d'un plan acceptable. L'article demontre egalement que si une solution existe dans l'espace de recherche du plan partiel defini par le CVM, il existe alors une voie conduisant a cette solution et celle-ci est sanctionnee par la coherence temporelle.

Ken Currie

Papers

O-Plan: the open planning architecture

Goal ordering in partially ordered plans

O-Plan: control in the open planning architecture

O-Plan: control in the Open Planning Architecture

Exploiting temporal coherence in nonlinear plan construction