Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.

/pdf/human-level-control-through-deep-reinforcement-learning-40h0gx9pjx.pdf

Human-level control through deep reinforcement learning

From the Publisher:
Probabilistic Reasoning in Intelligent Systems is a complete andaccessible account of the theoretical foundations and computational methods that underlie plausible reasoning under uncertainty. The author provides a coherent explication of probability as a language for reasoning with partial belief and offers a unifying perspective on other AI approaches to uncertainty, such as the Dempster-Shafer formalism, truth maintenance systems, and nonmonotonic logic. The author distinguishes syntactic and semantic approaches to uncertaintyand offers techniques, based on belief networks, that provide a mechanism for making semantics-based systems operational. Specifically, network-propagation techniques serve as a mechanism for combining the theoretical coherence of probability theory with modern demands of reasoning-systems technology: modular declarative inputs, conceptually meaningful inferences, and parallel distributed computation. Application areas include diagnosis, forecasting, image interpretation, multi-sensor fusion, decision support systems, plan recognition, planning, speech recognitionin short, almost every task requiring that conclusions be drawn from uncertain clues and incomplete information.
Probabilistic Reasoning in Intelligent Systems will be of special interest to scholars and researchers in AI, decision theory, statistics, logic, philosophy, cognitive psychology, and the management sciences. Professionals in the areas of knowledge-based systems, operations research, engineering, and statistics will find theoretical and computational tools of immediate practical use. The book can also be used as an excellent text for graduate-level courses in AI, operations research, or applied probability.

Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference

Background on genetic algorithms, LISP, and genetic programming hierarchical problem-solving introduction to automatically-defined functions - the two-boxes problem problems that straddle the breakeven point for computational effort Boolean parity functions determining the architecture of the program the lawnmower problem the bumblebee problem the increasing benefits of ADFs as problems are scaled up finding an impulse response function artificial ant on the San Mateo trail obstacle-avoiding robot the minesweeper problem automatic discovery of detectors for letter recognition flushes and four-of-a-kinds in a pinochle deck introduction to biochemistry and molecular biology prediction of transmembrane domains in proteins prediction of omega loops in proteins lookahead version of the transmembrane problem evolutionary selection of the architecture of the program evolution of primitives and sufficiency evolutionary selection of terminals evolution of closure simultaneous evolution of architecture, primitive functions, terminals, sufficiency, and closure the role of representation and the lens effect Appendices: list of special symbols list of special functions list of type fonts default parameters computer implementation annotated bibliography of genetic programming electronic mailing list and public repository

Genetic Programming: On the Programming of Computers by Means of Natural Selection

A translation approach to portable ontology specifications

The software world is one of great richness and diversity. Many thousands of software products are available to users today, providing a wide variety of information and services in a wide variety of domains. While most of these programs provide their users with significant value when used in isolation, there is increasing demand for programs that can interoperate – to exchange information and services with other programs and thereby solve problems that cannot be solved alone. Part of what makes interoperation difficult is heterogeneity. Programs are written by different people, at different times, in different languages; and, as a result, they often provide different interfaces. The difficulties created by heterogeneity are exacerbated by dynamics in the software environment. Programs are frequently rewritten; new programs are added; old programs removed. Agent-based software engineering was invented to facilitate the creation of software able to interoperate in such settings. In this approach to software development, application programs are written as software agents, i.e. software “components” that communicate with their peers by exchanging messages in an expressive agent communication language. Agents can be as simple as subroutines; but typically they are larger entities with some sort of persistent control (e.g. distinct control threads within a single address space, distinct processes on a single machine, or separate processes on different machines). The salient feature of the language used by agents is its expressiveness. It allows for the exchange of data and logical information, individual commands and scripts (i.e. programs). Using this language, agents can communicate complex information and goals, directly or indirectly “programming” each other in useful ways. Agent-based software engineering is often compared to object-oriented programming. Like an “object”, an agent provides a message-based interface independent of its internal data structures and algorithms. The primary difference between the two approaches lies in the language of the interface. In general object-oriented programming, the meaning of a message can vary from one object to another. In agent-based software engineering, agents use a common language with an agent-independent semantics. The concept of agent-based software engineering raises a number of important questions.

Software agents

Typographical Conventions 1 Introduction 1.1 Bibliographical and Historical Remarks Exercises 2 Declarative Knowledge 2.1 Conceptualization 2.2 Predicate Calculus 2.3 Semantics 2.4 Blocks World Example 2.5 Circuits Example 2.6 Algebraic Examples 2.7 List Examples 2.8 Natural-Language Examples 2.9 Specialized Languages 2.10 Bibliographical and Historical Remarks Exercises 3 Inference 3.1 Derivability 3.2 Inference Procedures 3.3 Logical Implication 3.4 Provability 3.5 Proving Provability 3.6 Bibliographical and Historical Remarks Exercises 4 Resolution 4.1 Clausal Form 4.2 Unification 4.3 Resolution Principle 4.4 Resolution 4.5 Unsatisfiability 4.6 True-or-False Questions 4.7 Fill-in-the-Blank Questions 4.8 Circuits Example 4.9 Mathematics Example 4.10 Soundness and Completeness 4.11 Resolution and Equality 4.12 Bibliographical and Historical Remarks Exercises 5 Resolution Strategies 5.1 Deletion Strategies 5.2 Unit Resolution 5.3 Input Resolution 5.4 Linear Resolution 5.5 Set of Support Resolution 5.6 Ordered Resolution 5.7 Directed Resolution 5.8 Sequential Constraint Satisfaction 5.9 Bibliographical and Historical Remarks Exercises 6 Nonmonotonic Reasoning 6.1 The Closed-World Assumption 6.2 Predicate Completion 6.3 Taxonomic Hierarchies and Default Reasoning 6.4 Circumscription 6.5 More General Forms of Circumscription 6.6 Default Theories 6.7 Bibliographical and Historical Remarks Exercises 7 Induction 7.1 Induction 7.2 Concept Formation 7.3 Experiment Generation 7.4 Bibliographical and Historical Remarks Exercises 8 Reasoning with Uncertain Beliefs 8.1 Probabilities of Sentences 8.2 Using Bayes' Rule in Uncertain Reasoning 8.3 Uncertain Reasoning in Expert Systems 8.4 Probabilistic Logic 8.5 Probabilistic Entailment 8.6 Computations Appropriate for Small Matrices 8.7 Dealing with Large Matrices 8.8 Probabilities Conditioned on Specific Information 8.9 Bibliographical and Historical Remarks Exercises 9 Knowledge and Belief 9.1 Preliminaries 9.2 Sentential Logics of Belief 9.3 Proof Methods 9.4 Nested Beliefs 9.5 Quantifying-In 9.6 Proof Methods for Quantified Beliefs 9.7 Knowing What Something Is 9.8 Possible-Worlds Logics 9.9 Properties of Knowledge 9.10 Properties of Belief 9.11 Group Knowledge 9.12 Equality, Quantification, and Knowledge 9.13 Bibliographical and Historical Remarks Exercises 10 Metaknowledge and Metareasoning 10.1 Metalanguage 10.2 Clausal Form 10.3 Resolution Principle 10.4 Inference Procedures 10.5 Derivability and Belief 10.6 Metalevel Reasoning 10.7 Bilevel Reasoning 10.8 Reflection 10.9 Bibliographical and Historical Remarks Exercises 11 State and Change 11.1 States 11.2 Actions 11.3 The Frame Problem 11.4 Action Ordering 11.5 Conditionality 11.6 Bibliographical and Historical Remarks Exercises 12 Planning 12.1 Initial State 12.2 Goals 12.3 Actions 12.4 Plans 12.5 Green's Method 12.6 Action Blocks 12.7 Conditional Plans 12.8 Planning Direction 12.9 Unachievability Pruning 12.10 State Alignment 12.11 Frame-Axiom Suppression 12.12 Goal Regression 12.13 State Differences 12.14 Bibliographical and Historical Remarks Exercises 13 Intelligent-Agent Architecture 13.1 Tropistic Agents 13.2 Hysteretic Agents 13.3 Knowledge-Level Agents 13.4 Stepped Knowledge-Level Agents 13.5 Fidelity 13.6 Deliberate Agents 13.7 Bibliographical and Historical Remarks Exercises Answers to Exercises A.1 Introduction A.2 Declarative Knowledge A.3 Inference A.4 Resolution A.5 Resolution Strategies A.6 Nonmonotonic Reasoning A.7 Induction A.8 Reasoning with Uncertain Beliefs A.9 Knowledge and Belief A.10 Metaknowledge and Metareasoning A.11 State and Change A.12 Planning A.13 Intelligent-Agent Architecture References Index

Logical Foundations of Artificial Intelligence

Knowledge Interchange Format (KIF) is a computer-oriented language for the interchange of knowledge among disparate programs. It has declarative semantics (i.e. the meaning of expressions in the representation can be understood without appeal to an interpreter for manipulating those expressions); it is logically comprehensive (i.e. it provides for the expression of arbitrary sentences in the rst-order predicate calculus); it provides for the representation of knowledge about the representation of knowledge; it provides for the representation of nonmonotonic reasoning rules; and it provides for the de nition of objects, functions, and relations.

Knowledge Interchange Format Version 3.0 Reference Manual

The Palo Alto Collaborative Testbed (PACT), a concurrent engineering infrastructure that encompasses multiple sites, subsystems, and disciplines, is discussed. The PACT systems include NVisage, a distributed knowledge-based integration environment for design tools; DME (Device Modeling Environment), a model formulation and simulation environment; Next-Cut, a mechanical design and process planning system; and Designworld, a digital electronics design, simulation, assembly, and testing system. The motivations for PACT and the significance of the approach for concurrent engineering is discussed. Initial experiments in distributed simulation and incremental redesign are reviewed, and PACT's agent-based architecture and lessons learned from the PACT experiments are described. >

PACT: an experiment in integrating concurrent engineering systems

Introduction Standard Upper Ontology Knowledge Interchange Format (SUO-KIF) is a language designed for use in the authoring and interchange of knowledge. SUO-KIF has declarative semantics. It is possible to understand the meaning of expressions in the language without appeal to an interpreter for manipulating those expressions. In this way, KIF differs from other languages that are based on specific interpreters, such as Emycin and Prolog. SUO-KIF is also logically comprehensive-at its most general, it provides for the expression of arbitrary logical sentences. In this way, it differs from relational database languages (like SQL) and logic programming languages (like Prolog). SUO-KIF is intended primarily a first-order language, which is a good compromise between the computational demands of reasoning and richness of representation. In later sections of this document we describe ways in which SUO-KIF is made as expressive as possible without becoming higher-order. This document contains a guide to writing knowledge in SUO-KIF as well as a more formal reference. Logicians may wish to skip the introductory section " Writing SUO-KIF " and move straight to the details of the language. This section attempts to give a very quick introduction to writing SUO-KIF. A full introduction to logic is however beyond the scope of this document and (Nolt et al, 1998) is recommended. This section will also assume the use of some of the basic ontological content of the SUMO (Niles & Pease, 2001), since otherwise we would have to define relations like instance and subclass. SUO-KIF terms can be either individuals such as BillClinton and The82ndAirborne, or classes, such as Person and MilitaryUnit. We can define relations such as agent and functions such as GovernmentFn. Note that relations and functions are instances of the class of all relations and the class of all functions, respectively. Terms are combined into sentences in order to make statements of fact, for example that " The 82nd Airborne is a military unit " , which would be stated in SUO-KIF as (instance The82ndAirborne MilitaryUnit) and " The class of all Person(s) is a subclass of the class of all animals. " (subclass Person Animal) SUO-KIF allows us to connect statements with and and or. " Kofi Annan is a human and he occupies the position of Secretary General at the United Nations. " (and (instance KofiAnnan Human) (occupiesPosition KofiAnnan SecretaryGeneral UnitedNations)) Note that the connective or differs from common English usage in that or …

Michael R. Genesereth

Papers

Software agents

Logical Foundations of Artificial Intelligence

Knowledge Interchange Format Version 3.0 Reference Manual

PACT: an experiment in integrating concurrent engineering systems

Knowledge interchange format