Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

The concept of an agent has become important in both Artificial Intelligence (AI) and mainstream computer science. Our aim in this paper is to point the reader at what we perceive to be the most important theoretical and practical issues associated with the design and construction of intelligent agents. For convenience, we divide these issues into three areas (though as the reader will see, the divisions are at times somewhat arbitrary). Agent theory is concerned with the question of what an agent is, and the use of mathematical formalisms for representing and reasoning about the properties of agents. Agent architectures can be thought of as software engineering models of agents;researchers in this area are primarily concerned with the problem of designing software or hardware systems that will satisfy the properties specified by agent theorists. Finally, agent languages are software systems for programming and experimenting with agents; these languages may embody principles proposed by theorists. The paper is not intended to serve as a tutorial introduction to all the issues mentioned; we hope instead simply to identify the most important issues, and point to work that elaborates on them. The article includes a short review of current and potential applications of agent technology.

/pdf/intelligent-agents-theory-and-practice-5dchj7p8pc.pdf

Intelligent Agents: Theory and Practice

The contract net protocol has been developed to specify problem-solving communication and control for nodes in a distributed problem solver. Task distribution is affected by a negotiation process, a discussion carried on between nodes with tasks to be executed and nodes that may be able to execute those tasks.

/pdf/the-contract-net-protocol-high-level-communication-and-237lqg0ddq.pdf

The Contract Net Protocol: High-Level Communication and Control in a Distributed Problem Solver

This survey characterizes an emerging research area, sometimes called coordination theory, that focuses on the interdisciplinary study of coordination. Research in this area uses and extends ideas about coordination from disciplines such as computer science, organization theory, operations research, economics, linguistics, and psychology.A key insight of the framework presented here is that coordination can be seen as the process of managing dependencies among activities. Further progress, therefore, should be possible by characterizing different kinds of dependencies and identifying the coordination processes that can be used to manage them. A variety of processes are analyzed from this perspective, and commonalities across disciplines are identified. Processes analyzed include those for managing shared resources, producer/consumer relationships, simultaneity constraints, and task/subtask dependencies.Section 3 summarizes ways of applying a coordination perspective in three different domains:(1) understanding the effects of information technology on human organizations and markets, (2) designing cooperative work tools, and (3) designing distributed and parallel computer systems. In the final section, elements of a research agenda in this new area are briefly outlined.

/pdf/the-interdisciplinary-study-of-coordination-22jsxamy1h.pdf

The interdisciplinary study of coordination

From the Publisher:
An agent is an entity with domain knowledge, goals and actions. Multi-agent systems are a set of agents which
interact in a common environment. Multi-agent systems deal with the construction of complex systems involving multiple agents and their coordination. A multi-agent system (MAS) is a distributed computing system with
autonomous interacting intelligent agents that coordinate their actions so as to achieve its goal(s) jointly or competitively.

Introduction to Multiagent Systems

We describe the concept of distributed problem solving and define it as the cooperative solution of problems by a decentralized and loosely coupled collection of problem solvers. This approach to problem solving offers the promise of increased performance and provides a useful medium for exploring and developing new problem-solving techniques.

We present a framework called the contract net that specifies communication and control in a distributed problem solver. Task distribution is viewed as an interactive process, a discussion carried on between a node with a task to be executed and a group of nodes that may be able to execute the task. We describe the kinds of information that must be passed between nodes during the discussion in order to obtain effective problem-solving behavior. This discussion is the origin of the negotiation metaphor: Task distribution is viewed as a form of contract negotiation.

We emphasize that protocols for distributed problem solving should help determine the content of the information transmitted, rather than simply provide a means of sending bits from one node to another.

The use of the contract net framework is demonstrated in the solution of a simulated problem in area surveillance, of the sort encountered in ship or air traffic control. We discuss the mode of operation of a distributed sensing system, a network of nodes extending throughout a relatively large geographic area, whose primary aim is the formation of a dynamic map of traffic in the area.

From the results of this preliminary study we abstract features of the framework applicable to problem solving in general, examining in particular transfer of control. Comparisons with PLANNER, CONNIVER, HEARSAY-11, and PUP6 are used to demonstrate that negotiation - the two-way transfer of information - is a natural extension to the transfer of control mechanisms used in earlier problem-solving systems.

/pdf/negotiation-as-a-metaphor-for-distributed-problem-solving-4spnd282uh.pdf

Negotiation as a metaphor for distributed problem solving

We present an Interesting Situation Generator (ISG) that assists in the synthesis of interpretation rules from basic domain knowledge. The ISG is a hierarchical clustering program that discovers equivalence classes of situations (e.g. types of geological formations) that give rise to qualitatively distinct manifestations (e.g. different patterns of geophysical measurements corresponding to types of geological formations). The equivalence classes can be used by a rule generator to construct an initial set of interpretation rules of the form manifestation ⇒ situation.

A conceptual clustering program for rule generation

ABSTRAm We describe the concepr of distributed problem solving and define it as the cooperative solution of problems by a decentralized and loosely coupled collection of problem solvers. This approach to problem solving offers the promise of increased performance and provides a useful medium for exploring and developing new problem-solving techniques. We present a framework called the contract net that specifies communication and control in a distributed problem solver. Task distribution is uiewed as an interactive process, a discussion carried on between a node with a task to be executed and a group of nodes that may be able to execute the task. We describe the kinds of information that must be passed between nodes during the discussion in order to obtain effective probleni-solving bchauior. This discussion is the origin of the negotiation metaphor: Task distribution is viewed as a form of contract negotiation. We emphasize thar protocols for distributedproblem solving should help determine the content of the information transmitted, rather than simplyprovide a means of sending bits from one node to another. The use of the contract net framework is demonstrated in the solution of a simulated problem in area surveillance, of the sort encountered in ship or air t r a m control. We discuss the mode of operation of a distributed sensing system, a network of nodes extending throughout a relatively large geographic area, whose primary aim is the formation of a dynamic map of t r a m in the area. From the results of this preliminary study we abstract features of the framework applicable to problem solving in general, examining in particular transfer of control. Comparisons with PLANNER, C O N N I V E R , HEARSAY-11, and PUP6 are used to demonstrate that negotiation-the two-way transfer of information-is a natural extension to the transfer of control mechanisms used in earlier problem-soluing systems.

/pdf/negotiation-distributed-as-a-metaphor-for-problem-solving-1jiack8297.pdf

Negotiation Distributed as a Metaphor for Problem Solving

A system and method for classifying unstructured text documents, without the need for pre-classified training examples. In general, the system and method provides for blending statistical, syntactic and semantic considerations to learn classifiers from an organization's unclassified internal and external unstructured text documents, as well as unclassified documents available via the Internet. In one form, for each class in a taxonomy the class name is expanded into semantically related words and phrases to build approximate classifiers. Each approximate classifier will almost certainly be erroneous but it can be used to identify an approximately correct set of documents. The process is recursive; e.g. the approximate classifier with the strongest evidence, is fed back into the system until a stale set of the strongest terms for each classifier has been selected.

Method And System For Unsupervised Learning Of Document Classifiers

: We discuss the design of hardware and software for inexpensive microprocessor-based terminal/microcomputers Such devices are fundamentally microcomputers that have been adapted, with specialized software, to operate as remote terminals for a host computer The discussion centers on a specific video terminal designed and constructed by the authors This terminal is based on the INTEL 8080 microprocessor and is equipped with software sufficient to emulate the characteristics of standard video terminals required by several available screen-oriented text editors in common use at sites through the ARPAnet We have found that the microprocessor adequately serves as the controller for such terminals, and that a software-based approach to the design of such terminals offers substantial advantages in capabilities, flexibility, and cost over the hardware-based approach We suggest guidelines for future designs of microprocessor-based terminals on the basis of our experience designing and using the terminal described here (Author)

/pdf/considerations-for-microprocessor-based-terminal-design-2hw04bznvl.pdf

Reid G. Smith

Papers

Negotiation as a metaphor for distributed problem solving

A conceptual clustering program for rule generation

Negotiation Distributed as a Metaphor for Problem Solving

Method And System For Unsupervised Learning Of Document Classifiers

Considerations for Microprocessor-Based Terminal Design.