Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Organizational learning has many virtues, virtues which recent writings in strategic management have highlighted. Learning processes, however, are subject to some important limitations. As is well-known, learning has to cope with confusing experience and the complicated problem of balancing the competing goals of developing new knowledge (i.e., exploring) and exploiting current competencies in the face of dynamic tendencies to emphasize one or the other. We examine the ways organizations approach these problems through simplification and specialization and how those approaches contribute to three forms of learning myopia, the tendency to overlook distant times, distant places, and failures, and we identify some ways in which organizations sustain exploration in the face of a tendency to overinvest in exploitation. We conclude that the imperfections of learning are not so great as to require abandoning attempts to improve the learning capabilities of organizations, but that those imperfections suggest a certain conservatism in expectations.

https://www.jstor.org/stable/pdfplus/2486499.pdf

The myopia of learning

https://econweb.ucsd.edu/~jandreon/Econ264/papers/Roth%20Erev%20GEB%201995.pdf

Learning in Extensive-Form Games: Experimental Data and Simple Dynamic Models in the Intermediate Term*

In ‘experience-weighted attraction’ (EWA) learning, strategies have attractions that reflect initial predispositions, are updated based on payoff experience, and determine choice probabilities according to some rule (e.g., logit). A key feature is a parameter δ that weights the strength of hypothetical reinforcement of strategies that were not chosen according to the payoff they would have yielded, relative to reinforcement of chosen strategies according to received payoffs. The other key features are two discount rates, φ and ρ, which separately discount previous attractions, and an experience weight. EWA includes reinforcement learning and weighted fictitious play (belief learning) as special cases, and hybridizes their key elements. When δ= 0 and ρ= 0, cumulative choice reinforcement results. When δ= 1 and ρ=φ, levels of reinforcement of strategies are exactly the same as expected payoffs given weighted fictitious play beliefs. Using three sets of experimental data, parameter estimates of the model were calibrated on part of the data and used to predict a holdout sample. Estimates of δ are generally around .50, φ around .8 − 1, and ρ varies from 0 to φ. Reinforcement and belief-learning special cases are generally rejected in favor of EWA, though belief models do better in some constant-sum games. EWA is able to combine the best features of previous approaches, allowing attractions to begin and grow flexibly as choice reinforcement does, but reinforcing unchosen strategies substantially as belief-based models implicitly do.

/pdf/experience-weighted-attraction-learning-in-normal-form-games-5fgpif7sno.pdf

Experience‐weighted Attraction Learning in Normal Form Games

NEARLY EVERYONE would see the truth as between Hamlet and Puck. Including Hamlet and Puck. Hamlet is feigning madness, and Puck is just being, well, puckish. Model-writing economists, however, tend not to the middle but to the “infinite in faculties” extreme. Although the postulate of unbounded rationality has dominated economic modeling for several decades, the dominance is relaxing. Is this encouraging? Why bounded rationality? In this survey, four reasons are given for incorporating bounded rationality in economic models. First, there is abundant empirical evidence that it is important. Second, models of bounded rationality have proved themselves in a wide range of impressive work. Third, the standard justifications for assuming unbounded rationality are unconvincing; their logic cuts both ways. Fourth, deliberation about an economic decision is a costly activity, and good economics requires that we entertain all costs. These four reasons, or categories of reasons, are developed in the following four sections. Deliberation cost will be a recurring theme. Most references are to the last 15 years, though many earlier works are also cited. A longer version of the survey, including many more references, is available from the author on request.

Why Bounded Rationality

Stochastic learning theory, 242. — Probability matching: an example, 244. — The model, 247. — Alternative states of the market: a monopoly model, 252. — A simple market model, 254. — Further properties of the single-firm model, 256. — The dynamics of adjustment, 258. — The firm in disequilibrium, 261. — A final note: rules of thumb, 264.

A Stochastic Learning Model of Economic Behavior

This book develops dynamic economic models using the perspective and analytic framework provided by psychological learning theory. This framework is used to resolve apparent contradictions between optimization theory, which lies at the heart of all modern economic theory, and day-to-day evidence that short-run economic behaviour cannot reasonably be described solely as the outcome of efficiently implemented self-interest. The author applies this viewpoint to a number of problem areas in which literal applications of maximization theory have not usually proved to be satisfactory. These include analyses of decision-making under uncertainty and gambling behaviour, the role of consumer-oriented advertising in influencing behaviour, the persistence of price dispersions in markets, and inflation.

A theory of adaptive economic behavior

This paper presents a discussion of the role of adapting expectations in the bargaining process. Negotiators are charactenzed as persons who choose bargaining strategies in their attempt to optimize their payoffs from the situation. These strategies are contingent on each party's perception of the strategy of his opponent, and if these perceptions contain errors, expectations will change and this will lead in turn to a modification of each party's strategy choice. The payoff demands and manipulative moves which charactenze the bargaining process are seen as combinations of actions which are specified in the original bargaining plans of the parties and of changes in the plans themselves. The influence of the learnmg process on the settlement point is described as well as some empirical implications of the theory in general.

/pdf/negotiation-as-a-learning-process-4unc57ix3v.pdf

Negotiation as a Learning Process

This paper represents an attempt to make use of a few postulates which are frequently found in cconomic theory, and to achieve thereby some insights into an important problem in the theory of political interactions-the formation of coalitions. It is suggested that the same competitive process which constrains market behavior has, in reality, an equally important role in the determination of alliances, and that the search for a “best” alliance is no different in principle from the search for a lowest price. The resulting analysis will be found to apply to a wide variety of alliance situations provided that two fundamental premises are satisfied: (1) alliance mcmbcrs are motivated by a desire to maximize the value (to themselves) of their membership,l and ( 2 ) membership in one grouping precludes membership in another, that is, choice among alternatives must be a dominant problem.?

Some theoretic characteristics of economic and political coalitions

https://escholarship.org/content/qt9hw951p4/qt9hw951p4.pdf?t=krn74m

John G. Cross

Papers

A Stochastic Learning Model of Economic Behavior

A theory of adaptive economic behavior

Negotiation as a Learning Process

Some theoretic characteristics of economic and political coalitions

Efficiency-inducing taxation for a monopolistically supplied depletable resource