Showing papers on "Stochastic game published in 1994"

The Folk Theorem with Imperfect Public Information

Abstract: The authors study repeated games in which players observe a public outcome that imperfectly signals the actions played. They provide conditions guaranteeing that any feasible, individually rational payoff vector of the stage game can arise as a perfect equilibrium of the repeated game with sufficiently little discounting. The central condition requires that there exist action profiles with the property that, for any two players, no two deviations--one by either player--give rise to the same probability distribution over public outcomes. The results apply to principal-agent, partnership, oligopoly, and mechanism-design models, and to one-shot games with transferable utilities. Copyright 1994 by The Econometric Society.

Information revelation and strategic delay in a model of investment

Abstract: We model investment as an N-player game with a pure informational externality. Each player's payoff depends only on his own action and the state of nature. However, because a player's action reveals his private information, players wait to see what other players will do. Equilibrium is inefficient because delay is costly and information is imperfectly revealed. We characterize the unique symmetric perfect Bayesian equilibrium and study the robustness of delay, which turns out to be sensitive to the reaction speed and the number of players. We establish the following results. (i) When the period length is very short, the game ends very quickly and there is a form of herding or informational cascade which results in a collapse of investment. (ii) As the period length increases, the possibility of herding disappears. (iii) As the number of players increases, the rate of investment and the information flow are eventually independent of the number of players; adding more players simply increases the number who delay. (iv) In the limit, the time-profile of investment is extreme, a period of low investment followed either by an investment surge or a collapse.

Decentralized learning of Nash equilibria in multi-person stochastic games with incomplete information

Abstract: A multi-person discrete game where the payoff after each play is stochastic is considered. The distribution of the random payoff is unknown to the players and further none of the players know the strategies or the actual moves of other players. A learning algorithm for the game based on a decentralized team of learning automata is presented. It is proved that all stable stationary points of the algorithm are Nash equilibria for the game. Two special cases of the game are also discussed, namely, game with common payoff and the relaxation labelling problem. The former has applications such as pattern recognition and the latter is a problem widely studied in computer vision. For the two special cases it is shown that the algorithm always converges to a desirable solution. >

More Spatial Games

Abstract: We extend our exploration of the dynamics of spatial evolutionary games [Nowak & May 1992, 1993] in three distinct but related ways. We analyse, first, deterministic versus stochastic rules; second, discrete versus continuous time (see Hubermann & Glance [1993]); and, third, different geometries of interaction in regular and random spatial arrays. We show that spatial effects can change some of the intuitive concepts in evolutionary game theory: (i) equilibria among strategies are no longer necessarily characterised by equal average payoffs; (ii) the strategy with the higher average payoff can steadily converge towards extinction; (iii) strategies can become extinct even though their basic reproductive rate (at very low frequencies) is larger than one. The equilibrium properties of spatial games are instead determined by “local relative payoffs.” We characterise the conditions for coexistence between cooperators and defectors in the spatial prisoner’s dilemma game. We find that cooperation can be maintain...

Fast algorithms for finding randomized strategies in game trees

Abstract: Interactions among agents can be conveniently described by game trees. In order to analyze a game, it is important to derive optimal (or equilibrium) strategies for the different players. The standard approach to finding such strategies in games with imperfect information is, in general, computationally intractable. The approach is to generate the normal form of the game (the matrix containing the payoff for each strategy combination), and then solve a linear program (LP) or a linear complementarity problem (LCP). The size of the normal form, however, is typically exponential in the size of the game tree, thus making this method impractical in all but the simplest cases. This paper describes a new representation of strategies which results in a practical linear formulation of the problem of two-player games with perfect recall (i.e., games where players never forget anything, which is a standard assumption). Standard LP or LCP solvers can then be applied to find optimal randomized strategies. The resulting algorithms are, in general, exponentially better than the standard ones, both in terms of time and in terms of space. ∗Computer Science Division, University of California, Berkeley, CA 94720; and IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120 †IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120; and School of Mathematical Sciences, Tel Aviv University, Tel Aviv, Israel. ‡Informatik 5, University of the Federal Armed Forces at Munich, 85577 Neubiberg, Germany. Research supported in part by ONR Contract N00014-91-C-0026, by the Air Force Office of Scientific Research (AFSC) under Contract F49620-91-C-0080, and by the Volkswagen Foundation. Some of the work was performed while the first author was at Stanford University. The United States Government is authorized to reproduce and distribute reprints for governmental purposes. In: Proceedings of the 26th ACM Symposium on the Theory of Computing, 1994, 750–759

Muddling Through: Noisy Equilibrium selection

Abstract: We examine an evolutionary model in which the primary source of "noise" that moves the model between equilibria is not random, arbitrarily improbable mutations but mistakes in learning. We find conditions under which the payoff-dominant equilibrium in a 2x2 game is selected by the model as well as conditions under which the risk- dominant equilibrium is selected. The relevant risk-dominance considerations, however, arise not in the original game but in a "fitness game" derived from the process by which payoffs in the original game are translated into evolutionary fitnesses. We also find that waiting times until the limiting distribution is reached can be shorter than in a mutation-driven model. To explore the robustness of the results to the specification of the model, we present a number of comparative static results as well as a "two-tiered" evolutionary model in which the rules by which agents learn to play the game are themselves subject to evolutionary pressure.

On relaxation algorithms in computation of noncooperative equilibria

Abstract: This paper considers a special class of numerical algorithms, the so-called relaxation algorithm, for Nash equilibrium points in noncooperative games. The relaxation algorithms have been studied by various authors for the deterministic case. Convergence conditions of this algorithm are based on fixed point theorems. For example, Basar (1987) and Li and Basar (1987) have proved its convergence for a two-player game via the contraction mapping theorem. For the quadratic case these conditions can be easily checked. For other nonlinear payoff functions it is sometimes difficult to check these convergence conditions. In this paper, the authors propose an alternative approach using the residual terms of the Nikaido-Isoda function. The convergence theorem is proved for nonsmooth weakly convex-concave Nikaido-Isoda functions. The family of weakly convex-concave functions is broad enough for applications, since if includes the family of smooth functions. When the payoff functions are twice continuously differentiable, the condition for the residual terms is reduced to strict positiveness of a matrix representing the difference of the Hessians of the Nikaido-Isoda function with respect to the first and second groups of variables. An analogous condition was used by Uryas'ev (1988) to prove convergence of the gradient-type algorithm for the Nash equilibrium problem. In this paper the authors discuss only the deterministic case; nevertheless this approach can be generalized for the stochastic Nash equilibrium problems with uncertainties in parameters. >

The folk theorem for repeated games: a neu condition'

Abstract: WE ARE CONCERNED here with perfect "folk theorems" for infinitely repeated games with complete information. Folk theorems assert that any feasible and individually rational payoff vector of the stage game is a (subgame perfect) equilibrium payoff in the associated infinitely repeated game with little or no discounting (where payoff streams are evaluated as average discounted or average values respectively). It is obvious that feasibility and individual rationality are necessary conditions for a payoff vector to be an equilibrium payoff. The surprising content of the folk theorems is that these conditions are also (almost) sufficient. Perhaps the first folk theorem type result is due to Friedman (1971) who showed that any feasible payoff which Pareto dominates a Nash equilibrium payoff of the stage game will be an equilibrium payoff in the associated repeated game with sufficiently patient players. This kind of result is sometimes termed a "Nash threats" folk theorem, a reference to its method of proof. For the more permissive kinds of folk theorems considered here, the seminal results are those of Aumann and Shapley (1976) and Rubinstein (1977, 1979). These authors assume that payoff streams are undiscounted.2 Fudenberg and Maskin (1986) establish an analogous result for discounted repeated games as the discount factor goes to 1. Their result uses techniques of proof rather different from those used by Aumann-Shapley and Rubinstein, respectively. See their paper for an insightful discussion of this point, and quite generally for more by way of background. It is a key reference for subsequent work in this area, including our own. For the two-player case, the result of Fudenberg and Maskin (1986) is a complete if and only if characterization (modulo the requirement of strict rather than weak individual rationality, which we retain in this note) and does not employ additional conditions. For three or more players Fudenberg and Maskin introduced a full dimensionality condition: The convex hull F, of the set of feasible payoff vectors of the stage game must have dimension n (where n is the number of players), or equivalently a nonempty interior. This condition has been widely adopted in proving folk theorems for related environments such as finitely repeated games (Benoit and Krishna (1985)), and overlapping generations games (Kandori (1992), Smith (1992)). Full dimensionality is a sufficient condition. Fudenberg and Maskin present an example of a three-player stage game in which the conclusion of the folk theorem is false. In this example all players receive the same payoffs in all contingencies; the (convex hull of the) set of feasible payoffs is one-dimensional. This example violates full dimensionality in a rather extreme way. Less extreme violations may also lead to

An algorithm for finding the nucleolus of assignment games

Abstract: Assignment games with side payments are models of certain two-sided markets. It is known that prices which competitively balance supply and demand correspond to elements in the core. The nucleolus, lying in the lexicographic center of the nonempty core, has the additional property that it satisfies each coalition as much as possible. The corresponding prices favor neither the sellers nor the buyers, hence provide some stability for the market. An algorithm is presented that determines the nucleolus of an assignment game. It generates a finite number of payoff vectors, monotone increasing on one side, and decreasing on the other. The decomposition of the payoff space and the lattice-type structure of the feasible set are utilized in associating a directed graph. Finding the next payoff is translated into determining the lengths of longest paths to the nodes, if the graph is acyclic, or otherwise, detecting the cycle(s). In an (m,n)-person assignment game withm = min(m,n) the nucleolus is found in at most 1/2·m(m + 3) steps, each one requiring at mostO(m·n) elementary operations.

Iterated Prisoner's Dilemma with Choice and Refusal of Partners

Abstract: This article extends the traditional iterated prisoner's dilemma (IPD) with round-robin partner matching by permitting players to choose and refuse partners in each iteration on the basis of continually updated expected payoffs. Comparative computer experiments are reported that indicate the introduction of partner choice and refusal accelerates the emergence of mutual cooperation in the IPD relative to round-robin partner matching. Moreover, in contrast to findings for round-robin partner matching (in which the average payoffs of the players tend to be either clustered around the mutual cooperation payoff or widely scattered), the average payoff scores of the players with choice and refusal of partners tend to cluster into two or more distinct narrow bands. Preliminary analytical and computational sensitivity studies are also reported for several key parameters. Related work can be accessed here: http://www2.econ.iastate.edu/tesfatsi/tnghome.htm

Reinforcement learning algorithms for average-payoff markovian decision processes

Abstract: Reinforcement learning (RL) has become a central paradigm for solving learning-control problems in robotics and artificial intelligence. RL researchers have focussed almost exclusively on problems where the controller has to maximize the discounted sum of payoffs. However, as emphasized by Schwartz (1993), in many problems, e.g., those for which the optimal behavior is a limit cycle, it is more natural and computationally advantageous to formulate tasks so that the controller's objective is to maximize the average payoff received per time step. In this paper I derive new average-payoff RL algorithms as stochastic approximation methods for solving the system of equations associated with the policy evaluation and optimal control questions in average-payoff RL tasks. These algorithms are analogous to the popular TD and Q-learning algorithms already developed for the discounted-payoff case. One of the algorithms derived here is a significant variation of Schwartz's R-learning algorithm. Preliminary empirical results are presented to validate these new algorithms.

Social dilemmas and cooperation

Abstract: Social orientation analysis of the common and individual interest problems.- Toward more locomotion in experimental games.- Individual reasoning process in the participation game with period.- The position effect: The role of a player's serial position in a resource dilemma game.- Positive and negative mood effects on solving a resource dilemma.- Fairness judgements in an asymmetric public goods dilemma.- Group size effects in social dilemmas: A review of the experimental literature and some new results for one-shot N-PD games.- Provision of step-level public goods: Effects of different information structures.- Conditional contributions and public good provision.- Convergence in the orange grove: Learning processes in a social dilemma setting.- Leadership and group identity as determinants of resource consumption in a social dilemma.- Prisoner's dilemma networks: Selection strategy versus action strategy.- Choice of strategies in social dilemma supergames.- Social dilemmas exist in space.- Commuting by car or by public transportation? An interdependence theoretical approach.- Evolution of norms without metanorms.- Computer simulations of the relation between individual heuristics and global cooperation in prisoner's dilemmas.- What risk should a selfish partner take in order to save the life of a nonrelative, selfish friend? - A stochastic game approach to the prisoner's dilemma.- Learning models for the prisoner's dilemma game: A review.- Social capital and cooperation: Communication, bounded rationality, and behavioral heuristics.- Cooperation in an asymmetric volunteer's dilemma game: Theory and experimental evidence.- Ten rules of bargaining sequences: A boundedly rational model of coalition bargaining in characteristic function games.- Aspiration processing in multilateral bargaining: Experiment, theory and simulation.- Resistance against mass immigration - An evolutionary explanation.- Authors index.

Construction of stationary Markov equilibria in a strategic market game

Abstract: This paper studies stationary noncooperative equilibria in an economy with fiat money, one nondurable commodity, countably many time-periods, no credit or futures market, and a measure space of agents—who may differ in their preferences and in the distributions of their (random) endowments. These agents are immortal, and hold fiat money as a means of hedging against the random fluctuations in their endowments of the commodity. In the aggregate, these fluctuations offset each other, and equilibrium prices are constant. We carry out an equilibrium analysis that focuses on distribution of wealth, on consumption, and on price formation. A careful analysis of the one-agent, infinite-horizon optimization problem, and of the invariant measure for the associated optimally controlled Markov chain, leads by aggregation to a stationary noncooperative or competitive equilibrium. This consists of a price for the commodity and of a distribution of wealth across agents which, under appropriate simple strategies for the ...

The demand commitment bargaining and snowballing cooperation

Abstract: A multi-person bargaining model based on sequential demands is studied for coalitional games with increasing returns to scale for cooperation. We show that for such games the (subgame perfect) equilibrium behavior leads to a payoff distribution which approaches the Shapley value as the money unit approaches 0. Subgame consistency and strategic equilibria are the main tools used in the analysis. The model is then applied to study a problem of public good consumption.

Flow control using the theory of zero sum Markov games

Abstract: Considers the problem of dynamic flow control of arriving packets into an infinite buffer. The service rate may depend on the state of the system, may change in time, and is unknown to the controller. The goal of the controller is to design an efficient policy which guarantees the best performance under the worst service conditions. The cost is composed of a holding cost, a cost of rejecting customers (packets), and a cost that depends on the quality of the service. The problem is studied in the framework of zero-sum Markov games, and a value iteration algorithm is used to solve it. It is shown that there exists an optimal stationary policy (such that the decisions depend only on the actual number of customers in the queue); it is of a threshold type, and it uses randomization in at most one state. >

Equivalence of games and markets

Abstract: The author proves an equivalence between large games with effective small groups of players and games generated by markets. Small groups are effective if all or almost all gains to collective activities can be achieved by groups bounded in size. A market is an exchange economy where all participants have concave, quasi-linear payoff functions. The market approximating a game is socially homogeneous-all participants have the same monotonic nondecreasing, and 1-homogeneous payoff function. Our results imply that any market (more generally, any economy with effective small groups) can be approximated by a socially homogeneous market.

The theory of normal form games from the differentiable viewpoint

Abstract: An alternative definition of regular equilibria is introduced and shown to have the same properties as those definitions already known from the literature. The system of equations used to define regular equilibria induces a globally differentiable structure on the space of mixed strategies. Interpreting this structure as a vector field, called the Nash field, allows for a reproduction of a number of classical results from a differentiable viewpoint. Moreover, approximations of the Nash field can be used to suitably define indices of connected components of equilibria and to identify equilibrium components which are robust against small payoff perturbations.

Advances in Dynamic Games and Applications

Abstract: I. Zero-sum differential games: Theory and applications in worst-case controller design.- A Theory of Differential Games.- H?-Optimal Control of Singularly Perturbed Systems with Sampled-State Measurements.- New Results on Nonlinear H? Control Via Measurement Feedback.- Reentry Trajectory Optimization under Atmospheric Uncertainty as a Differential Game.- II. Zero-sum differential games: Pursuit-evasion games and numerical schemes.- Fully Discrete Schemes for the Value Function of Pursuit-Evasion Games.- Zero Sum Differential Games with Stopping Times: Some Results about its Numerical Resolution.- Singular Paths in Differential Games with Simple Motion.- The Circular Wall Pursuit.- III. Mathematical programming techniques.- Decomposition of Multi-Player Linear Programs.- Convergent Stepsizes for Constrained Min-Max Algorithms.- Algorithms for the Solution of a Large-Scale Single-Controller Stochastic Game.- IV. Stochastic games: Differential, sequential and Markov Games.- Stochastic Games with Average Cost Constraints.- Stationary Equilibria for Nonzero-Sum Average Payoff Ergodic Stochastic Games and General State Space.- Overtaking Equilibria for Switching Regulator and Tracking Games.- Monotonicity of Optimal Policies in a Zero Sum Game: A Flow Control Model.- V. Applications.- Capital Accumulation Subject to Pollution Control: A Differential Game with a Feedback Nash Equilibrium.- Coastal States and Distant Water Fleets Under Extended Jurisdiction: The Search for Optimal Incentive Schemes.- Stabilizing Management and Structural Development of Open-Access Fisheries.- The Non-Uniqueness of Markovian Strategy Equilibrium: The Case of Continuous Time Models for Non-Renewable Resources.- An Evolutionary Game Theory for Differential Equation Models with Reference to Ecosystem Management.- On Barter Contracts in Electricity Exchange.- Preventing Minority Disenfranchisement Through Dynamic Bayesian Reapportionment of Legislative Voting Power.- Learning by Doing and Technology Sharing in Asymmetric Duopolies.

Stability Results for Stochastic Programs and Sensors, Allowing for Discontinuous Objective Functions

Abstract: The paper examines the stability of the optimal value and the solutions of stochastic programming problems. Stability is checked with respect to variations in both the problem formulation and the probability distribution that describes the uncertainty. Of particular interest is the case where the payoff functions may be discontinuous. These results are applied to analyze the stability of sensors that model the possibility of making inquiries to improve the probabilistic information available about the uncertain quantities.

Efficiency and Observability with Long-Run and Short-Run Players

Abstract: We present a general algorithm for computing the limit, as I´ → 1, of the set of payoffs of perfect public equilibria of repeated games with long-run and short-run players, allowing for the possibility that the players′ actions are not observable by their opponents. We illustrate the algorithm with two economic examples. In a simple partnership we show how to compute the equilibrium payoffs when the folk theorem fails. In an investment game, we show that two competing capitalists subject to moral hazard may both become worse off if their firms are merged and they split the profits from the merger. Finally, we show that with short-run players each long-run player′s highest equilibrium payoff is generally greater when their realized actions are observed.

Zero-sum stochastic games with unbounded costs: Discounted and average cost cases

Abstract: Zero-sum stochastic games with countable state space and with finitely many moves available to each player in a given state are treated. As a function of the current state and the moves chosen, player I incurs a nonnegative cost and player II receives this as a reward. For both the discounted and average cost cases, assumptions are given for the game to have a finite value and for the existence of an optimal randomized stationary strategy pair. In the average cost case, the assumptions generalize those given in Sennott (1993) for the case of a Markov decision chain. Theorems of Hoffman and Karp (1966) and Nowak (1992) are obtained as corollaries. Sufficient conditions are given for the assumptions to hold. A flow control example illustrates the results.

Bayesian learning leads to correlated equilibria in normal form games

Abstract: Consider an infinitely repeated normal form game where each player is characterized by a “type” which may be unknown to the other players of the game. Impose only two conditions on the behavior of the players. First, impose the Savage (1954) axioms; i.e., each player has some beliefs about the evolution of the game and maximizes its expected payoffs at each date given those beliefs. Second, suppose that any event which has probability zero under one player's beliefs also has probability zero under the other player's beliefs. We show that under these two conditions limit points of beliefs and of the empirical distributions (i.e., sample path averages or histograms) are correlated equilibria of the “true” game (i.e., the game characterized by the true vector of types).

A Model of Migration

Abstract: A simple game-theoretic model of migration is proposed, in which the players are animals, the strategies are territories in a landscape to which they may migrate, and the payoffs for each animal are determined by its ultimate location and the number of other animals there. If the payoff to an animal is a decreasing function of the number of other animals sharing its territory, we show the resultant game has a pure strategy Nash equilibrium (PSNE). Furthermore, this PSNE is generated via "natural" myopic behavior on the part of the animals. Finally, we compare this type of game with congestion games and potential games.