Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

This book presents the definitive exposition of 'prospect theory', a compelling alternative to the classical utility theory of choice. Building on the 1982 volume, Judgement Under Uncertainty, this book brings together seminal papers on prospect theory from economists, decision theorists, and psychologists, including the work of the late Amos Tversky, whose contributions are collected here for the first time. While remaining within a rational choice framework, prospect theory delivers more accurate, empirically verified predictions in key test cases, as well as helping to explain many complex, real-world puzzles. In this volume, it is brought to bear on phenomena as diverse as the principles of legal compensation, the equity premium puzzle in financial markets, and the number of hours that New York cab drivers choose to drive on rainy days. Theoretically elegant and empirically robust, this volume shows how prospect theory has matured into a new science of decision making.

Choices, Values, and Frames

We discuss the cognitive and the psy- chophysical determinants of choice in risky and risk- less contexts. The psychophysics of value induce risk aversion in the domain of gains and risk seeking in the domain of losses. The psychophysics of chance induce overweighting of sure things and of improbable events, relative to events of moderate probability. De- cision problems can be described or framed in multiple ways that give rise to different preferences, contrary to the invariance criterion of rational choice. The pro- cess of mental accounting, in which people organize the outcomes of transactions, explains some anomalies of consumer behavior. In particular, the acceptability of an option can depend on whether a negative outcome is evaluated as a cost or as an uncompensated loss. The relation between decision values and experience values is discussed. Making decisions is like speaking prose—people do it all the time, knowingly or unknowingly. It is hardly surprising, then, that the topic of decision making is shared by many disciplines, from mathematics and statistics, through economics and political science, to sociology and psychology. The study of decisions ad- dresses both normative and descriptive questions. The normative analysis is concerned with the nature of rationality and the logic of decision making. The de- scriptive analysis, in contrast, is concerned with peo- ple's beliefs and preferences as they are, not as they should be. The tension between normative and de- scriptive considerations characterizes much of the study of judgment and choice. Analyses of decision making commonly distin- guish risky and riskless choices. The paradigmatic example of decision under risk is the acceptability of a gamble that yields monetary outcomes with specified probabilities. A typical riskless decision concerns the acceptability of a transaction in which a good or a service is exchanged for money or labor. In the first part of this article we present an analysis of the cog- nitive and psychophysical factors that determine the value of risky prospects. In the second part we extend this analysis to transactions and trades. Risky Choice Risky choices, such as whether or not to take an umbrella and whether or not to go to war, are made without advance knowledge of their consequences. Because the consequences of such actions depend on uncertain events such as the weather or the opponent's resolve, the choice of an act may be construed as the acceptance of a gamble that can yield various out- comes with different probabilities. It is therefore nat- ural that the study of decision making under risk has focused on choices between simple gambles with monetary outcomes and specified probabilities, in the hope that these simple problems will reveal basic at- titudes toward risk and value. We shall sketch an approach to risky choice that

http://psycnet.apa.org/doi/10.1037/0003-066X.39.4.341

Early studies of intuitive judgment and decision making conducted with the late Amos Tversky are reviewed in the context of two related concepts: an analysis of accessibility, the ease with which thoughts come to mind; a distinction between effortless intuition and deliberate reasoning. Intuitive thoughts, like percepts, are highly accessible. Determinants and consequences of accessibility help explain the central results of prospect theory, framing effects, the heuristic process of attribute substitution, and the characteristic biases that result from the substitution of nonextensional for extensional attributes. Variations in the accessibility of rules explain the occasional corrections of intuitive judgments. The study of biases is compatible with a view of intuitive thinking and decision making as generally skilled and successful.

https://www2.econ.iastate.edu/tesfatsi/JudgementAndChoice.MappingBoundedRationality.DKahneman2003.pdf

A perspective on judgment and choice: Mapping bounded rationality.

A menu of paired lottery choices is structured so that the crossover point to the high-risk lottery can be used to infer the degree of risk aversion With normal laboratory payoffs of several dollars, most subjects are risk averse and few are risk loving Scaling up all payoffs by factors of twenty, fifty, and ninety makes little difference when the high payoffs are hypothetical In contrast, subjects become sharply more risk averse when the high payoffs are actually paid in cash A hybrid "power/expo" utility function with increasing relative and decreasing absolute risk aversion nicely replicates the data patterns over this range of payoffs from several dollars to several hundred dollars

Risk Aversion and Incentive Effects

EVER SINCE mathematicians first began to study the measurement of risk there has been general agreement on the following proposition: Expected values are computed by multiplying each possible gain by the number of ways in which it can occur, and then dividing the sum of these products by the total number of possible cases where, in this theory, the consideration of cases which are all of the same probability is insisted upon. If this rule be accepted, what remains to be done within the framework of this theory amounts to the enumeration of all alternatives, their breakdown into equi-probable cases and, finally, their insertion into corresponding classifications…

Exposition of a new theory on the measurement of risk

Exposition of a New Theory on the Measurement of Risk

‘I simply wish that, in a matter which so closelyconcerns the wellbeing of the human race, no deci-sion shall be made without all the knowledge whicha little analysis and calculation can provide’Daniel Bernoulli 1760.INTRODUCTIONShould the general population be vaccinatedagainst smallpox (Variola Major)? Would the bene-ﬁts of mass vaccination outweigh the risks? Howmany deaths would occur as the result of a massvaccination campaign against smallpox? Canmathematical models of smallpox vaccination beused to determine health policy? Although small-pox was declared eradicated by the World HealthOrganization in 1979, these questions have all beenrecently debated based upon the premise thatsmallpox may be used as a weapon of bioterror-ism. Hence, a series of analyses has recently beenpublished that use mathematical models to try todetermine the most effective public healthresponse in the event of such an attack [1–4]. How-ever, these same controversial public health ques-tions were debated in the 18th century whensmallpox was endemic and Reviews in Medical Vir-ology has published two classic papers describingthe natural history of smallpox in 1902 and 1913to help inform these discussions [5,6]. We nowpublish an even earlier paper.In 1760 Daniel Bernoulli (1700–1782), one of thegreatest scientists of the 18th century, wrote amathematical analysis of the problem in order totry to inﬂuence public health policy by encoura-ging the universal inoculation against smallpox;his analysis was ﬁrst presented at the Royal Acad-emy of Sciences in Paris in 1760 and later pub-lished in 1766 [7]. Here, we republish anddiscuss both the historical and the current signiﬁ-cance of Bernoulli’s classic paper. A detailed dis-cussion of the mathematics of Bernoulli’s analysishas previously been presented by Dietz andHeesterbeck [8].According to Creighton [9] smallpox ﬁrstappeared in England in the 16th century. Smallpoxwas known in Western Europe in medieval times,but a particularly virulent strain emerged in theearly 17th century and gradually the case fatalityrate increased [10]. By the 18th century smallpoxwas endemic. Bernoulli calculated that approxi-mately three quarters of all living people (in the18th century) had been infected with smallpox[7]. One-tenth of all mortality at that time wasdue to smallpox, although there was considerableannual variation in smallpox mortality due to epi-demic outbreaks overlaying the endemic smallpoxmortality rate. For example, in London during theperiod 1761–1796 the annual number of deathsdue to smallpox varied from 3000 to 15000. Wheresmallpox was endemic it was almost wholly a dis-ease of childhood, with a case-fatality rate of 20%–30%; the mean age of death due to smallpox hasbeen estimated as 2.6 years [10] or 4.5 years [11].

/pdf/an-attempt-at-a-new-analysis-of-the-mortality-caused-by-25lftalye9.pdf

Daniel Bernoulli

Papers

Exposition of a new theory on the measurement of risk

Exposition of a New Theory on the Measurement of Risk

An attempt at a new analysis of the mortality caused by smallpox and of the advantages of inoculation to prevent it