How to Do Things With Words

An Introduction to MultiAgent Systems.

Nos últimos trinta trinta anos, houve profundas mudanças na forma como compreendemos o crime e a justiça criminal. O crime tornou-se um evento simbólico, um verdadeiro teste para a ordem social e para as políticas governamentais, um desafio para a sociedade civil, para a democracia e para os direitos humanos. Segundo David Garland, professor da Faculdade de Direito da New York University, um dos principais autores no campo da Sociologia da Punição e com artigo publicado na Revista de Sociologia e Política , número 13, na modernidade tardia houve uma verdadeira obsessão securitária, direcionando as políticas criminais para um maior rigor em relação às penas e maior intolerância com o criminoso. Há trinta anos, nos EUA e na Inglaterra essa tendência era insuspeita. O livro mostra que os dois países compartilham intrigantes similaridades em suas práticas criminais, a despeito da divisão racial, das desigualdades econômicas e da letalidade violenta que marcam fortemente o cenário americano. Segundo David Garland, encontram-se nos dois países os “mesmos tipos de riscos e inseguranças, a mesma percepção a respeito dos problemas de um controle social não-efetivo, as mesmas críticas da justiça criminal tradicional, e as mesmas ansiedades recorrentes sobre mudança e ordem sociais”1 (GARLAND, 2001, p. 2). O argumento principal da obra é o seguinte: a modernidade tardia, esse distintivo padrão de relações sociais, econômicas e culturais, trouxe consigo um conjunto de riscos, inseguranças e problemas de controle social que deram uma configuração específica às nossas respostas ao crime, ao garantir os altos custos das políticas criminais, o grau máximo de duração das penas e a excessivas taxas de encarceramento.

http://www.scielo.br/pdf/rsocp/n20/n20a15.pdf

The culture of control: crime and social order in contemporary, society

人教版高中英语新课程教材中,语言运用（Using Language）是每个单元必不可少的部分,提供了围绕单元中心话题的听、说、读、写的综合性练习,是单元中心话题的延续和升华.如何设计Using Language部分的教学,使自己的教学模式既不落俗套,又能真正体现新课程标准所倡导的教学理念,正是广大一线英语教师一直努力探索的问题.

打磨Using Language,倡导新理念

We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multi-agent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

The current mainstream approach to train natural language systems is to expose them to large amounts of text. This passive learning is problematic if we are interested in developing interactive machines, such as conversational agents. We propose a framework for language learning that relies on multi-agent communication. We study this learning in the context of referential games. In these games, a sender and a receiver see a pair of images. The sender is told one of them is the target and is allowed to send a message from a fixed, arbitrary vocabulary to the receiver. The receiver must rely on this message to identify the target. Thus, the agents develop their own language interactively out of the need to communicate. We show that two networks with simple configurations are able to learn to coordinate in the referential game. We further explore how to make changes to the game environment to cause the "word meanings" induced in the game to better reflect intuitive semantic properties of the images. In addition, we present a simple strategy for grounding the agents' code into natural language. Both of these are necessary steps towards developing machines that are able to communicate with humans productively.

Multi-Agent Cooperation and the Emergence of (Natural) Language

What explains variability in norms of cooperation across organizations and cultures? One answer comes from the internalization of norms prescribing behavior that is typically successful under the institutions that govern one’s daily life. These norms are then carried over into atypical situations beyond the reach of institutions. Here we experimentally demonstrate such spillovers. First, we immerse subjects in environments that do or do not support cooperation using repeated Prisoner’s Dilemmas. Afterwards, we measure their intrinsic prosociality in one-shot games. Subjects from environments that support cooperation are more prosocial, more likely to punish selfishness, and more generally trusting. Furthermore, these effects are most pronounced among subjects who use heuristics, suggesting that intuitive processes play a key role in the spillovers we observe. Our findings help to explain variation in one-shot anonymous cooperation, linking this intrinsically motivated prosociality to the externally imposed institutional rules experienced in other settings.

https://papers.ssrn.com/sol3/Delivery.cfm/SSRN_ID2298749_code1038894.pdf?abstractid=2294242&mirid=3

Habits of Virtue: Creating Norms of Cooperation and Defection in the Laboratory

We consider the problem of zero-shot coordination - constructing AI agents that can coordinate with novel partners they have not seen before (e.g. humans). Standard Multi-Agent Reinforcement Learning (MARL) methods typically focus on the self-play (SP) setting where agents construct strategies by playing the game with themselves repeatedly. Unfortunately, applying SP naively to the zero-shot coordination problem can produce agents that establish highly specialized conventions that do not carry over to novel partners they have not been trained with. We introduce a novel learning algorithm called other-play (OP), that enhances self-play by looking for more robust strategies, exploiting the presence of known symmetries in the underlying problem. We characterize OP theoretically as well as experimentally. We study the cooperative card game Hanabi and show that OP agents achieve higher scores when paired with independently trained agents. In preliminary results we also show that our OP agents obtains higher average scores when paired with human players, compared to state-of-the-art SP agents.

"Other-Play" for Zero-Shot Coordination

How can behavioral science incorporate tools from machine learning (ML)? We propose that ML models can be used as upper bounds for the “explainable” variance in a given data set and thus serve as upper bounds for the potential power of a theory. We demonstrate this method in the domain of uncertainty. We ask over 600 individuals to make a total of 6000 choices with randomized parameters and compare standard economic models to ML models. In the domain of risk, a version of expected utility that allows for non-linear probability weighting (as in cumulative prospect theory) and individual-level parameters performs as well out-of-sample as ML techniques. By contrast, in the domain of ambiguity, two of the most widely studied models (a linear version of maximin preferences and second order expected utility) fail to compete with the ML methods. We open the “black boxes” of the ML methods and show that under risk we “rediscover” expected utility with probability weighting. However, in the case of ambiguity the form of ambiguity aversion implied by our ML models suggests that there is gain from theoretical work on a portable model of ambiguity aversion. Our results highlight ways in which behavioral scientists can incorporate ML techniques in their daily practice to gain genuinely new insights.

Using methods from machine learning to evaluate behavioral models of choice under risk and ambiguity

Most daily decisions involve uncertainty about outcome probabilities arising from incomplete knowledge, i.e., ambiguity. We explore how the addition of partial information affects these types of choices using theoretical and empirical methods. Our experiments in both gain and loss domains demonstrate that when such information supports a favorable outcome, it strongly increases valuation of an ambiguous financial prospect. However, when information supports an unfavorable outcome, it has significantly less impact. We find that two mechanisms drive this asymmetry. First, unfavorable information decreases estimates of a good outcome occurring but also reduces aversive uncertainty. These factors act in opposition, minimizing the effects of unfavorable information. Second, when information can be subjectively interpreted, unfavorable information is less likely to be integrated into evaluations. Our findings reveal mechanisms not captured by traditional models of decision making under uncertainty and highlight...

/pdf/asymmetric-effects-of-favorable-and-unfavorable-information-10gxv1blk7.pdf

Alexander Peysakhovich

Papers

Multi-Agent Cooperation and the Emergence of (Natural) Language

Habits of Virtue: Creating Norms of Cooperation and Defection in the Laboratory

"Other-Play" for Zero-Shot Coordination

Using methods from machine learning to evaluate behavioral models of choice under risk and ambiguity

Asymmetric Effects of Favorable and Unfavorable Information on Decision Making Under Ambiguity