This chapter studies the bidirectional causal interactions between curiosity and learning and discusses how understanding these interactions can be leveraged in educational technology applications. First, we review recent results showing how state curiosity, and more generally the experience of novelty and surprise, can enhance learning and memory retention. Then, we discuss how psychology and neuroscience have conceptualized curiosity and intrinsic motivation, studying how the brain can be intrinsically rewarded by novelty, complexity, or other measures of information. We explain how the framework of computational reinforcement learning can be used to model such mechanisms of curiosity. Then, we discuss the learning progress (LP) hypothesis, which posits a positive feedback loop between curiosity and learning. We outline experiments with robots that show how LP-driven attention and exploration can self-organize a developmental learning curriculum scaffolding efficient acquisition of multiple skills/tasks. Finally, we discuss recent work exploiting these conceptual and computational models in educational technologies, showing in particular how intelligent tutoring systems can be designed to foster curiosity and learning.

/pdf/intrinsic-motivation-curiosity-and-learning-theory-and-42spckyh2l.pdf

Intrinsic motivation, curiosity, and learning: Theory and applications in educational technologies.

A negotiation between agents is typically an incomplete information game, where the agents initially do not know their opponent's preferences or strategy. This poses a challenge, as efficient and effective negotiation requires the bidding agent to take the other's wishes and future behavior into account when deciding on a proposal. Therefore, in order to reach better and earlier agreements, an agent can apply learning techniques to construct a model of the opponent. There is a mature body of research in negotiation that focuses on modeling the opponent, but there exists no recent survey of commonly used opponent modeling techniques. This work aims to advance and integrate knowledge of the field by providing a comprehensive survey of currently existing opponent models in a bilateral negotiation setting. We discuss all possible ways opponent modeling has been used to benefit agents so far, and we introduce a taxonomy of currently existing opponent models based on their underlying learning techniques. We also present techniques to measure the success of opponent models and provide guidelines for deciding on the appropriate performance measures for every opponent model type in our taxonomy.

https://link.springer.com/content/pdf/10.1007%2Fs10458-015-9309-1.pdf

Learning about the opponent in automated bilateral negotiation: a comprehensive survey of opponent modeling techniques

Despite much progress in training artificial intelligence (AI) systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce Cicero, the first AI agent to achieve human-level performance in Diplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online Diplomacy league, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game. Description AI masters Diplomacy The game Diplomacy has been a major challenge for artificial intelligence (AI). Unlike other competitive games that AI has recently mastered, such as chess, Go, and poker, Diplomacy cannot be solved purely through self-play; it requires the development of an agent to understand other players’ motivations and perspectives and to use natural language to negotiate complex shared plans. The Meta Fundamental AI Research Diplomacy Team (FAIR) et al. developed an agent that is able to play the full natural language form of the game and demonstrates performance well above the human average in an online Diplomacy league. The present work has far-reaching implications for the development of cooperative AI and language models for communication with people, even when interactions involve a mixture of aligned and competing interests. —YS Artificial intelligence demonstrates human-level performance in the strategic board game Diplomacy.

Human-level play in the game of Diplomacy by combining language models with strategic reasoning

The spread of Internet connectivity and advancement of computer hardware increases the power of information technologies to perform practical services in many areas of our everyday life. Agent technologies play essential roles in the design of service-oriented systems such as computer-mediated collaboration, education and training, electronic commerce, information retrieval, and data mining.Agents and multiagent systems also provide a new way of thinking--conceptualizing performance elements as autonomous changes the paradigm for analyzing problems, designing systems, and dealing with complexity, distribution, and interactivity. Concept of agents and multiagent architecture are shaping landscapes of basic research areas including social simulation, games, pervasive and ubiquitous computing, robotics, and user interfaces.The Autonomous Agents and Multiagent Systems (AAMAS) conference series brings together researchers from around the world to share the latest advances in the field. It was initiated in 2002 as a merger of three highly successful related events: the International Conference on Autonomous Agents (AGENTS), the International Conference on Multi-Agent Systems (ICMAS), and the International Workshop on Agent Theories, Architectures, and Languages (ATAL). The AAMAS conference series provides a single, high-profile forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS 2002, the first of the series, was held in Bologna, followed by AAMAS 2003 in Melbourne, AAMAS 2004 in New York, and AAMAS 2005 in Utrecht. AAMAS 2006 is held in Hakodate, Japan.The five-day event consists of 6 tutorials, 32 workshops, a main conference track, a separate industry paper track with 15 papers, demonstration sessions with 17 demonstrations, an exhibition, and a doctoral mentoring program with 24 students. These and other conference elements are the result of great efforts on the part of many dedicated volunteers. We are grateful to all the AAMAS-06 conference officials and those who worked with them to put together such a successful research meeting.AAMAS 2006 received 553 submissions for the main conference. Each paper was reviewed by at least three program committee members (guided by a senior program committee member). 127 papers were accepted as full 8-page papers, and an additional 123 were accepted as short 3-page papers (acceptance rates of 23.0% for full papers and 45.2% overall).Several innovations were added to this year's conference program and review process. In order to make the review process more interactive, authors were given the opportunity to respond to their reviews prior to the discussion phase and final decisions. In many cases, the reviews, and sometimes the decisions, were altered as a result. In addition, we introduced awards for "best program committee member" and "best senior program committee member" in recognition of the exceptional time and effort contributed by our program committee.At the conference itself, for the first time, the emphasis was on poster presentations as the main form of technical exchange. All accepted papers were presented at poster sessions, with just about half of the full papers presented orally in two parallel tracks. The oral presentations were chosen primarily so as to provide a broad representative of the accepted papers. The goal of this format was to allow for more focused oral sessions while maximizing the opportunity for one-on-one detailed technical exchanges at the posters.

Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems

This paper presents a general framework for multilateral turn-taking protocols and two fully specified protocols namely Stacked Alternating Offers Protocol (SAOP) and Alternating Multiple Offers Protocol (AMOP). In SAOP, agents can make a bid, accept the most recent bid or walk way (i.e., end the negotiation without an agreement) when it is their turn. AMOP has two different phases: bidding and voting. The agents make their bid in the bidding phase and vote the underlying bids in the voting phase. Unlike SAOP, AMOP does not support walking away option. In both protocols, negotiation ends when the negotiating agents reach a joint agreement or some deadline criterion applies. The protocols have been evaluated empirically, showing that SAOP outperforms AMOP with the same type of conceder agents in a time-based deadline setting. SAOP was used in the ANAC 2015 competition for automated negotiating agents.

Alternating Offers Protocols for Multilateral Negotiation

Existing work on automated negotiations has mainly focused on bilateral negotiations with linear utility functions. It is often assumed that all possible agreements and their utility values are given beforehand. Most real-world negotiations however are much more complex. We introduce a new family of negotiation algorithms that is applicable to domains with many agents, an intractably large space of possible agreements, non-linear utility functions and limited time so an exhaustive search for the best proposals is not feasible. We assume that agents are selfish and cannot be blindly trusted, so the algorithm does not rely on any mediator. This family of algorithms is called $$\hbox {NB}^{3}$$NB3 and applies heuristic Branch & Bound search to find good proposals. Search and negotiation happen simultaneously and therefore strongly influence each other. It applies a new time-based negotiation strategy that considers two utility aspiration levels: one for the agent itself and one for its opponents. Also, we introduce a negotiation protocol that imposes almost no restrictions and is therefore better applicable to negotiations with humans. We present the Negotiating Salesmen Problem (NSP): a variant of the Traveling Salesman Problem with multiple negotiating agents, as a test case. We describe an implementation of $$\hbox {NB}^{3}$$NB3 designed for the NSP and present the results of experiments with this implementation. We conclude that the algorithm is able to decrease the costs of the agents significantly, that the heuristic search is efficient and that the algorithm scales well with increasing complexity of the problem.

/pdf/hbox-nb-3-nb3-a-multilateral-negotiation-algorithm-for-large-3i8jbvuasi.pdf

$$\hbox {NB}^{3}$$NB3: a multilateral negotiation algorithm for large, non-linear agreement spaces with limited time

The game of Diplomacy has been used as a test case for complex automated negotiations for a long time, but to date very few successful negotiation algorithms have been implemented for this game. We have therefore decided to include a Diplomacy tournament within the annual Automated Negotiating Agents Competition (ANAC). In this paper we present the setup and the results of the ANAC 2017 Diplomacy Competition and the ANAC 2018 Diplomacy Challenge. We observe that none of the negotiation algorithms submitted to these two editions have been able to significantly improve the performance over a non-negotiating baseline agent. We analyze these algorithms and discuss why it is so hard to write successful negotiation algorithms for Diplomacy. Finally, we provide experimental evidence that, despite these results, coalition formation and coordination do form essential elements of the game.

/pdf/the-challenge-of-negotiation-in-the-game-of-diplomacy-23xxhseka8.pdf

The challenge of negotiation in the game of Diplomacy

In this paper we present a new algorithm for negotiations in non-zero-sum games Although games have been studied extensively, most game playing algorithms have been developed under the assumption that players do not communicate Many real-world problems, however, can be modeled as non-zero-sum games in which players may mutually benefit if they coordinate their actions, which requires negotiation The field of Automated Negotiations is another important topic in AI, but in this field one usually assumes that utility functions have explicit expressions and can therefore be calculated easily Traditional approaches do not apply to domains in which the utility values are instead determined by the rules of a complex game In this paper we aim to bridge the gap between General Game Playing and Automated Negotiations Our algorithm is an adaptation of Monte Carlo Tree Search that allows players to negotiate It is completely domain-independent in the sense that it is not tailored to any specific game It can be applied to any non-zero-sum game, provided that its rules are described in Game Description Language

/pdf/automated-negotiations-for-general-game-playing-2afit8j7wu.pdf

Automated Negotiations for General Game Playing

Negotiation is an essential skill for agents in a multiagent system. Much work has been published on this subject, but traditional approaches assume negotiators are able to evaluate all possible deals and pick the one that is best according to some negotiation strategy. Such an approach fails when the set of possible deals is too large to analyze exhaustively. For this reason the Annual Negotiating Agents Competition of 2014 has focused on negotiations over very large agreement spaces. In this paper we present a negotiating agent that explores the search space by means of a Genetic Algorithm. It has participated in the competition successfully and finished in 2nd and 3rd place in the two categories of the competition respectively.

/pdf/gangster-an-automated-negotiator-applying-genetic-algorithms-3hatmy17ax.pdf

GANGSTER: An Automated Negotiator Applying Genetic Algorithms

Existing work on Automated Negotiations commonly assumes the negotiators' utility functions have explicit closed-form expressions, and can be calculated quickly. In many real-world applications however, the calculation of utility can be a complex, time-consuming problem and utility functions cannot always be expressed in terms of simple formulas. The game of Diplomacy forms an ideal test bed for research on Automated Negotiations in such domains where utility is hard to calculate. Unfortunately, developing a full Diplomacy player is a hard task, which requires more than just the implementation of a negotiation algorithm. The performance of such a player may highly depend on the underlying strategy rather than just its negotiation skills. Therefore, we introduce a new Diplomacy playing agent, called D-Brane, which has won the first international Computer Diplomacy Challenge. It is built up in a modular fashion, disconnecting its negotiation algorithm from its game-playing strategy, to allow future researchers to build their own negotiation algorithms on top of its strategic module. This will allow them to easily compare the performance of different negotiation algorithms. We show that D-Brane strongly outplays a number of previously developed Diplomacy players, even when it does not apply negotiations. Furthermore, we explain the negotiation algorithm applied by D-Brane, and present a number of additional tools, bundled together in the new BANDANA framework, that will make development of Diplomacy-playing agents easier.

Dave de Jonge

Papers

$$\hbox {NB}^{3}$$NB3: a multilateral negotiation algorithm for large, non-linear agreement spaces with limited time

The challenge of negotiation in the game of Diplomacy

Automated Negotiations for General Game Playing

GANGSTER: An Automated Negotiator Applying Genetic Algorithms

D-Brane: a diplomacy playing agent for automated negotiations research