Ad hoc teamwork is the well-established research problem of designing agents that can collaborate with new teammates without prior coordination. This survey makes a two-fold contribution. First, it provides a structured description of the different facets of the ad hoc teamwork problem. Second, it discusses the progress that has been made in the field so far, and identifies the immediate and longterm open problems that need to be addressed in the field of ad hoc teamwork.

A Survey of Ad Hoc Teamwork: Definitions, Methods, and Open Problems

Artificial intelligence-empowered smart things (e.g., robots, autonomous vehicles, and unmanned aerial vehicles) have been transforming the world. The “brains” of smart things can be abstracted as the agents or cybertwins residing on end devices and edge servers. The next-generation communication networks (i.e., 6G) will become the nervous system for these agents and natively support multiagent cooperation. By sharing local observations and intentions via communication channels, the agents could better understand the environments and make right decisions. Due to the limited channel bandwidth, the communication is considered as a bottleneck of multiagent cooperation. In this article, we propose a graph convolutional communication method (GraphComm) for multiagent cooperation to relive the bottleneck. Specifically, a variational information bottleneck is used to encode the observations and intentions compactly. Furthermore, a graph information bottleneck with the attention-based neighbor sampling mechanism is utilized to improve the effectiveness and robustness of the multiround communication process. The experimental results show that GraphComm can improve the effectiveness, robustness, and efficiency of communication in multiagent cooperative tasks as compared to baseline methods.

GraphComm: Efficient Graph Convolutional Communication for Multiagent Cooperation

Successful deployment of multi-agent reinforcement learning often requires agents to adapt their behaviour. In this work, we discuss the problem of teamwork adaptation in which a team of agents needs to adapt their policies to solve novel tasks with limited fine-tuning. Motivated by the intuition that agents need to be able to identify and distinguish tasks in order to adapt their behaviour to the current task, we propose to learn multi-agent task embeddings (MATE). These task embeddings are trained using an encoder-decoder architecture optimised for reconstruction of the transition and reward functions which uniquely identify tasks. We show that a team of agents is able to adapt to novel tasks when provided with task embeddings. We propose three MATE training paradigms: independent MATE, centralised MATE, and mixed MATE which vary in the information used for the task encoding. We show that the embeddings learned by MATE identify tasks and provide useful information which agents leverage during adaptation to novel tasks.

Learning Task Embeddings for Teamwork Adaptation in Multi-Agent Reinforcement Learning

Effectively training a robust agent that can cooperate with unseen agents requires diverse training partner agents. Nonetheless, obtaining cooperative agents with diverse behaviors is a challenging task. Previous work proposes learning a diverse set of agents by diversifying the state-action distribution of the agents. However, without information about the task’s goal, the diversified behaviors are not motivated to find other important, albeit non-optimal, solutions, resulting in only local variations of a solution. In this work, we propose to learn diverse behaviors by looking at policy compatibility while using state-action information to induce local variations of behaviors. Conceptually, policy compatibility measures whether policies of interest can collectively solve a task. We posit that incompatible policies can be behaviorally different. Based on this idea, we propose a novel objective to learn diverse behaviors. We theoretically show that our novel objective can generate a dissimilar policy, which we incorporate into a population-based training scheme. Empirically, the proposed method outperforms the baselines in terms of the number of discovered solutions given the same number of agents.

Generating Diverse Cooperative Agents by Learning Incompatible Policies

Ad hoc teamwork (AHT) is the problem of creating an agent that must collaborate with previously unseen teammates without prior coordination. Many existing AHT methods can be categorised as type-based methods, which require a set of predeﬁned teammates for training. Designing teammate types for training is a challenging issue that de-termines the generalisation performance of agents when dealing with teammate types unseen during training. In this work, we propose a method to discover diverse teammate types based on maximising best response diversity metrics. We show that our proposed approach yields teammate types that require a wider range of best responses from the learner during collaboration, which potentially improves the robustness of a learner’s performance in AHT compared to alternative methods.

Towards Robust Ad Hoc Teamwork Agents By Creating Diverse Training Teammates

Ad hoc teamwork is the challenging problem of designing an autonomous agent which can adapt quickly to collaborate with previously unknown teammates. Prior work in this area has focused on closed teams in which the number of agents is fixed. In this work, we consider open teams by allowing agents of varying types to enter and leave the team without prior notification. Our proposed solution builds on graph neural networks to learn scalable agent models and value decompositions under varying team sizes, which can be jointly trained with a reinforcement learning agent using discounted returns objectives. We demonstrate empirically that our approach results in agent policies which can robustly adapt to dynamic team composition, and is able to effectively generalize to larger teams than were seen during training.

Open Ad Hoc Teamwork using Graph-based Policy Learning

/pdf/towards-open-ad-hoc-teamwork-using-graph-based-policy-lrpac2lnb0.pdf

Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning

Enabling reinforcement learning (RL) agents to leverage a knowledge base while learning from experience promises to advance RL in knowledge intensive domains. However, it has proven difficult to leverage knowledge that is not manually tailored to the environment. We propose to use the subclass relationships present in open-source knowledge graphs to abstract away from specific objects. We develop a residual policy gradient method that is able to integrate knowledge across different abstraction levels in the class hierarchy. Our method results in improved sample efficiency and generalisation to unseen objects in commonsense games, but we also investigate failure modes, such as excessive noise in the extracted class knowledge or environments with little class structure.

/pdf/leveraging-class-abstraction-for-commonsense-reinforcement-e5vgaqts.pdf

Leveraging class abstraction for commonsense reinforcement learning via residual policy gradient methods

Ad hoc teamwork is the challenging problem of designing an autonomous agent which can adapt quickly to collaborate with teammates without prior coordination mechanisms, including joint training. Prior work in this area has focused on closed teams in which the number of agents is fixed. In this work, we consider open teams by allowing agents with different fixed policies to enter and leave the environment without prior notification. Our solution builds on graph neural networks to learn agent models and joint-action value models under varying team compositions. We contribute a novel action-value computation that integrates the agent model and joint-action value model to produce action-value estimates. We empirically demonstrate that our approach successfully models the effects other agents have on the learner, leading to policies that robustly adapt to dynamic team compositions and significantly outperform several alternative methods.

Niklas Hopner

Papers

Open Ad Hoc Teamwork using Graph-based Policy Learning

Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning

Leveraging class abstraction for commonsense reinforcement learning via residual policy gradient methods

Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning