arXiv AI

JaxAHT: A JAX-Based Library for Ad Hoc Teamwork

JaxAHT is a new open‑source library built with JAX that speeds up and standardizes research in Ad Hoc Teamwork (AHT). It offers a unified framework for generating teammates, training ego agents, and evaluating performance against unseen partners, delivering roughly 95× faster wall‑clock times than comparable PyTorch implementations. The library also supplies a diverse set of evaluation teammates for Level‑Based Foraging, Overcooked, and Hanabi, and is used to run a large‑scale benchmark that shows no single algorithm dominates and that agent modeling mainly helps in role‑based, diverse teammate settings.

arXiv AI
Sep 25

Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork

The paper introduces ICRL4AHT, a large-scale benchmark for evaluating In-Context Reinforcement Learning (ICRL) in Ad-Hoc Teamwork (AHT) scenarios using Overcooked-V2. It provides a diverse teammate suite, a reproducible pipeline, and evaluates history-conditioned ICRL algorithms such as Algorithm Distillation and Decision-Pretrained Transformer. The results show that these methods often perform worse than random baselines and do not improve with longer horizons, underscoring the difficulty of strategic inference under partial observability in AHT.

By Yuheng Jing, Kai Li, Ziwen Zhang, Jiajun Zhang, Zeyao Ma, Jiaxi Yang, Lei Zhang, Zhe Wu, Jinmin He, Junliang Xing, Jian Cheng
arXiv AI
Sep 11

ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork

The paper introduces ROTATE, a regret-driven open‑ended training framework that jointly improves an Ad Hoc Teamwork (AHT) agent and an adversarial teammate generator. Unlike traditional two‑stage pipelines, ROTATE alternates between enhancing the agent and generating teammates that specifically probe its collaboration weaknesses. Experiments on Overcooked and Level‑Based Foraging show that ROTATE outperforms existing baselines on unseen teammates, setting a new benchmark for robust, generalizable teamwork.

By Caroline Wang, Arrasy Rahman, Benjamin Nativi, Jiaxun Cui, Yoonchang Sung, Peter Stone
arXiv AI
Sep 7

MaxKernel: Agentic Kernel Generation for TPUs

MaxKernel is a multi‑agent system designed to generate high‑performance custom kernels for TPUs. It offers three paradigms: a Human‑in‑the‑Loop agent for collaborative design, an Autonomous agent that runs a fully automated optimization loop, and a Graph‑Based Autonomous Search for global exploration. All paradigms share specialized sub‑agents for planning, implementation, debugging, testing, and profiling, and the system consistently matches expert hand‑tuned baselines on the JaxBench suite and real‑world workloads.

By Shangkun Wang, Nina Cai, Charles Hoong, Julian Walker, Gerson Kroiz, George Vanica, Deepak Patil, Andi Gavrilescu, Hassan Sipra, Sethu Sankaran
arXiv AI
Oct 31

EvoTeam: Multi-Agent Collaboration via Evolving Teams

EvoTeam is a collaboration‑centric evolution framework for large‑language‑model based multi‑agent systems. It jointly evolves team organization, interaction rules, and individual agent skills using execution experience, while maintaining a library of reusable teams and guiding collaboration through team‑specific interaction rules. Experiments on four benchmarks show its effectiveness, and the code is publicly available on GitHub.

By Qinuo Cao, Xin He, Dianxuan Fu, Lizhi Zhang, Zhaoqing Xing, Jiapu Wang, Xin Wang, Na Zou, Yi Chang, Qinggang Zhang
arXiv AI
Nov 5

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability

MASBench is a benchmark for evaluating large language model (LLM)-based multi‑agent systems (MAS) under partial observability, a realistic constraint where agents only see limited environmental information. The benchmark is structured into three progressive task categories—Reasoning, Scheduling, and Game—and assesses three collaboration mechanisms: Protocol, Memory, and Routing. It provides deterministic evaluation metrics such as performance score, communication cost, and cost effectiveness, and includes empirical results across various LLM backbones to guide effective MAS design.

By Qizhi Chu, Zekai Yu, Sijie Wen, Yang Liu, Chen Qian, Cheng Yang, Chuan Shi, Zhiyuan Liu
arXiv Computation and Language
Sep 23

CONCAT: Consensus- and Confidence-Driven Ad Hoc Teaming for Efficient LLM-Based Multi-Agent Systems

CONCAT is a training‑free framework that improves the efficiency of large language model (LLM) based multi‑agent systems by clustering agents according to their initial answers and selecting cluster leaders based on confidence. It uses a Theory‑of‑Mind‑inspired heuristic to predict collaboration benefits between leaders, then prunes communications to form an ad‑hoc network that reduces latency. Experiments on three LLMs and benchmarks show up to 2.02× higher accuracy/latency ratio than LLM‑Debate and a 50.1% latency reduction on Qwen2.5‑14B‑Instruct without task‑specific training.

By Ziyang Ma, Dingyi Zhang, Sichu Liang, Jiajia Chu, Pengfei Xia, Hui Zang, Deyu Zhou
arXiv AI
Jun 9

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

arXiv:2606. 08340v1 Announce Type: new Abstract: As language models are increasingly deployed as autonomous agents, they must coordinate with others over long horizons in open-ended interactive tasks.

By Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker, Alexander Rutherford, Davide Paglieri, Aidan Scannell, Henry Gouk, Elliot J. Crowley, Tim Rockt\"aschel, Amos Storkey
arXiv AI
Oct 16

MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments

MAI-Gym is an open, all-in-one platform that unifies GUI agent evaluation, data collection, and online reinforcement learning training across mobile, computer, and web environments. It supports ten established benchmarks, scales to 20,000 parallel environment instances, and enables automated synthesis of diverse GUI applications and tasks with execution-validated verifiers. Using this platform, the authors trained a 9B-parameter agent, MAI-Gym-9B, achieving state‑of‑the‑art performance on MobileWorld and OSWorld-Verified, and demonstrated that training on a mocked train‑ticketing app improves real‑world task success from 28.8% to 48.1% on a physical phone.

By Panrong Tong, Quyu Kong, Pengxiang Li, Tianyu Xia, Hanzhang Zhou, Lei Wang, Chenglin Cai, Long Li, Long Chen, Jianan Zhang, Zhengwen Zeng, Xu Zhang, Gongjie Zhang, Yue Wang, Steven Hoi
arXiv AI
Oct 28

AgentBaton: Efficient Model Collaboration in Agentic Tasks

AgentBaton explores how capable and efficient language models can collaborate on long‑horizon agentic tasks by handoffs, where one model takes over from another. The study finds that handoffs from a capable to an efficient model at appropriate stages preserve or improve accuracy while significantly cutting cost, whereas the reverse handoff is less effective. AgentBaton, a lightweight detector, learns the optimal handoff point and achieves an average API cost reduction of 65.9% across 11 model pairs while largely maintaining task accuracy, even gaining up to 8.8 percentage points over capable‑only execution.

By Junlong Li, Jindong Jiang, Yonggan Fu, Jiarui Yao, Hanrong Ye, Wonmin Byeon, Peter Belcak, Hao Zhang, Pavlo Molchanov