← Back to all news
arXiv AI By Qizhi Chu, Zekai Yu, Sijie Wen, Yang Liu, Chen Qian, Cheng Yang, Chuan Shi, Zhiyuan Liu

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability

Read the original on arXiv AI →

MASBench is a benchmark for evaluating large language model (LLM)-based multi‑agent systems (MAS) under partial observability, a realistic constraint where agents only see limited environmental information. The benchmark is structured into three progressive task categories—Reasoning, Scheduling, and Game—and assesses three collaboration mechanisms: Protocol, Memory, and Routing. It provides deterministic evaluation metrics such as performance score, communication cost, and cost effectiveness, and includes empirical results across various LLM backbones to guide effective MAS design.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 21

OpenMAS-GCom. A Diagnostic Benchmark for Graph-enhanced Multi-Agent Systems

OpenMAS-GCom is a diagnostic benchmark designed to isolate the impact of communication structures, role assignments, and information flows in graph‑enhanced multi‑agent systems (G‑MAS). It evaluates systems by systematically modifying one component—such as rewiring communication edges, removing specialist or critic agents, or corrupting intermediate messages—while keeping tasks, models, prompts, and budget limits constant. The benchmark tests 17 configurations across 29 datasets in six domains, including 400 new G‑MAS‑Complex tasks that require agents to combine and reconcile information from multiple documents.

By Kairui Yang, Xunkai Li, Kaixiang Zhang, Minghao An, Zekai Chen, Yuxuan Ba, Rong-Hua Li
llmsagentsbenchmarks
More like this →
arXiv AI
Oct 21

MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems

MAS-Guard is a unified framework designed to safeguard both single- and multi-agent large language model (LLM) systems. It introduces a local Agent Guardrail (A-Guard) layer that uses task-specific experts to evaluate agentic events, and a system-level Control Plane that applies deterministic structural checks and semantic assessments to routes, capabilities, and message content. The framework enforces interventions—allow, reflection, confirmation, and block—across benchmarks, reducing attack success rates by 71.5‑86.3% while maintaining utility on benign tasks.

By Mohammad Kachuee, Phu Mon Htut, Sheng Li, Nikolaos Pappas, Miguel Ballesteros Martinez, Yassine Benajiba, Sujith Ravi, Dan Roth
llmsagentsroboticsbenchmarkssafety
More like this →