The paper argues that the architecture of multi‑agent large language model (LLM) frameworks, rather than just the intelligence of the underlying models, largely determines system performance. It introduces a taxonomy of architectural dimensions—such as orchestration, memory, planning interfaces, specialization, and communication topology—and presents MAFBench, a unified evaluation suite. An empirical study across nine frameworks, keeping the LLM constant, reveals six design principles and shows that choices like orchestration and communication topology can dramatically affect latency, accuracy, and coordination success.
arXiv:2606. 01385v1 Announce Type: cross Abstract: Software architecture design is a critical yet inherently complex and knowledge-intensive phase that requires balancing competing quality attributes and adapting to evolving requirements.
UniACE is a unified framework that standardizes the evaluation of large language model (LLM) agents by representing each benchmark as an instruction–tool–environment triplet and running models through a shared, task‑agnostic harness in isolated runtimes. It preserves native success criteria, offers an offline mode for dynamic‑resource tasks, and standardizes efficiency metrics, execution records, and failure attribution. Applying UniACE to 7 benchmarks across 24 domains and 15 models revealed significant score shifts, ranking reversals, and sensitivity to evidence representation, highlighting the impact of evaluation configuration on reported agent performance.
arXiv:2605. 27898v2 Announce Type: replace Abstract: As LLMs are increasingly deployed as agents, reliable assessment of their agentic capabilities has become essential.
arXiv:2606. 13003v1 Announce Type: new Abstract: Prevailing wisdom posits that Multi-Agent Systems (MAS) are superior to Single-Agent Systems (SAS), citing advantages like context protection, parallel processing and distributed decision-making.
MASBench is a benchmark for evaluating large language model (LLM)-based multi‑agent systems (MAS) under partial observability, a realistic constraint where agents only see limited environmental information. The benchmark is structured into three progressive task categories—Reasoning, Scheduling, and Game—and assesses three collaboration mechanisms: Protocol, Memory, and Routing. It provides deterministic evaluation metrics such as performance score, communication cost, and cost effectiveness, and includes empirical results across various LLM backbones to guide effective MAS design.
arXiv:2607. 28629v1 Announce Type: new Abstract: The rapid transition from reactive large language models (LLMs) to persistent, action-capable systems has exposed critical gaps in the architectural understanding of Agentic AI, particularly in separating inference, orchestration, and execution layers for autonomous AI agents.
The paper introduces the Agent Planning Benchmark (APB), a diagnostic framework designed to evaluate the planning capabilities of large language model (LLM) agents. APB contains 4,209 multimodal cases across 22 domains and five settings, testing holistic planning, step-wise planning with feedback, and robustness to extraneous or broken tools and unsolvable tasks. Evaluation on 12 MLLMs reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement, while APB-guided refinement improves plan correctness and downstream execution metrics on ToolSandbox and τ²‑bench tasks.
arXiv:2608.23035v1 Announce Type: new Abstract: As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capabili...
arXiv:2606. 01533v1 Announce Type: cross Abstract: Computer use agents (CUAs) today are primarily deployed as single serial agents.
EMMA is a framework that treats the configuration of a multi‑agent system—controller routing, specialist prompts, and procedural skills—as structured episodic memory. It explicitly assigns ownership and limits updates through a reflector and backbone‑independent guards, ensuring edits are scoped, incremental, and bounded. Experiments on KramaBench, AppWorld, and BBEH show EMMA outperforms four baselines, achieving up to 32.5% higher performance and reducing training executions by 17–38%.
arXiv:2608.27969v2 Announce Type: replace Abstract: Long-horizon coding agents operate over evolving repository states while increasingly relying on heterogeneous capabilities, delegated agents, and...
Jev-LiteMAS is a multi‑agent framework that separates decision‑making from generative execution in large‑language‑model workflows. By using a lightweight decision model (Jev) to dynamically select experts and screen artifacts, it closes the planning and review gaps left by pure generative planners and reviewers. Experiments on four benchmarks show that Jev‑LiteMAS cuts generative‑LLM token usage by 12.7–54.6% while keeping accuracy within 1.34 percentage points and reducing latency in most settings.