arXiv AI
Jul 7
Toward Efficient Agents: Memory, Tool learning, and Planning
arXiv:2601. 14192v2 Announce Type: replace Abstract: Recent years have witnessed increasing interest in extending large language models into agentic systems.
Rethinking the Evaluation of Efficiency Methods for Multi-Agent Systems
The paper critiques current evaluations of efficiency methods for large language model–based multi‑agent systems, arguing that reported gains are often inflated by method‑specific prompts and starting topologies. It introduces a controlled, MAS‑demanding diagnostic benchmark that standardizes the backbone model, agent registry, and runtime, and systematically varies topology, scale, depth, and tool use. The authors find that many claimed efficiency improvements are setup‑dependent, sometimes stemming from structural collapse or random pruning rather than genuine, robust gains.
Related stories
arXiv AI
Aug 10
An End-to-End Agent Auditing Engine
arXiv:2608. 07346v1 Announce Type: new Abstract: With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains.
arXiv AI
Aug 11
$A^2E$ : An End-to-End Agent Auditing Engine
arXiv:2608. 07346v2 Announce Type: replace Abstract: With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains.
arXiv AI
Sep 2
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
UniACE is a unified framework that standardizes the evaluation of large language model (LLM) agents by representing each benchmark as an instruction–tool–environment triplet and running models through a shared, task‑agnostic harness in isolated runtimes. It preserves native success criteria, offers an offline mode for dynamic‑resource tasks, and standardizes efficiency metrics, execution records, and failure attribution. Applying UniACE to 7 benchmarks across 24 domains and 15 models revealed significant score shifts, ranking reversals, and sensitivity to evidence representation, highlighting the impact of evaluation configuration on reported agent performance.
arXiv AI
Sep 7
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Harbor Adapters is a unified evaluation infrastructure that ports over 80 agentic benchmarks, enabling arbitrary agents to be tested across complex environments. The authors performed a large‑scale evaluation of 8 models on 54 benchmarks, using Terminus‑2 and three native harnesses, revealing detailed agent capabilities and failure modes. They also created Harbor‑Index, a curated set of 82 challenging tasks from 29 benchmarks, designed to be affordable yet comprehensive, with the best model achieving a 28.0% pass rate.
arXiv AI
Aug 7
Matching Matters: A Fair Quality-Efficiency Benchmark for Command-Line Agents
arXiv:2606. 21140v2 Announce Type: replace-cross Abstract: Rapid advances in large language models have improved the task-solving capabilities of command-line-interface (CLI)-based agents, whose CLIs determine how models invoke tools, maintain interaction history, and recover from failures.
arXiv AI
Jun 15
MASLab: A Unified and Comprehensive Codebase for LLM-based Multi-Agent Systems
arXiv:2505. 16988v2 Announce Type: replace-cross Abstract: LLM-based multi-agent systems (MAS) have demonstrated significant potential in enhancing single LLMs to address complex and diverse tasks in practical applications.
arXiv AI
Sep 15
MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
arXiv:2512.24565v4 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol...
arXiv Machine Learning
Jul 7
MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
arXiv:2605. 08678v3 Announce Type: replace Abstract: Modern AI progress has been driven by ML methods that are generalizable across settings and scalable to larger regimes.
arXiv AI
Jul 16
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
arXiv:2607. 13705v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical.
arXiv AI
Jul 24
ExecuGraph: A Multi-Agent, Execution-Grounded Framework for Reliable Backend Code Synthesis with Large Language Models
arXiv:2607. 20499v1 Announce Type: new Abstract: Large Language Models generate plausible backend code, but a single-pass paradigm provides no guarantee of correctness or runtime reliability.
arXiv AI
Jul 7
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
arXiv:2607. 05174v1 Announce Type: new Abstract: Language agents, i.