DuMateBench is a new benchmark for autonomous agents that uses real user sessions from a large production platform, preserving interaction history, configurations, and workspace state. It contains 200 tasks across 8 scenarios and 17 capability categories, many requiring coordination of multiple capabilities. The benchmark tests agents in Docker containers with real-world complexities—Insufficient, Unstable, and Noisy—and evaluates performance with a hybrid deterministic and LLM-as-Judge protocol, revealing significant gaps in task completion across various agent frameworks and LLMs.
arXiv:2605. 27898v2 Announce Type: replace Abstract: As LLMs are increasingly deployed as agents, reliable assessment of their agentic capabilities has become essential.
arXiv:2604. 13072v2 Announce Type: replace-cross Abstract: OpenClaw-style personal assistants extend LLM agents from isolated tool use to open-ended, stateful, and personalized software environments.
arXiv:2606. 09426v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools.
arXiv:2607. 22689v1 Announce Type: new Abstract: Graphical user interface (GUI) agents are systems powered by large multimodal models (LMMs).
RealClawBench is a live benchmark framework derived from real OpenClaw developer‑agent sessions, designed to capture the distribution, diversity, and real‑world difficulty of deployed agent use. It reconstructs execution environments and uses deterministic verifiable scorers to convert real sessions into reproducible, automatically scored tasks, yielding 281 executable tasks with minimal distribution shift. Evaluation of 14 contemporary models shows the best system solves only 65.8% of tasks, highlighting significant room for improvement on realistic workloads.
arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.
arXiv:2605. 25160v2 Announce Type: replace Abstract: GUI agents powered by large language models are advancing rapidly, creating urgent needs for evaluation and training based on realistic environments.
arXiv:2608. 15127v1 Announce Type: cross Abstract: Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state.
arXiv:2608. 00101v1 Announce Type: cross Abstract: AI coding agents like GitHub Copilot, Claude Code, and Codex interleave multi-step LLM inference with tool execution, creating a workload different from chatbots.
arXiv:2606. 02302v1 Announce Type: cross Abstract: Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services.
arXiv:2605.27898v3 Announce Type: replace Abstract: Agent benchmarks are increasingly used to compare large language models (LLMs) across domains, yet a reported score reflects a complete model--harn...
PeakBench is a new benchmark designed to evaluate how large language model agents invoke multiple tools while respecting resource constraints and parallel execution. It provides executable multi‑tool workflows with dependency annotations and measured resource profiles, and introduces a two‑part evaluation framework that separates logical planning from physical scheduling. The study shows that strong logical planning alone does not guarantee safe or efficient execution, and that providing resource information can reduce overflows and improve utilization.