arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.
arXiv:2607. 04542v1 Announce Type: cross Abstract: Every LLM agent run re-derives its behavior token by token on a frontier model: brilliant, expensive, slow, and unbounded.
arXiv:2604. 13072v2 Announce Type: replace-cross Abstract: OpenClaw-style personal assistants extend LLM agents from isolated tool use to open-ended, stateful, and personalized software environments.
BekchiAI introduces a benchmark and platform for evaluating large language model agents. The benchmark comprises 13 tool‑using ReAct agents across seven task categories, totaling 2,057 deterministic test tasks with verifier‑checkable gold answers. The platform offers web‑based observability, token and latency telemetry, and remote run termination for live agents.
arXiv:2607. 16345v1 Announce Type: cross Abstract: Modern agentic systems increasingly rely on skills: installable packages of natural language and code that teach an LLM agent to perform a domain task.
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
UniACE is a unified framework that standardizes the evaluation of large language model (LLM) agents by representing each benchmark as an instruction–tool–environment triplet and running models through a shared, task‑agnostic harness in isolated runtimes. It preserves native success criteria, offers an offline mode for dynamic‑resource tasks, and standardizes efficiency metrics, execution records, and failure attribution. Applying UniACE to 7 benchmarks across 24 domains and 15 models revealed significant score shifts, ranking reversals, and sensitivity to evidence representation, highlighting the impact of evaluation configuration on reported agent performance.
The paper introduces a coroutine-bridge harness that lets a language model emit a Python program to manage tool calls in the CAR-bench evaluation. By decoupling model invocations from tool round-trips, the approach reduces model calls to a median of two per task while maintaining seven agent turns, achieving a median latency of 1.8 s on a Cerebras gpt‑oss‑120b. The harness achieved 60.0 % Pass³ on the official hidden evaluation, outperforming the baseline by 4.5× and matching frontier-model agents on GPT‑5.5, all while keeping the prompt largely cached and minimizing input compute.
Agent Skills can specify procedural and resource obligations for tool use, and language models instantiate them as concrete programs. However, when models turn this guidance into code for existing too...
SkillEffect is a checked‑lowering runtime that ensures agent tool calls stay within memory limits by verifying each proposed program against an immutable input before execution. It uses audited relation plugins to provide source recognition, bounded intermediate representation construction, and postconditions, while a shared runtime handles selection, bounded VM execution, and atomic capacity leasing. Experiments across six operator families show that bounded access significantly reduces peak memory usage and improves completion rates under fixed memory caps.
The paper introduces an action‑class diagnostic framework for multi‑turn tool‑calling in large language model agents, breaking failures into action‑class miscalibration and action‑execution failure across a four‑class action space (TOOL_CALL, ASK, REFUSE, CONFIRM). It defines a self‑revealing upper bound (Acc GAR) to expose state‑grader masking of miscalibration and shows that miscalibration is a significant, previously hidden failure mode, especially for heavily tool‑trained families. The study demonstrates that calibration can be reshaped by context‑only perturbations, but the effects vary widely across models and perturbation mechanisms, underscoring the need for diagnostics beyond aggregate accuracy.
arXiv:2605. 27898v2 Announce Type: replace Abstract: As LLMs are increasingly deployed as agents, reliable assessment of their agentic capabilities has become essential.