arXiv:2606. 05872v1 Announce Type: new Abstract: AI agents are commonly evaluated using task success, reward, latency, and cost.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
AI agents are commonly evaluated using task success, reward, latency, and cost. These metrics are useful, but they often miss important aspects of agent behavior: whether an agent explores too much, repeats itself too rigidly, uses tools effectively, reduces uncertainty over time, or remains robust across repeated runs.
AuditKIT is a source‑available evaluation library that unifies benchmark accuracy, judge‑based assessment, runtime performance, tool‑use behavior, and retrieval quality into a single result representation. It offers direction‑aware metric contracts, configuration fingerprints, and extensible metrics and backends, while providing turn‑aware tool‑call matching and distinguishing verified task outcomes from trajectory‑level diagnostics. Seven case studies demonstrate how answer extraction, runtime behavior, and metric selection can alter evaluation outcomes across model comparison, serving runtimes, retrieval, tool calling, and interoperability.
arXiv:2602. 16666v3 Announce Type: replace Abstract: AI agents are increasingly deployed to execute important tasks.
AgentTune is a source‑available Python library that offers a unified training lifecycle for tool‑using large language model agents. It introduces a two‑tier EventLog representation to preserve agent trajectories and enable consistent training, evaluation, and auditing. The framework supplies a shared runtime, reinforcement learning for tool‑calling, agentic distillation, retrieval‑augmented generation, decision workflows, evaluation utilities, and a path toward self‑healing agents, all exposed through composable interfaces for flexible combination of strategies and methods.
The article "Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks" surveys the lack of a standard definition for AI agents and organizes this ambiguity into five dimensions: environmental interaction, learning and adaptation, autonomy, goal‑directed behavior, and temporal coherence. It reviews how each dimension has been conceptualized in prior work and compiles the metrics, benchmarks, and evaluation frameworks used to assess them. The authors also introduce the Agent Compendium, a public digital resource that extends these evaluation methods, aiming to provide a common structure for evaluating and comparing agent capabilities across AI systems.
arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...