arXiv AI
ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
6d ago
ATLAS: Dual-Horizon Diagnostic Evaluation for Tool-Use Agents in Meituan Xiaotua
ATLAS is a dual‑horizon diagnostic evaluation framework designed for industrial tool‑use agents on web‑scale service platforms. It combines within‑turn trajectory‑wise signals (thinking, reflection, tool execution, response generation) with across‑turn user‑wise signals over multiple lifecycles to pinpoint execution failures and unmet capabilities. The framework employs rule‑based checks and calibrated LLM judges, distilling them into compact diagnostic models that enable developers to attribute failures, prioritize fixes, and reassess improvements. In Meituan Xiaotuan, ATLAS‑guided agents achieved higher recommendation relevance, factual reliability, user engagement (+2.84% answer read‑through rate), and order volume (+6.13% paid orders), leading to full production deployment.
arXiv AI
Oct 19
AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems
AuditKIT is a source‑available evaluation library that unifies benchmark accuracy, judge‑based assessment, runtime performance, tool‑use behavior, and retrieval quality into a single result representation. It offers direction‑aware metric contracts, configuration fingerprints, and extensible metrics and backends, while providing turn‑aware tool‑call matching and distinguishing verified task outcomes from trajectory‑level diagnostics. Seven case studies demonstrate how answer extraction, runtime behavior, and metric selection can alter evaluation outcomes across model comparison, serving runtimes, retrieval, tool calling, and interoperability.
arXiv Computation and Language
Oct 22
Bridging Foundational and Agentic Capabilities of LLMs
The paper introduces a framework that links the foundational capabilities of large language models (LLMs) to their agentic performance by measuring operation-level gaps. It constructs foundational profiles from benchmark data and operation profiles from real interaction trajectories, then quantifies how much observed operation performance deviates from predictions based on foundational capabilities. Experiments on fifteen model services in the τ²-Bench demonstrate that foundational profiles alone only partially predict agentic outcomes, while incorporating operation profiles significantly improves task-score predictions and reveals consistent execution gaps across task subsets.
arXiv AI
Sep 2
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
UniACE is a unified framework that standardizes the evaluation of large language model (LLM) agents by representing each benchmark as an instruction–tool–environment triplet and running models through a shared, task‑agnostic harness in isolated runtimes. It preserves native success criteria, offers an offline mode for dynamic‑resource tasks, and standardizes efficiency metrics, execution records, and failure attribution. Applying UniACE to 7 benchmarks across 24 domains and 15 models revealed significant score shifts, ranking reversals, and sensitivity to evidence representation, highlighting the impact of evaluation configuration on reported agent performance.
arXiv AI
Aug 26
ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents
ToolRobustBench is a stage-wise diagnostic benchmark designed to evaluate and diagnose failures in tool‑calling agents, which are large language models that select tools, provide structured arguments, and interpret tool feedback. The benchmark aligns four perturbation families—tool‑interface, user‑intent, tool‑output/observation, and runtime‑environment—with the tool‑use pipeline, attributing failures to specific stages such as tool selection, schema grounding, argument binding, and feedback handling. Experiments across 15,456 instances, 7 models, and 16 local tools reveal that while overall performance is high, robustness degrades significantly, especially under tool‑output/observation perturbations, and mixed‑family perturbations produce non‑additive failure patterns.
arXiv AI
Jun 2
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.