Hugging Face Blog

Trace & Evaluate your Agent with Arize Phoenix

arXiv AI
Oct 5

Lightweight, Rubric-Guided Trajectory Evaluation for Production AI Agents

LiteTrajEval is a lightweight architecture designed for budget‑bounded trajectory evaluation of large‑language‑model agents. It creates compact domain‑specific rule profiles offline, preprocesses trajectories online to flag heuristic failure signals, and then uses a single rubric‑guided LLM judge to generate structured diagnostic reports. On public Magnetic‑One‑style and τ‑bench‑style datasets, LiteTrajEval improves failure‑localization alignment with human annotations by 20–35 percentage points and reduces cost and evaluation time by 6× and 8× respectively, and has been deployed in an enterprise agentic platform.

By Linh-An Phan, MingXue Wang, Guangyu Wu, Feng Pan, Zhaoyu Pang, Yanbin Zhang
arXiv AI
Jun 8

TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents

arXiv:2606. 07054v1 Announce Type: cross Abstract: Autonomous LLM agents can pursue hidden malicious objectives through sequences of individually benign actions, making sabotage difficult to detect using standard trajectory-level monitoring.

By Vijitha Mittapalli, Shreyaa Jayant Dani, Satya Srujana Pilli, Snigdha Ansu, Mohammadreza Teymoorianfard, Franck Dernoncourt, Hongjie Chen, Yu Wang, Ryan A. Rossi, Nesreen K. Ahmed
arXiv AI
Oct 7

A Trust Layer for Agent Evaluation

The paper introduces a Trust Layer for Agent Evaluation, a post‑hoc framework that appends a belief flag to each benchmark score. It checks four properties: alignment with the benchmark’s grading logic, traceable computation of a passing answer, consistency between the agent’s claim and actual behavior, and stability across repeated runs. Applied to five agent configurations on 108 tasks, the study finds that only 22.6% of recorded passes satisfy all checks, revealing widespread issues such as untraceable computation, false completion claims, and unstable results.

By Mohammadreza Sediqin, Shivali Dalmia, Srinivasa Karthikeya Reddy Kovvuri, Abhishek Mukherji
arXiv AI
Nov 6

Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration

The paper introduces a closed‑loop evaluation framework for deployed agents, moving beyond one‑off benchmarks to continuous, multi‑level assessment. It defines four analysis levels—span, trace, conversation, and engagement—each measured along Execution, Substance, Trajectory, and Results, while enforcing Conduct constraints. The method employs provenance, adjudication, and evaluator agents from an Agent Factory, with inner and outer loops auditing evidence and driving agent improvement, demonstrated on the SharpInsight business‑analysis agent and public conversation data.

By Ruifeng Ren, Gun Ahn
arXiv AI
Oct 26

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

The paper introduces TraceTrial-Bench, a benchmark that evaluates the reliability of AI agent harnesses across seven components—context, execution, governance, lifecycle, observability, tools, and verification—using 95 expert-level tasks with executable or state-based ground truth. It employs a Trajectory Tracing module to record agent execution, a two-layer evaluation framework to separately assess agent performance and harness reliability, and a Multi-source Evidence Acquisition module to gather evidence for LLM-based evaluation. The benchmark’s analysis of 3,420 runs across four harnesses and nine models shows harness scores between 59.8% and 72.0%, with 31.1% of runs succeeding despite failing at least one harness criterion, highlighting the need for auditable and reproducible harness evaluation.

By Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao, Sicheng Zhou
arXiv AI
Nov 5

Signals: Continuous Diagnostics of AI Agents

The paper "Signals: Continuous Diagnostics of AI Agents" introduces a method for diagnosing problems in AI agents by analyzing agentic traces. It defines 45 problem types across eight families, categorizes findings into engineering issues called Signals, and assesses their severity. The method was evaluated on public agent executions, achieving a 91.43% F1 score in matching Signals to reference issues, outperforming a one-pass LLM control by about 18 percentage points.

By Amey Rambatla, Vatsal Goel, Sagar Gaur, Siddhant Loya, Anurag Mishra, Asif Mohammad
arXiv AI
1d ago

Can You Trust Your Finance Agent? Drift, Diagnosis and Auditable Intervention on One Shipped Financial Research Agent

The paper evaluates the trustworthiness of a shipped finance agent from the Vals Finance Agent Benchmark v2 by running it unmodified across multiple trials. It reports that the agent’s leaderboard score is unstable, with performance varying by up to 17% and passing only 105–123 of 149 rubric criteria. The authors diagnose failures by attributing them to specific agent components, confirm the diagnosis with blind readers, and test an automated optimization loop that can reduce cost or improve accuracy, though the latter’s gains are within noise.

By Cosimo Galeone, Anna Ettorre, Ginevra Mariani, Minsu Park, Giuseppe Ettorre, Daniele Ligorio
arXiv AI
Sep 10

Explaining AI Agents Through Execution Traces

arXiv:2609.06063v1 Announce Type: new Abstract: AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human ov...

By Vittoria Vineis, Fabiano Veglianti, Lorenzo Antonelli, Claudia Di Carlo, Matteo Silvestri, Gabriele Tolomei