arXiv AI
Oct 5

Lightweight, Rubric-Guided Trajectory Evaluation for Production AI Agents

LiteTrajEval is a lightweight architecture designed for budget‑bounded trajectory evaluation of large‑language‑model agents. It creates compact domain‑specific rule profiles offline, preprocesses trajectories online to flag heuristic failure signals, and then uses a single rubric‑guided LLM judge to generate structured diagnostic reports. On public Magnetic‑One‑style and τ‑bench‑style datasets, LiteTrajEval improves failure‑localization alignment with human annotations by 20–35 percentage points and reduces cost and evaluation time by 6× and 8× respectively, and has been deployed in an enterprise agentic platform.

By Linh-An Phan, MingXue Wang, Guangyu Wu, Feng Pan, Zhaoyu Pang, Yanbin Zhang
arXiv AI
Jun 8

TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents

arXiv:2606. 07054v1 Announce Type: cross Abstract: Autonomous LLM agents can pursue hidden malicious objectives through sequences of individually benign actions, making sabotage difficult to detect using standard trajectory-level monitoring.

By Vijitha Mittapalli, Shreyaa Jayant Dani, Satya Srujana Pilli, Snigdha Ansu, Mohammadreza Teymoorianfard, Franck Dernoncourt, Hongjie Chen, Yu Wang, Ryan A. Rossi, Nesreen K. Ahmed
arXiv AI
Oct 7

A Trust Layer for Agent Evaluation

The paper introduces a Trust Layer for Agent Evaluation, a post‑hoc framework that appends a belief flag to each benchmark score. It checks four properties: alignment with the benchmark’s grading logic, traceable computation of a passing answer, consistency between the agent’s claim and actual behavior, and stability across repeated runs. Applied to five agent configurations on 108 tasks, the study finds that only 22.6% of recorded passes satisfy all checks, revealing widespread issues such as untraceable computation, false completion claims, and unstable results.

By Mohammadreza Sediqin, Shivali Dalmia, Srinivasa Karthikeya Reddy Kovvuri, Abhishek Mukherji
arXiv AI
Nov 6

Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration

The paper introduces a closed‑loop evaluation framework for deployed agents, moving beyond one‑off benchmarks to continuous, multi‑level assessment. It defines four analysis levels—span, trace, conversation, and engagement—each measured along Execution, Substance, Trajectory, and Results, while enforcing Conduct constraints. The method employs provenance, adjudication, and evaluator agents from an Agent Factory, with inner and outer loops auditing evidence and driving agent improvement, demonstrated on the SharpInsight business‑analysis agent and public conversation data.

By Ruifeng Ren, Gun Ahn