AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation
arXiv:2607. 06624v1 Announce Type: new Abstract: We present AgentLens, a production-assessed benchmark for interactive code agents.
arXiv:2607. 06624v1 Announce Type: new Abstract: We present AgentLens, a production-assessed benchmark for interactive code agents.
LiteTrajEval is a lightweight architecture designed for budget‑bounded trajectory evaluation of large‑language‑model agents. It creates compact domain‑specific rule profiles offline, preprocesses trajectories online to flag heuristic failure signals, and then uses a single rubric‑guided LLM judge to generate structured diagnostic reports. On public Magnetic‑One‑style and τ‑bench‑style datasets, LiteTrajEval improves failure‑localization alignment with human annotations by 20–35 percentage points and reduces cost and evaluation time by 6× and 8× respectively, and has been deployed in an enterprise agentic platform.
arXiv:2606. 07054v1 Announce Type: cross Abstract: Autonomous LLM agents can pursue hidden malicious objectives through sequences of individually benign actions, making sabotage difficult to detect using standard trajectory-level monitoring.
The paper introduces a Trust Layer for Agent Evaluation, a post‑hoc framework that appends a belief flag to each benchmark score. It checks four properties: alignment with the benchmark’s grading logic, traceable computation of a passing answer, consistency between the agent’s claim and actual behavior, and stability across repeated runs. Applied to five agent configurations on 108 tasks, the study finds that only 22.6% of recorded passes satisfy all checks, revealing widespread issues such as untraceable computation, false completion claims, and unstable results.
The paper introduces a closed‑loop evaluation framework for deployed agents, moving beyond one‑off benchmarks to continuous, multi‑level assessment. It defines four analysis levels—span, trace, conversation, and engagement—each measured along Execution, Substance, Trajectory, and Results, while enforcing Conduct constraints. The method employs provenance, adjudication, and evaluator agents from an Agent Factory, with inner and outer loops auditing evidence and driving agent improvement, demonstrated on the SharpInsight business‑analysis agent and public conversation data.
arXiv:2606. 19782v1 Announce Type: new Abstract: Financial chart question answering in regulated settings demands more than accuracy: practitioners must know which answers to trust before acting on them, and many institutions cannot send client data to external model providers.
arXiv:2607. 07052v1 Announce Type: cross Abstract: AI agents deployed for IT operations are typically permanent cost centers because every execution requires full LLM inference, even for previously solved problems.
The paper introduces TraceTrial-Bench, a benchmark that evaluates the reliability of AI agent harnesses across seven components—context, execution, governance, lifecycle, observability, tools, and verification—using 95 expert-level tasks with executable or state-based ground truth. It employs a Trajectory Tracing module to record agent execution, a two-layer evaluation framework to separately assess agent performance and harness reliability, and a Multi-source Evidence Acquisition module to gather evidence for LLM-based evaluation. The benchmark’s analysis of 3,420 runs across four harnesses and nine models shows harness scores between 59.8% and 72.0%, with 31.1% of runs succeeding despite failing at least one harness criterion, highlighting the need for auditable and reproducible harness evaluation.
The paper "Signals: Continuous Diagnostics of AI Agents" introduces a method for diagnosing problems in AI agents by analyzing agentic traces. It defines 45 problem types across eight families, categorizes findings into engineering issues called Signals, and assesses their severity. The method was evaluated on public agent executions, achieving a 91.43% F1 score in matching Signals to reference issues, outperforming a one-pass LLM control by about 18 percentage points.
The paper evaluates the trustworthiness of a shipped finance agent from the Vals Finance Agent Benchmark v2 by running it unmodified across multiple trials. It reports that the agent’s leaderboard score is unstable, with performance varying by up to 17% and passing only 105–123 of 149 rubric criteria. The authors diagnose failures by attributing them to specific agent components, confirm the diagnosis with blind readers, and test an automated optimization loop that can reduce cost or improve accuracy, though the latter’s gains are within noise.
arXiv:2609.06063v1 Announce Type: new Abstract: AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human ov...