arXiv AI
Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 8
Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
arXiv:2605. 21347v3 Announce Type: replace Abstract: Diagnosing failures in LLM agents remains largely manual.
arXiv AI
Aug 19
AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysis
AdaLens is an interactive system designed to monitor and steer long-running, autonomous data analysis workflows powered by large language models. It provides a storyline-based interface that unifies analytical plans, execution progress, intermediate findings, and data-column involvement, enabling analysts to observe evolving reasoning and evidence. The system also offers steering interactions that allow users to redirect low-value directions or deepen promising ones during execution.
arXiv AI
Jun 9
Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting
arXiv:2606. 09809v1 Announce Type: new Abstract: AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs.
arXiv Computation and Language
Sep 11
Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers
The article introduces AgentActionBench, a benchmark designed to evaluate agent-based experiment reproduction across machine learning and AI4Science papers. It employs an MCP-based Action Recorder to capture agents’ behavior during reproduction and assesses the resulting traces against paper-specific rubrics. The benchmark includes 150 papers, with a human-annotated subset and model-assisted augmentation expanding it to over 10,000 rubric items, revealing that current systems face execution bottlenecks but that model-generated rubrics correlate strongly with human judgments.
arXiv AI
Jun 2
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.
arXiv AI
Aug 11
$A^2E$ : An End-to-End Agent Auditing Engine
arXiv:2608. 07346v2 Announce Type: replace Abstract: With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains.