arXiv AI
What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
5d ago
Canonical Procedural Actions: An Auditable Annotation Protocol for Tool-Use Agent Traces
arXiv:2609.24264v1 Announce Type: new Abstract: Tool-use agent traces identify messages and API calls, but procedural analyses also need explicit units of action and inspectable links to their eviden...
arXiv AI
Jun 3
StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systems
arXiv:2606. 03467v1 Announce Type: new Abstract: LLM-based multi-agent systems exhibit remarkable collaborative capabilities in complex multi-step tasks.
arXiv AI
Jul 13
Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
arXiv:2607. 09510v1 Announce Type: cross Abstract: Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern.
arXiv AI
Sep 1
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...
arXiv AI
Jun 9
REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces
arXiv:2606. 09071v1 Announce Type: new Abstract: Large language model (LLM) agents now solve complex tasks through long plan-and-execution traces, yet the ability to locate errors in a completed traces still lags far behind, especially in the \emph{silent failure} regime.
arXiv AI
Aug 17
Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model
arXiv:2608. 13867v1 Announce Type: cross Abstract: AI coding agents are commonly evaluated as models but deployed as systems.