arXiv Machine Learning By Yue Zhao

GRADE: Graph Representation of LLM Agent Dependency and Execution

Read the original on arXiv Machine Learning →

The paper introduces GRADE, a graph-based representation of large language model (LLM) agent executions that captures both execution steps and their dependencies. By adding graded dependency edges—observed, declared, or inferred—to the trace, the authors evaluate how this dependency layer affects failure prediction across six corpora involving tool use, coding, and web tasks. Experiments show that the dependency block can improve prediction in some settings, but its effectiveness varies with the evaluation probe and corpus, and controlled experiments demonstrate that the observed structure is not merely a degree-matched artifact.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Oct 7

Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently?

The study evaluates how runtime gates for agent tool calls stack, assuming their errors multiply. Using 1,119 labeled agent actions from three corpora, the authors compare a deterministic rule layer with four LLM judges, measuring the effective number of multiplication-equivalent layers (n_mult). Results show that two judges compose to about 1.2–1.4 layers, while a rule layer plus one judge composes to roughly 1.86–2.09 layers, with variations depending on miss definitions and corpora.

By Chenglin Yang
arXiv Computation and Language
Oct 9

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.

By Radhika Gaonkar