arXiv AI
Trace Integrity for LLM Data Agents: Auditing Structured-Data Computation Beyond Answer Accuracy in Real-World Systems
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Aug 27
Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems
The paper argues that answer accuracy alone is insufficient for evaluating large language model (LLM) data agents, especially in structured-data tasks where a correct answer can be produced by an invalid trace. It introduces Trace Integrity as a reliability criterion that ensures the computation behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. The authors operationalize this concept with execution contracts and present the CAIT (Correct Answer / Invalid Trace) Rate to quantify how often answer-only evaluations mistakenly reward unsupported outputs, demonstrating that accuracy, trace validity, and silent-failure risk are distinct signals.
Hugging Face Trending Papers
Aug 26
Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems
Answer accuracy is an insufficient reliability signal for LLM data agents. In structured-data tasks, a benchmark-correct answer can be produced by an invalid trace. This paper introduces Trace Integri...
arXiv Computation and Language
Aug 25
TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification
arXiv:2608.17795v2 Announce Type: replace Abstract: Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at in...
arXiv AI
Jun 2
Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability
arXiv:2606. 01365v1 Announce Type: new Abstract: Tool-using multi-agent large language model (LLM) systems spend computation through model tokens, tool calls, retries, and code execution before producing an answer.
arXiv AI
Sep 11
ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance
ContractEval is a diagnostic framework that makes active obligations in procedural instructions explicit by representing them as query‑conditioned obligations. It matches these obligations against response or trace evidence, identifying omissions, wrong branches, ordering errors, extra actions, invariant breaches, and output‑contract violations as distinct conformance failures. In tests on audited procedural contracts, ContractEval detects and localizes all injected structural failures that output‑only and trace‑aware LLM judges miss, though it is not a compliance guarantee and remains calibration‑sensitive.
arXiv AI
Oct 26
Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
The paper introduces TraceTrial-Bench, a benchmark that evaluates the reliability of AI agent harnesses across seven components—context, execution, governance, lifecycle, observability, tools, and verification—using 95 expert-level tasks with executable or state-based ground truth. It employs a Trajectory Tracing module to record agent execution, a two-layer evaluation framework to separately assess agent performance and harness reliability, and a Multi-source Evidence Acquisition module to gather evidence for LLM-based evaluation. The benchmark’s analysis of 3,420 runs across four harnesses and nine models shows harness scores between 59.8% and 72.0%, with 31.1% of runs succeeding despite failing at least one harness criterion, highlighting the need for auditable and reproducible harness evaluation.