Counterfactual Evidence Audits Predict LLM-Agent Susceptibility to Ranked Context

The paper introduces a counterfactual evidence audit to assess how large language model agents respond to different sets of documents. By exposing agents to two mirrored sets of five documents and measuring changes in six downstream decisions, the authors predict how the agents will behave when presented with a full 45‑document context. Across multiple open‑weight model families and tasks, the five‑document audit accurately forecasts full‑context effects, reducing prediction error by 62% and correctly identifying the direction of most material effects.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• rag
• agents