Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

The paper critiques current agent memory benchmarks that rely on downstream QA accuracy, arguing they conflate memory correctness with retrieval and generation quality, leading to misleading results. It introduces FEB-30, a new benchmark of 30 fact‑evolution trajectories across six diagnostic categories and three difficulty levels, generated automatically to provide deterministic ground truth. Using FEB-30, the authors show a significant divergence between intrinsic and extrinsic evaluations, revealing that a representative memory system correctly classifies memory operations only 20% of the time—an error invisible to traditional QA metrics.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• agents
• benchmarks