DreamBench‑SWE is a multi‑session benchmark designed to evaluate software agents’ memory hygiene, where later tasks rely on evidence from earlier sessions and are scored by executable hidden oracles. The study reports results from an original scaled v2 fold and a preregistered v2.1 successor audit, detailing performance across various memory configurations and conditions. The findings confirm that DreamBench‑SWE can discriminate between different memory setups, but do not establish superiority, equivalence, or broad generality of any external‑system mechanism.
DreamBench-SWE is a multi-session benchmark for software-agent memory hygiene in which later software tasks depend on non-inferable evidence from earlier sessions and are scored by executable hidden o...
arXiv:2608.21867v1 Announce Type: new Abstract: LLM agents are moving from single-prompt use to long task streams in which reusable memory becomes a core capability for terminal, software-engineering...
arXiv:2607. 27080v1 Announce Type: cross Abstract: Memory systems allow agents to retain and reuse information from past interactions, but they can also let malicious content persist.
arXiv:2610.27714v1 Announce Type: new Abstract: When does experience memory help a closed-loop LLM driving agent? Published gains cannot say, because the two factors we find to matter most, which war...
The paper introduces LSREP, a Longitudinal State‑Replay Evaluation Protocol designed to assess how conversational memory evolves over time, incorporating ordered replay, lifecycle schedules, repeated probes, evolving reference answers, and mechanism‑fidelity checks. It applies LSREP to ICE v2, a local‑first memory middleware, and reports that on three ordinary‑density datasets ICE v2 achieves near‑zero mean quality difference from vector‑RAG while using fewer fragments but slightly more prompt tokens, yet fails catastrophically on a dense dataset. In a public diagnostic, ICE v2 underperforms pure vector‑RAG on LongMemEval, revealing significant multi‑session and temporal failures and a quality‑cost trade‑off rather than superior efficiency.
arXiv:2608. 07429v1 Announce Type: new Abstract: Long-term memory enables language agents to reuse past facts, preferences, and task experience.
The paper investigates how persistent memory in AI agents can lead to over‑trust in stale facts, creating a "Memory Trust Gap" that worsens as model capability increases. Using a benchmark with Benefit and Safety suites across Qwen3 models of varying sizes, the authors show that larger models are more prone to harmful over‑trust, especially when metadata is absent or misleading. They also demonstrate that mitigation strategies such as exposing metadata or pre‑resolving conflicts improve accuracy, but the effectiveness depends on model size and dataset.
The paper introduces a scale‑conditioned evaluation protocol for agent memory that measures how usable evidence degrades as irrelevant sessions accumulate. It logs memory trajectories and reports four diagnostics—budget‑compliant reliability, tail memory‑call burden, failure‑regime decomposition, and the usable‑scale boundary. Experiments on LongMemEval and LoCoMo show that reliability loss varies across memory interfaces and agents, highlighting that scalable‑memory claims must be conditioned on agent, interface, scale range, and interaction budget.
arXiv:2608. 12476v1 Announce Type: new Abstract: Long-term agent memory is usually treated as select--store--retrieve, but retrieval does not decide whether contradictory, superseded, retracted, deleted, or stale records may support an outgoing claim.
arXiv:2608. 13883v1 Announce Type: new Abstract: Most agent-memory benchmarks test post-hoc recall, whereas MemoryArena evaluates whether memory supports interdependent, multi-session task completion.
arXiv:2606. 01138v2 Announce Type: replace-cross Abstract: Agent-memory frameworks -- mem0, Letta/MemGPT, Cognee, Zep/Graphiti, MemoryOS, MemTensor -- each ship their own SDK, storage layout, and operational vocabulary.
arXiv:2609.23570v1 Announce Type: cross Abstract: Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluatio...