arXiv Machine Learning
Jun 25
Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One
arXiv:2606. 25449v1 Announce Type: cross Abstract: A language model's memory can be worse than having no memory at all.
Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One
A language model's memory can be worse than having no memory at all. Give a model a memory that kept a wrong conclusion but dropped the work behind it, and it emits that stale value as a confident answer; give the same model an empty memory and it abstains.
Related stories
arXiv AI
Sep 7
Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents
arXiv:2609. 04875v1 Announce Type: cross Abstract: Long-running LLM agents are stateful: beyond the transcript they accrete compressed summaries, plaintext memory, pending tool plans, and, under every serving API, a KV cache.
arXiv AI
Aug 12
From Faulty Memories to Corrected Actions: Dependency-Guided Rollback Repair for Memory-Augmented Agents
arXiv:2608. 10502v1 Announce Type: new Abstract: Persistent memory lets language-model agents reuse information across sessions, but it also makes errors durable: a poisoned, stale, or misattributed record can alter reasoning, tool use, answers, and subsequent memory writes.
arXiv Machine Learning
Sep 10
Can an AI Assistant Really Forget? Auditable Deletion from Addressable Memory
This paper introduces a deletion interface for a pretrained language model, measuring how effectively deleted records are removed from the model’s memory. By retrofitting a support‑vector memory gate into the global attention layers of a frozen Gemma 3, the authors show that deletions can be performed without altering weights and that the resulting state is close to a reference state that never stored the record. Experiments on 4B‑parameter models demonstrate low perplexity impact and strong evidence that deleted content is hard to recover, while larger or smaller models fail to achieve the same guarantees.
arXiv AI
Aug 14
Governed Persistent Memory: Source-Bound State Semantics and Fail-Closed Release for Long-Horizon Agents
arXiv:2608. 12476v1 Announce Type: new Abstract: Long-term agent memory is usually treated as select--store--retrieve, but retrieval does not decide whether contradictory, superseded, retracted, deleted, or stale records may support an outgoing claim.
arXiv AI
Aug 14
Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues
arXiv:2608. 12599v1 Announce Type: new Abstract: Multi-turn dialogues let users revoke constraints as easily as impose them, but revocation does not reliably take effect: models keep enacting withdrawn requirements (occasionally beneath comments asserting their removal), a failure we call \emph{behavioral relapse}, or revocation inertia.
arXiv Computation and Language
Sep 23
MoM: Memory of Memory
arXiv:2609.25054v1 Announce Type: new Abstract: For a long-horizon LLM agent, the memory question is not what was once recorded but what \emph{currently holds}. Most designs answer it only indirectly...
arXiv AI
Sep 10
What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory
arXiv:2609.08279v1 Announce Type: cross Abstract: Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the...
arXiv AI
Sep 10
When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents
The paper introduces MERIT, a benchmark that evaluates the marginal benefit of long‑term memory for tool‑using large language model agents while explicitly accounting for cost. MERIT provides episodic tool‑use tasks across three domains, verifies dependence on earlier‑episode facts, and measures memory operations in tokens and dollars. Experiments on GPT‑4.1‑mini, Claude Haiku 4.5, and Claude Sonnet 5 show that memory can significantly improve task success, but its utility varies widely across models and memory implementations, and full replay is rarely cost‑effective.
arXiv AI
Sep 2
trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
The paper "trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories" examines the limitations of outcome-only evaluation for large language model agents. Using a deterministic tool‑using support‑desk environment with a scripted oracle policy and a fault injector, the authors compare five different judging approaches—programmatic rules, outcome‑only, step‑rubric at two model sizes, and a self‑consistency ensemble—on metrics such as detection, step localisation, fault typing, calibration, and cost across 400 trajectories. The study finds that outcome‑only judges miss many silent faults and generate false positives, while step‑rubric judges achieve higher recall with no false alarms but at greater cost, and that none of the judges read the final reply, allowing fabricated promises to evade detection. "whyItMatters":"The findings highlight that current production‑default outcome‑only evaluations can overlook critical failures in agent behavior, underscoring the need for more nuanced, step‑level judging methods to ensure reliable LLM agent performance."
arXiv Machine Learning
Jul 20
Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation
arXiv:2607. 16019v1 Announce Type: new Abstract: AI systems increasingly retrieve from records that revise themselves: issue threads, encyclopedic histories, policy logs, and long conversations.
arXiv AI
Aug 10
TEPA: Revoking Stale Memories for Conflict-Robust Language Agents
arXiv:2608. 07429v1 Announce Type: new Abstract: Long-term memory enables language agents to reuse past facts, preferences, and task experience.