AfterDeleteBench is a controlled behavioral audit designed to test whether large language model agents retain information after a memory record is deleted. The benchmark compares agents that have been exposed to a synthetic secret and then had the original memory removed against matched agents that never saw the secret, attributing any post‑deletion recovery to the initial exposure. Results show that a single query can recover deleted secrets from models such as Qwen2.5‑7B‑Instruct in about two thirds of cases, and that reconstruction can succeed even when the full secret string is absent, with success rates of 62‑99% across Qwen, Mistral, and Phi‑3.
arXiv:2609. 04875v1 Announce Type: cross Abstract: Long-running LLM agents are stateful: beyond the transcript they accrete compressed summaries, plaintext memory, pending tool plans, and, under every serving API, a KV cache.
This paper introduces a deletion interface for a pretrained language model, measuring how effectively deleted records are removed from the model’s memory. By retrofitting a support‑vector memory gate into the global attention layers of a frozen Gemma 3, the authors show that deletions can be performed without altering weights and that the resulting state is close to a reference state that never stored the record. Experiments on 4B‑parameter models demonstrate low perplexity impact and strong evidence that deleted content is hard to recover, while larger or smaller models fail to achieve the same guarantees.
arXiv:2606.24595v2 Announce Type: replace Abstract: Long-term memory promises LLM agents that grow more capable across sessions, maintaining an accurate, evolving understanding of the user that inter...
K-Bench is a new benchmark designed to evaluate large language model (LLM) unlearning when the models are deployed as agents. Unlike previous benchmarks that only inspect the final answer, K-Bench examines all six channels of a ReAct agent—including chain-of-thought, tool calls, tool observations, and elicited summaries—to determine if a secret is leaked. The benchmark measures leakage for secrets placed in the model weights, prompt, or retrieval store, and finds that many existing unlearning methods fail to prevent leaks in deployed agents, especially when secrets reside in the prompt or retrieval store.
arXiv:2608. 07429v1 Announce Type: new Abstract: Long-term memory enables language agents to reuse past facts, preferences, and task experience.
The study shows that in memory‑agent benchmarks, simply retrieving the correct fact does not guarantee accurate answers. On the FactConsolidation task, increasing the evidence budget from 64 to 512 tokens caused accuracy to drop, even though the newest statement was present in every retrieved set. Experiments with Qwen models revealed that replacing an unrelated statement with an obsolete competing statement drastically reduced accuracy, indicating that outdated information can be more harmful than irrelevant evidence.
MemCut introduces a benchmark to test user-requested forgetting in large language model agents, featuring 456 hand-designed secrets embedded in long-horizon conversations with fully annotated evidence trails. The benchmark defines two recovery metrics—native and residual—to assess how well secrets can be retrieved after deletion. Experiments across five memory designs and four LLMs show that deletion reduces native recovery more than residual recovery, highlighting candidate coverage as a key bottleneck.
arXiv:2607. 00605v1 Announce Type: cross Abstract: Limited Memory Language Models (LMLMs) externalize factual knowledge to a database to enable deletion-based unlearning without retraining.
arXiv:2609.08279v1 Announce Type: cross Abstract: Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the...
arXiv:2608. 12476v1 Announce Type: new Abstract: Long-term agent memory is usually treated as select--store--retrieve, but retrieval does not decide whether contradictory, superseded, retracted, deleted, or stale records may support an outgoing claim.
The paper introduces auditability as a fourth evaluation axis for machine unlearning, measuring the separation between forgotten and retained data via a statistic computed by the provider. Experiments on eleven unlearning methods across TOFU, MUSE, and RWKU datasets show none satisfy all four axes, prompting the authors to propose a two‑stage approach: GapWiden, which widens the audit statistic gap using a hinge loss, and GapVeil, an inference‑time rejection sampler that refines responses based on audit statistics. The combined method achieves oracle‑level auditability and leakage control for all tested unlearning techniques, while improving retain performance for most methods.
arXiv:2607. 27539v1 Announce Type: new Abstract: Exact deletion from persistent language-model memory depends on how that memory represents a record.