arXiv AI
Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 10
When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents
The paper introduces MERIT, a benchmark that evaluates the marginal benefit of long‑term memory for tool‑using large language model agents while explicitly accounting for cost. MERIT provides episodic tool‑use tasks across three domains, verifies dependence on earlier‑episode facts, and measures memory operations in tokens and dollars. Experiments on GPT‑4.1‑mini, Claude Haiku 4.5, and Claude Sonnet 5 show that memory can significantly improve task success, but its utility varies widely across models and memory implementations, and full replay is rarely cost‑effective.
arXiv Machine Learning
Jul 10
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
arXiv:2607. 08032v1 Announce Type: new Abstract: Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions.
arXiv AI
3d ago
MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents
MemCut introduces a benchmark to test user-requested forgetting in large language model agents, featuring 456 hand-designed secrets embedded in long-horizon conversations with fully annotated evidence trails. The benchmark defines two recovery metrics—native and residual—to assess how well secrets can be retrieved after deletion. Experiments across five memory designs and four LLMs show that deletion reduces native recovery more than residual recovery, highlighting candidate coverage as a key bottleneck.
arXiv AI
Jun 16
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
arXiv:2605. 18421v2 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution.
arXiv AI
Sep 7
Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents
arXiv:2609. 04875v1 Announce Type: cross Abstract: Long-running LLM agents are stateful: beyond the transcript they accrete compressed summaries, plaintext memory, pending tool plans, and, under every serving API, a KV cache.
arXiv AI
Jul 2
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
arXiv:2607. 01071v1 Announce Type: cross Abstract: Memory has emerged as a cornerstone of modern LLM-based agents, supporting their evolution from single-turn assistants to long-term collaborators.