← Back to all news
arXiv AI By Cheng Zhang, Xuanning Cao, Zhicheng Zheng, Hui Xiong

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Read the original on arXiv AI →

MemCut introduces a benchmark to test user-requested forgetting in large language model agents, featuring 456 hand-designed secrets embedded in long-horizon conversations with fully annotated evidence trails. The benchmark defines two recovery metrics—native and residual—to assess how well secrets can be retrieved after deletion. Experiments across five memory designs and four LLMs show that deletion reduces native recovery more than residual recovery, highlighting candidate coverage as a key bottleneck.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 10

Deployment-Time Memorization in Foundation-Model Agents

arXiv:2606. 10062v1 Announce Type: new Abstract: Foundation-model agents are increasingly long-lived systems that remember users across interactions, making memorization an explicit deployment-time function rather than solely a property of model weights.

By Lei (Rachel), Chen, Guilin Zhang, Kai Zhao, Dalmo Cirne, Andy Olsen, Xu Chu, Zeke Miller, Alet Blanken, Amine Anoun, Jerry Ting
llmsagentsnlpsafety
More like this →
arXiv AI
Sep 10

When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

The paper introduces MERIT, a benchmark that evaluates the marginal benefit of long‑term memory for tool‑using large language model agents while explicitly accounting for cost. MERIT provides episodic tool‑use tasks across three domains, verifies dependence on earlier‑episode facts, and measures memory operations in tokens and dollars. Experiments on GPT‑4.1‑mini, Claude Haiku 4.5, and Claude Sonnet 5 show that memory can significantly improve task success, but its utility varies widely across models and memory implementations, and full replay is rarely cost‑effective.

By Shweta Mishra, Shashank Mishra
llmsragagentsnlpbenchmarks
More like this →
arXiv AI
Oct 5

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

DyadMem introduces a new benchmark for evaluating how long‑term agents remember user‑specific relational information, called User‑conditioned Relational Agent Memory (URAM). The dataset contains 3,065 episodes, 50,961 sessions, and 61,210 QA pairs, with detailed annotations for memory capture, update, and recall across multi‑session trajectories. Experiments on 20 models show strong performance in a gold‑memory setting but a significant drop in full‑pipeline QA, highlighting gaps in current LLM memory capabilities.

By Yifei Tao, Xinyu Zhong, Henry Hengyuan Zhao, Fanyi Wang, Tengda Guo, Wentao Qiu, Ying Wang, Liujian Tang
llmsagentsbenchmarks
More like this →