← Back to all news
arXiv AI

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

DyadMem introduces a new benchmark for evaluating how long‑term agents remember user‑specific relational information, called User‑conditioned Relational Agent Memory (URAM). The dataset contains 3,065 episodes, 50,961 sessions, and 61,210 QA pairs, with detailed annotations for memory capture, update, and recall across multi‑session trajectories. Experiments on 20 models show strong performance in a gold‑memory setting but a significant drop in full‑pipeline QA, highlighting gaps in current LLM memory capabilities.

Related stories

arXiv AI
Sep 10

When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

The paper introduces MERIT, a benchmark that evaluates the marginal benefit of long‑term memory for tool‑using large language model agents while explicitly accounting for cost. MERIT provides episodic tool‑use tasks across three domains, verifies dependence on earlier‑episode facts, and measures memory operations in tokens and dollars. Experiments on GPT‑4.1‑mini, Claude Haiku 4.5, and Claude Sonnet 5 show that memory can significantly improve task success, but its utility varies widely across models and memory implementations, and full replay is rarely cost‑effective.

By Shweta Mishra, Shashank Mishra
llmsragagentsnlpbenchmarks
More like this →
arXiv AI
Jun 30

Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

arXiv:2507. 05257v4 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks.

By Yuanzhe Hu, Yu Wang, Julian McAuley
llmsragagentsbenchmarks
More like this →
arXiv AI
Jul 8

From Passive Retrieval to Active Memory Navigation: Learning to Use Memory as a Structured Action Space

arXiv:2607. 05794v1 Announce Type: new Abstract: Long-term user memory is essential for personalized conversational agents, yet many memory systems still expose memory through passive retrieval interfaces, making the model a consumer of pre-selected evidence.

By Yue Xu, Yutao Sun, Yihao Liu, Mengyu Zhou, Jiayi Qiao, Lu Ma, Kai Tang, Wenjie Wang, Xiaoxi Jiang, Guanjun Jiang
agentsreinforcement-learning
More like this →
arXiv AI
Jul 21

Accurate and Efficient Long-Term Memory for LLM Agents

arXiv:2607. 16211v1 Announce Type: new Abstract: LLM agents augmented with persistent memory can recall past interactions, but existing systems suffer from two limitations: flat, unstructured storage loses relational context needed for multi-hop and temporal reasoning, and reliance on expensive LLM-based classification makes them impractical for latency-sensitive deployment.

By Zicheng Zhao, Xinyang Guo, Luyao Lv, Menghan Wang, Ming Li, Shuaicheng Li
llmsagents
More like this →
arXiv AI
Aug 18

QUMem: Personalized Memory for Query-Conditioned User-State Inference in LLM Agents

arXiv:2608. 16168v1 Announce Type: cross Abstract: Large language model (LLM) agents increasingly use external memory systems to support personalization by drawing on long and evolving interaction histories, in which user preferences may be distributed across time, change with context, and conflict with earlier evidence.

By Heng Wang, Yifei Li, Lingling Zhang, Pengyu Li, Xinyu Che, Xinyu Zhang, Zesheng Yang
llmsagentsbenchmarks
More like this →
arXiv Computation and Language
Sep 21

MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

MemoryArena is a new evaluation gym that benchmarks agent memory in interdependent multi‑session tasks. Unlike prior benchmarks that test memorization or single‑session action in isolation, MemoryArena requires agents to acquire memory while interacting with the environment and then use that memory to guide future decisions across a range of tasks such as web navigation, planning, information search, and formal reasoning. The benchmark reveals that agents excelling on existing long‑context memory tests perform poorly here, highlighting a gap in current memory evaluation methods.

By Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, Alex Pentland
agentsbenchmarks
More like this →