arXiv AI

When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents

The paper introduces LOCOMO-CONV, a conversational memory benchmark that expands on the existing LoCoMo dataset with four query styles—dialog, implicit, counterfactual, and composed—designed to evaluate memory systems in realistic conversational settings. Experiments across five memory systems reveal that conversational framing uncovers significant retrieval gaps missed by traditional QA benchmarks, particularly for implicit and composed queries, and that strong retrieval does not necessarily translate into higher response quality. The study also highlights silent grounding in implicit queries, where memory enhances contextual grounding without explicitly presenting the gold fact, suggesting a need for reasoning-based memory elaboration.

arXiv Computation and Language

Sep 1

UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory

UTILMEM is a new diagnostic benchmark that tests how conversational agents use long‑term memory, focusing on reasoning over dense histories, spotting implicitly relevant memories, synthesizing distributed evidence, and resisting interference from similar distractors. It contains 1,717 instances across five domains and evaluates a range of retrieval‑based and memory‑augmented systems. The study shows that strong performance on traditional factual recall does not guarantee effective memory utilization, highlighting a gap between retrieving information and integrating it into coherent, task‑oriented outputs.

By Peijun Qing, Fobo Shi, Soroush Vosoughi

agents benchmarks

arXiv AI

Jun 16

T-Mem: Memory That Anticipates, Not Archives

arXiv:2606. 15405v1 Announce Type: cross Abstract: Long-term memory is essential for conversational agents to remain coherent across extended dialogues, follow through on commitments made many sessions earlier, and adapt their behaviour to each user.

By Weidong Guo, Dakai Wang, Zixuan Wang, Hui Liu, Yu Xu

llms agents benchmarks

arXiv AI

Jul 2

Learning User-Aware Recall: Personalized Retrieval in Long-Term Conversational Memory

arXiv:2607. 00017v1 Announce Type: cross Abstract: Long-term conversational agents are expected to remember past interactions, but memory is useful only when the right evidence is recalled for the right user.

By ZhiShu Jiang, Haibo Liu, Xin Shen, Guanqiang QI, Chenxi Miao, Weikang Li, Liwei Qian, Xin Pei, Jizhou Huang

llms agents benchmarks

arXiv AI

Jul 24

RUMBA: Russian User Memory Benchmark

arXiv:2607. 21447v1 Announce Type: cross Abstract: The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning.

By Elizaveta Shevtsova, Inna Glebkina, Mark Baushenko, Pavel Gulyaev, Alena Fenogenova

llms benchmarks

arXiv AI

Jun 2

Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue

arXiv:2606. 01223v1 Announce Type: cross Abstract: Despite substantial progress in long-context modeling, existing benchmarks remain confined to factual memory for explicit recall, failing to measure the reflective memory required to synthesize fragmented, multimodal cues into high-level interpretations.

By Jingjie Lin, Bingbing Wang, Zihan Wang, Zhengda Jin, Weiming Qiao, Jing Li, Ruifeng Xu

multimodal benchmarks safety

arXiv AI

Aug 18

QUMem: Personalized Memory for Query-Conditioned User-State Inference in LLM Agents

arXiv:2608. 16168v1 Announce Type: cross Abstract: Large language model (LLM) agents increasingly use external memory systems to support personalization by drawing on long and evolving interaction histories, in which user preferences may be distributed across time, change with context, and conflict with earlier evidence.

By Heng Wang, Yifei Li, Lingling Zhang, Pengyu Li, Xinyu Che, Xinyu Zhang, Zesheng Yang

llms agents benchmarks

arXiv AI

2d ago

PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations

PRAGMA is a benchmark designed to evaluate personalized guidance in long‑term conversations. It includes curated longitudinal conversation histories, evidence annotations, and guidance scenarios that reflect evolving user contexts and incorrect assumptions. Experiments show that current retrieval, memory, and long‑context models struggle to recover relevant conversational evidence and to use it effectively for personalized guidance.

By Hyojeong Yu, Hyukhun Koh, Minsung Kim, Yunah Jang, Kyomin Jung

llms benchmarks safety

arXiv AI

Jun 9

Goal-Oriented Reasoning for RAG-based Memory in Conversational Agentic LLM Systems

arXiv:2605. 12213v2 Announce Type: replace Abstract: LLM-based conversational AI agents struggle to maintain coherent behavior over long horizons due to limited context.

By Jiazhou Liang, Armin Toroghi, Yifan Simon Liu, Faeze Moradi Kalarde, Liam Gallagher, Scott Sanner

llms rag agents

arXiv AI

Jul 28

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

arXiv:2607. 22690v1 Announce Type: new Abstract: Long-term memory lets LLM agents reuse past interactions, but raw dialogue histories are verbose and information-sparse.

By Jing Yu, Yibo Zhao, Jiaming Zhang, Xiang Li

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 12

MemRefine: LLM-Guided Compression for Long-Term Agent Memory

arXiv:2606. 13177v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly expected to operate over long-term interactions, where information from past dialogues must be preserved and recalled to support future tasks.

By Minjae Kim, Jinheon Baek, Soyeong Jeong, Sung Ju Hwang

llms agents benchmarks

arXiv AI

Aug 5

LeanMem: Simple and Efficient Long-Term Memory for LLM Agents

arXiv:2608. 03463v1 Announce Type: new Abstract: Long-term memory is essential for LLM-based agents to sustain interactions and reliably leverage distant history.

By Yuxin Liao, Le Wu, Min Hou, Hao Liu, Han Wu, Zishu Wang

llms agents nlp

arXiv AI

Jul 8

From Passive Retrieval to Active Memory Navigation: Learning to Use Memory as a Structured Action Space

arXiv:2607. 05794v1 Announce Type: new Abstract: Long-term user memory is essential for personalized conversational agents, yet many memory systems still expose memory through passive retrieval interfaces, making the model a consumer of pre-selected evidence.

By Yue Xu, Yutao Sun, Yihao Liu, Mengyu Zhou, Jiayi Qiao, Lu Ma, Kai Tang, Wenjie Wang, Xiaoxi Jiang, Guanjun Jiang

agents reinforcement-learning

Related stories