arXiv AI

Knowing the User Is Not Knowing What We Established: Grounding Awareness in Personal Agents

arXiv Computation and Language
Sep 21

MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

MemoryArena is a new evaluation gym that benchmarks agent memory in interdependent multi‑session tasks. Unlike prior benchmarks that test memorization or single‑session action in isolation, MemoryArena requires agents to acquire memory while interacting with the environment and then use that memory to guide future decisions across a range of tasks such as web navigation, planning, information search, and formal reasoning. The benchmark reveals that agents excelling on existing long‑context memory tests perform poorly here, highlighting a gap in current memory evaluation methods.

By Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, Alex Pentland
arXiv AI
Sep 3

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

AdaMem introduces adaptive memory policies that allow personalized agents to decide what information to write into long‑term memory based on user preferences for each interaction context. Each policy is updated from periodic feedback and controls subsequent memory writing, aiming to improve relevance and reduce unnecessary memory persistence. In experiments on AdaMem‑Bench, AdaMem raises QA accuracy from 80.0% to 84.35% while cutting persistent memory by 9.27%, though models still struggle to execute policies reliably.

By Xingyu Chen, Rui Wang, Zhaopeng Tu, Liefeng Bo
arXiv AI
Oct 19

Testing the Planning Claims of LLM-Based Agents against Formal Semantics

The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.

By Oriol Miro-Lopez-Feliu, Oscar Mir\'o L\'opez-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna, Sergio Alvarez-Napagao
arXiv Computation and Language
Oct 23

Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences

The paper introduces MemPABench, an interactive environment designed to evaluate how personal agents remember and act on user interaction preferences. The benchmark features long‑horizon tasks where agents must decide when to clarify, confirm, or act autonomously, using data from four simulated users who provided 99 sessions over four months and five preference changes each. Experiments across four memory systems and LLM backbones show that current memory approaches offer limited improvement in preference selection, with retrieval‑based methods performing no better than no memory and file‑based memory sometimes applying preferences in irrelevant contexts.

By Jialin Li, Zhenhao Chen, Hanjun Luo, Hanan Salam
arXiv AI
Sep 10

Latent Preference Modeling for Multi-Session Personalized Tool Calling

The paper introduces the Multi-Session Personalized Tool Calling (MPT) benchmark, containing 4,695 instances across 459 multi‑session histories that test Preference Recall, Induction, and Transfer. It proposes PRefine, a test‑time memory method that refines a user’s latent preference via a generate‑verify‑refine loop. Experiments with five LLMs show that PRefine outperforms existing memory systems and even full‑history prompting on Preference Transfer, suggesting that personalized agents should encode behavior as preferences rather than merely storing past interactions.

By Yejin Yoon, Minseo Kim, Taeuk Kim
arXiv AI
Oct 7

MemCo: Memory-Centric Collaboration for Generalizing LLM Agents to Unseen Environments

MemCo introduces a memory‑centric collaboration framework that equips large language model agents with both local and global memory spaces. Local memory preserves environment‑specific details, while global memory captures transferable workflows from local trajectories. During online interaction, MemCo selectively routes relevant memories based on the agent’s state and decision phase, enabling reuse of other agents’ experiences without over‑generalizing environment‑specific information.

By Xinting Liao, Siyan Liu, Rabab K. Ward, Holger R. Roth, Xiaoxiao Li