The Flow has not summarised this story yet — read it at arXiv Computation and Language.
arXiv:2603.04191v2 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as personal assistants, where users may share individual preferences over extended interactio...
The paper introduces the Multi-Session Personalized Tool Calling (MPT) benchmark, containing 4,695 instances across 459 multi‑session histories that test Preference Recall, Induction, and Transfer. It proposes PRefine, a test‑time memory method that refines a user’s latent preference via a generate‑verify‑refine loop. Experiments with five LLMs show that PRefine outperforms existing memory systems and even full‑history prompting on Preference Transfer, suggesting that personalized agents should encode behavior as preferences rather than merely storing past interactions.
The paper introduces MemPABench, an interactive environment designed to evaluate how personal agents remember and act on user interaction preferences. The benchmark features long‑horizon tasks where agents must decide when to clarify, confirm, or act autonomously, using data from four simulated users who provided 99 sessions over four months and five preference changes each. Experiments across four memory systems and LLM backbones show that current memory approaches offer limited improvement in preference selection, with retrieval‑based methods performing no better than no memory and file‑based memory sometimes applying preferences in irrelevant contexts.
The paper investigates whether using a language model’s memory to personalize responses aligns with user preferences. By collecting about 1,000 pairwise judgments from 87 participants on ChatGPT queries that included personal memories, the authors find that memory‑conditioned responses are not consistently preferred and that users sometimes strongly favor non‑memory responses. An automatic reward model trained on these judgments achieves 72.8% preference prediction accuracy and outperforms existing baselines, highlighting the need for evaluation and optimization objectives that better reflect user preferences for memory use.
arXiv:2608. 08300v1 Announce Type: new Abstract: Conversational assistants increasingly rely on persistent long-term memory to personalize responses across sessions.
The paper introduces CAPTURE, a system designed to help personalized language agents distinguish genuine preference changes from temporary context shifts or malicious memory poisoning. CAPTURE employs a neural differential-equation belief tracker, a multi-timescale memory ledger, uncertainty-triggered clarification, and counterfactual auditing to resolve ambiguity. Experiments on 480 episodes from 96 users show CAPTURE outperforms baseline methods, limiting poisoning success while accepting most real preference updates.