The paper introduces StructMemEval, a benchmark designed to assess how well large language model (LLM) agents can organize their long‑term memory rather than merely recall facts. It compiles tasks that humans typically solve by structuring knowledge—such as transaction ledgers, to‑do lists, and trees—and evaluates agents on these. Experiments show that simple retrieval‑augmented LLMs struggle with such organization tasks, while memory‑augmented agents perform better when explicitly prompted to structure their memory, yet many modern LLMs still fail to recognize memory structures without prompting.
arXiv:2507. 05257v4 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks.
The paper presents a comprehensive evaluation of 11 memory substrates for long‑horizon LLM agents, using a unified set of 26 performance and efficiency metrics across three backbone models and four benchmark suites. It finds that no single substrate consistently outperforms others; retrieval‑heavy substrates aid long‑context factual QA but can impair sequential decision‑making, and scalability can shift performance trade‑offs. The study demonstrates that selecting a memory substrate tailored to the operating regime yields better results than any fixed choice, offering empirical guidance for designing efficient, reliable, and regime‑aware long‑term memory systems.
arXiv:2606. 04315v1 Announce Type: new Abstract: LLM agents accumulate histories that outgrow their context windows, motivating a growing literature on memory systems.
arXiv:2605. 18421v2 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution.
arXiv:2606. 06787v1 Announce Type: new Abstract: Large Language Models (LLMs) show promise as tool-using agents but remain limited in long-horizon tasks that require remembering, organizing, and reusing knowledge.
AtomMem is a long‑term memory system for large language models that stores high‑value atomic facts extracted from long‑form interactions. It organizes these facts into hierarchical event structures and temporal profiles, enabling coherent episodic context and dynamic tracking of user attributes. During retrieval, an associative memory graph connects fragmented memories, and experiments on the LoCoMo benchmark show state‑of‑the‑art performance across reasoning tasks.
arXiv:2606. 07909v1 Announce Type: new Abstract: Modern large language model (LLM) agents can use external tools to help users solve complex tasks.
arXiv:2607. 16211v1 Announce Type: new Abstract: LLM agents augmented with persistent memory can recall past interactions, but existing systems suffer from two limitations: flat, unstructured storage loses relational context needed for multi-hop and temporal reasoning, and reliance on expensive LLM-based classification makes them impractical for latency-sensitive deployment.
MemoryArena is a new evaluation gym that benchmarks agent memory in interdependent multi‑session tasks. Unlike prior benchmarks that test memorization or single‑session action in isolation, MemoryArena requires agents to acquire memory while interacting with the environment and then use that memory to guide future decisions across a range of tasks such as web navigation, planning, information search, and formal reasoning. The benchmark reveals that agents excelling on existing long‑context memory tests perform poorly here, highlighting a gap in current memory evaluation methods.
DyadMem introduces a new benchmark for evaluating how long‑term agents remember user‑specific relational information, called User‑conditioned Relational Agent Memory (URAM). The dataset contains 3,065 episodes, 50,961 sessions, and 61,210 QA pairs, with detailed annotations for memory capture, update, and recall across multi‑session trajectories. Experiments on 20 models show strong performance in a gold‑memory setting but a significant drop in full‑pipeline QA, highlighting gaps in current LLM memory capabilities.
arXiv:2608. 03463v1 Announce Type: new Abstract: Long-term memory is essential for LLM-based agents to sustain interactions and reliably leverage distant history.
arXiv:2606. 13177v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly expected to operate over long-term interactions, where information from past dialogues must be preserved and recalled to support future tasks.