← Back to all news
arXiv Machine Learning By Alina Shutova, Alexandra Olenina, Ivan Vinogradov, Anton Sinitsin

Evaluating Memory Structure in LLM Agents

Read the original on arXiv Machine Learning →

The paper introduces StructMemEval, a benchmark designed to assess how well large language model (LLM) agents can organize their long‑term memory rather than merely recall facts. It compiles tasks that humans typically solve by structuring knowledge—such as transaction ledgers, to‑do lists, and trees—and evaluates agents on these. Experiments show that simple retrieval‑augmented LLMs struggle with such organization tasks, while memory‑augmented agents perform better when explicitly prompted to structure their memory, yet many modern LLMs still fail to recognize memory structures without prompting.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jun 30

Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

arXiv:2507. 05257v4 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks.

By Yuanzhe Hu, Yu Wang, Julian McAuley
llmsragagentsbenchmarks
More like this →
arXiv Computation and Language
2d ago

Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents

The paper presents a comprehensive evaluation of 11 memory substrates for long‑horizon LLM agents, using a unified set of 26 performance and efficiency metrics across three backbone models and four benchmark suites. It finds that no single substrate consistently outperforms others; retrieval‑heavy substrates aid long‑context factual QA but can impair sequential decision‑making, and scalability can shift performance trade‑offs. The study demonstrates that selecting a memory substrate tailored to the operating regime yields better results than any fixed choice, offering empirical guidance for designing efficient, reliable, and regime‑aware long‑term memory systems.

By Wei-Chieh Huang, Weizhi Zhang, Yuchen Wu, Yankai Chen, Eric Hanchen Jiang, Wooseong Yang, Yiwei Yang, Henry Peng Zou, Hanrong Zhang, Ying Nian Wu, Haolun Wu, Kai-Wei Chang, Philip S. Yu, Xue Liu, Aylin Caliskan
llmsagentsnlpbenchmarks
More like this →
arXiv Computation and Language
3d ago

AtomMem: Building Simple and Effective Memory System for LLM Agents via Atomic Facts

AtomMem is a long‑term memory system for large language models that stores high‑value atomic facts extracted from long‑form interactions. It organizes these facts into hierarchical event structures and temporal profiles, enabling coherent episodic context and dynamic tracking of user attributes. During retrieval, an associative memory graph connects fragmented memories, and experiments on the LoCoMo benchmark show state‑of‑the‑art performance across reasoning tasks.

By Yanyu Yao, Shangze Li, Zhi Zheng, Hui Zheng, Qi Liu, Tong Xu, Enhong Chen
llmsagentsbenchmarks
More like this →